IDEFICS

IDEFICS (Image-aware Decoder Enhanced à la Flamingo with Interleaved Cross-attentionS) is an open-weight visual language model released by Hugging Face in August 2023 as a reproduction of Google DeepMind's closed Flamingo system. Built from a frozen LLaMA language backbone and an OpenCLIP vision encoder connected by Flamingo-style gated cross-attention, it shipped in 9B and 80B parameter variants, each with a base and an instruction-tuned version, and accepted arbitrary interleaved sequences of images and text.¹

Purpose and training

The project's stated goal was transparency: Flamingo's weights and training data were never released, so IDEFICS was trained exclusively on openly available data to give researchers a reproducible reference point for that class of model. Its centerpiece was OBELICS, an openly released corpus of interleaved image-text documents filtered from web crawls, described in a dedicated dataset paper (arXiv:2306.16527) and reused by later multimodal efforts.² Hugging Face reported performance approaching the original Flamingo on standard image-text benchmarks such as visual question answering and captioning, with the usual caveat that closed baselines could not be evaluated directly.¹

Successors

Later revisions moved away from the strict Flamingo recipe. IDEFICS2 (2024) switched to a Mistral 7B-based backbone with a simpler connector design at roughly 8B parameters, and Idefics3 (also 2024) adopted a Llama 3-based backbone; both reported substantial gains over the original at a fraction of its size, tracking the field's drift toward compact vision-language models exemplified by PaliGemma, Moondream, and Qwen2.5-VL.³

Position

IDEFICS is cited as an early proof that a competitive interleaved VLM could be assembled entirely from public components, a milestone for open multimodal research alongside open text-side efforts such as BLOOM, Pythia, and OLMo.²

References