Flamingo

Flamingo is a family of visual language models (VLMs) announced by DeepMind in April 2022.¹ It was among the first systems to demonstrate that a large language model could be adapted to interleaved sequences of images, video, and text with only lightweight new components, and that the resulting model could learn new vision-language tasks from a handful of in-context examples, mirroring the few-shot prompting behavior that GPT-3 had established for text.²

Architecture

Flamingo connects two frozen pretrained networks: a contrastively trained NFNet vision encoder (conceptually similar to CLIP's image tower) and a Chinchilla language model. Visual features are compressed into a fixed number of tokens by a Perceiver Resampler, then injected into the language model through newly inserted gated cross-attention dense layers whose gates are initialized so the model starts as the unmodified language model.¹ Because both backbones stay frozen, only the bridging modules are trained. The family spans roughly 3B, 9B, and 80B parameters; the flagship Flamingo-80B pairs the 70B-parameter Chinchilla with the added adapter layers.

Training and capabilities

The models were trained on a mixture of web pages with interleaved images and text (the M3W dataset) plus paired image-text and video-text corpora.¹ At release, developer-reported results showed Flamingo-80B setting a new few-shot state of the art on a wide range of benchmarks, including visual question answering and captioning, and surpassing fine-tuned specialist models on several tasks using only 32 in-context examples.² The model handled multi-image dialogue-style prompts, an ability that prefigured later chat-oriented multimodal assistants.

Legacy

Flamingo was a research system and its weights were never released, but its recipe of grafting visual inputs onto a frozen language model became a template for subsequent open efforts, including the OpenFlamingo reproduction and the IDEFICS models published on Hugging Face.³ Its interleaved-data training and gated cross-attention design influenced later multimodal work at Google, including PaLM-era VLMs and the natively multimodal Gemini line, and it is a standard point of comparison for compact VLMs such as Moondream and PaliGemma.

References