LLaVA

LLaVA (Large Language and Vision Assistant) is the April 2023 model and recipe from Haotian Liu and collaborators that defined how the open ecosystem builds vision-language models: take a frozen CLIP vision encoder, bridge it to an open LLM through a small projection, and instruction-tune the pair on multimodal conversations synthesized by a stronger model. Nearly every open VLM since is a descendant of this template.¹

Method

LLaVA's insight was data, not architecture: GPT-4 (text-only) was prompted with image captions and bounding boxes to generate 158,000 visual instruction-following conversations, the first application of synthetic-data pipelines to multimodal training. A single linear projection (later a small MLP) maps CLIP features into the token space of Vicuna, and two short training stages, feature alignment then instruction tuning, produce a conversational assistant on academic hardware budgets.¹

Impact

LLaVA-1.5 (October 2023) reached state-of-the-art across eleven benchmarks with trivially simple modifications, embarrassing far heavier architectures, and LLaVA-NeXT (2024) extended to higher resolutions and video. The recipe's descendants span the open multimodal wave this wiki documents: the Qwen-VL and InternVL lines industrialized it at scale, and its evaluation suites became the standard MMLU-analogues of vision-language work. Its GPT-4-generated training data also made it an early exhibit in the distillation-from-API debates.² The project's fully open pipeline, data generation scripts, training code, and checkpoints on Hugging Face, made it the Alpaca of multimodality: reproducible on academic budgets, forked constantly, and cited as the baseline in essentially every open VLM paper since. Its descendants include the compact deployment-focused lines documented at Moondream (model family), and its benchmark suite remains the standard first evaluation for new vision-language models.³

See also

• CLIP
• Vicuna
• Moondream (model family)

References