Instruction tuning is the supervised fine-tuning of a pretrained language model on instruction-response demonstrations so that it performs tasks described in natural language rather than merely continuing text. It is the universal first stage of modern post-training, preceding preference methods like RLHF.1

How it works

A pretrained model is trained with the ordinary next-token objective on curated pairs of instructions and target responses, drawn from templated NLP datasets, human demonstrations, or synthetic generation. The 2021 FLAN and T0 papers showed that multitask instruction tuning produces zero-shot generalization to held-out task types; InstructGPT made human-written demonstrations the SFT foundation beneath its preference training.12

Variants

Self-Instruct (2022) bootstrapped instruction data from a model's own generations, and Stanford's Alpaca showed in early 2023 that a few thousand dollars of synthetic instructions could turn Meta AI's leaked LLaMA weights into a passable assistant, launching the open fine-tuning wave documented across the Llama family pages. Modern SFT datasets mix human, synthetic, and distilled data from stronger models, a pipeline industrialized by every lab from OpenAI and Anthropic to Mistral AI and the open recipes of Hugging Face and EleutherAI-adjacent groups.23

Limitations

SFT teaches format and compliance more reliably than judgment: models imitate demonstrations, including their errors, and cannot express preference margins between better and worse responses. Knowledge benchmarks such as MMLU move little under instruction tuning, while instruction-following evals move enormously; the gap is why preference-based methods and Constitutional AI-style feedback follow SFT in the standard stack of Transformer assistants.13

References


  1. Wei, J., et al., "Finetuned Language Models Are Zero-Shot Learners," arXiv:2109.01652, September 2021. ↩↩↩

  2. Ouyang, L., et al., "Training language models to follow instructions with human feedback," arXiv:2203.02155, March 2022. ↩↩

  3. Wang, Y., et al., "Self-Instruct: Aligning Language Models with Self-Generated Instructions," arXiv:2212.10560, December 2022. ↩↩