InstructGPT is the family of RLHF-aligned GPT-3 variants that OpenAI deployed in early 2022, described in the paper "Training language models to follow instructions with human feedback." It is the direct technical ancestor of ChatGPT and the demonstration that made post-training a discipline of its own.1

What it showed

The paper's headline result reordered the field's priorities: human raters preferred the 1.3-billion-parameter InstructGPT over the raw 175-billion-parameter GPT-3. Alignment training, a comparatively tiny amount of compute, produced more perceived quality than a hundredfold of scale. The models followed instructions, produced less toxic output, hallucinated somewhat less, and generalized instruction-following to formats unseen in training, at a measurable but small cost on academic benchmarks that the paper called an "alignment tax."1

Method

The three-stage pipeline became the industry standard documented at Reinforcement learning from human feedback: supervised fine-tuning on contractor demonstrations, a reward model trained on ranked comparisons, and PPO optimization against that reward with a KL penalty to the SFT baseline. The paper's careful discussion of labeler demographics and disagreement remains a reference point in debates about whose preferences alignment encodes.1

Legacy

ChatGPT (November 2022) was, per OpenAI, a sibling of InstructGPT trained with the same methods on a GPT-3.5-series base; GPT-4 inherited the pipeline. The recipe spread to every lab: Anthropic extended it into Constitutional AI, open implementations from Hugging Face and academic groups reproduced it on Llama-class bases, and successor objectives like DPO were derived directly against its formulation. Few papers on this wiki have a larger downstream footprint.23

References


  1. Ouyang, L., et al., "Training language models to follow instructions with human feedback," arXiv:2203.02155, March 2022. ↩↩↩

  2. OpenAI, "Introducing ChatGPT," November 2022. ↩

  3. Bai, Y., et al., "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback," arXiv:2204.05862, April 2022. ↩