GPT-2 is a 1.5-billion-parameter language model released by OpenAI in February 2019. Its paper, "Language Models are Unsupervised Multitask Learners," argued that a sufficiently large Transformer trained only on next-token prediction begins to perform tasks it was never explicitly trained for, reading comprehension, translation, summarization, purely from prompting: the observation that seeded the LLM paradigm.1

Background and training

GPT-2 scaled the GPT-1 recipe roughly tenfold, training four model sizes on WebText, a corpus of about 40 gigabytes of text from outbound Reddit links chosen as a quality heuristic. No fine-tuning stage was involved; every evaluation probed the raw language model zero-shot.1

The staged release

OpenAI initially withheld the full model, citing misuse concerns around synthetic text, releasing sizes incrementally through 2019 and publishing the 1.5B weights that November alongside a report on misuse monitoring. The decision split the field: critics called it publicity-driven security theater that broke open-publication norms; defenders framed it as the first serious release-policy experiment for capable models. Every subsequent open-weights debate on this wiki, from Meta AI's Llama licensing to xAI's Grok-1 drop, replays arguments first rehearsed over GPT-2.2

Impact

Technically, GPT-2 validated the scaling path that scaling-law work would formalize a year later and that GPT-3 would cash out at 175 billion parameters.3 Culturally, its "too dangerous to release" coverage introduced the public to generative text models, and its open weights became the standard research substrate for a generation of interpretability and fine-tuning work, a role later inherited by EleutherAI's replications and the open lines of Hugging Face's ecosystem.2

Lineage

Predecessor: GPT-1 (2018). Successor: GPT-3 (2020). See GPT (model family).

References


  1. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., "Language Models are Unsupervised Multitask Learners," OpenAI, February 2019. ↩↩

  2. OpenAI, "Better Language Models and Their Implications" and the November 2019 release report, openai.com, 2019. ↩↩

  3. Kaplan, J., et al., "Scaling Laws for Neural Language Models," arXiv:2001.08361, January 2020. ↩