GPT-1

GPT-1 (originally just "GPT," for Generative Pre-trained Transformer) is a language model announced by OpenAI in June 2018 alongside the paper "Improving Language Understanding by Generative Pre-Training" by Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.¹ It was the first model in the GPT family and is widely credited with demonstrating that generative pretraining on unlabeled text, followed by discriminative fine-tuning, could outperform task-specific architectures across a broad range of natural language understanding benchmarks.

Architecture and training

GPT-1 is a 12-layer, decoder-only Transformer with masked Self-attention, 768-dimensional hidden states, and 12 attention heads, totaling roughly 117 million parameters. It was pretrained with a standard left-to-right language modeling objective on the BookCorpus dataset, a collection of several thousand unpublished books chosen because its long contiguous passages let the model learn long-range dependencies. The model used byte-pair encoding for tokenization and a context window of 512 tokens.¹

After pretraining, the model was adapted to downstream tasks (natural language inference, question answering, semantic similarity, and text classification) through supervised fine-tuning, with task inputs converted into ordered token sequences so that a single architecture could serve many tasks. At release it achieved state-of-the-art results on 9 of the 12 evaluated datasets.¹

Significance

GPT-1 arrived months before Google's BERT, and the two models framed a period of rapid progress in transfer learning for NLP: GPT-1 established the decoder-only generative pretraining recipe, while BERT popularized bidirectional masked-language-model pretraining.² OpenAI released the code and trained weights publicly, and the paper's zero-shot analyses, showing that pretraining alone conferred some task ability without fine-tuning, foreshadowed the scaling-driven agenda pursued in GPT-2 and GPT-3.³ The lineage that followed, through GPT-3.5, GPT-4, and later systems, retained GPT-1's core decoder-only design while growing parameter counts by several orders of magnitude, a trajectory later formalized in work on Scaling laws for large language models.

Legacy

Although GPT-1 itself saw little direct deployment, its recipe became the template for essentially all subsequent frontier language models, including families from labs such as Anthropic and Google. Retrospectives commonly treat it as the starting point of the modern LLM era, and its modest size makes it a frequent reference point when illustrating how far model scale has grown since 2018.

References