Large language model

A large language model (LLM) is a neural network, in nearly every modern case a decoder-only Transformer, trained on next-token prediction over internet-scale text corpora, with parameter counts running from billions to (reportedly) trillions. The term denotes less a specific architecture than a paradigm: general capability acquired as a byproduct of scale, elicited through prompting and refined through post-training.¹

Three findings define the LLM era. First, self-attention made it practical to train very large sequence models in parallel. Second, GPT-3 demonstrated in 2020 that a sufficiently large pretrained model performs new tasks from instructions and examples in its context window, without gradient updates: in-context learning.¹ Third, post-training methods, above all RLHF, turned raw next-token predictors into usable assistants, a transition completed publicly by ChatGPT in late 2022 and GPT-4 in 2023.²³

Role in the network

An LLM computes a conditional probability distribution over text. A tokenizer splits raw text into subword units from a fixed vocabulary of tens to hundreds of thousands,⁴ each id is mapped to a learned embedding vector, and the Transformer stack transforms the sequence position by position. A final unembedding matrix scores every vocabulary entry as a next-token candidate, so the model's output at each position is a distribution over the whole vocabulary. This turns any text corpus into a supervised dataset for free: every token supplies its own label, making internet-scale pretraining possible without annotation.

Training fits the weights by making the corpus more probable under the model. With teacher forcing the model scores each token given those before it, and backpropagation of the prediction loss drives updates with an Adam-family optimizer.⁵ Pretraining runs once at large cost and yields a base model that post-training adapts into a product-facing assistant. In deployment the model conditions on a fixed context window, its working memory, served through APIs or as open weights.

The mathematics

An LLM factors a document's probability into next-token probabilities by the chain rule:

$$p_\theta(x) = \prod_{t=1}^{T} p_\theta(x_t \mid x_{<t})$$

where \(x = (x_1, \ldots, x_T)\) is a token sequence, \(x_{<t}\) the prefix before position \(t\), \(\theta\) the model parameters, and each factor the probability of the observed next token.

Training maximizes this likelihood, equivalently minimizing the total negative log-likelihood:

$$\mathcal{L}(\theta) = -\sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t})$$

Symbol	Shape	Meaning
\(x_t\)	scalar (token id)	observed token at position \(t\)
\(x_{<t}\)	sequence of \(t - 1\) ids	conditioning prefix
\(p_\theta(\cdot \mid x_{<t})\)	distribution over \(V\) tokens	predictive distribution at position \(t\)
\(T\)	scalar	sequence length
\(\mathcal{L}\)	scalar	corpus negative log-likelihood, in nats

Each factor is a softmax over the vocabulary: with \(h_t \in \mathbb{R}^{d_{\text{model}}}\) the stack's output vector at position \(t\) and \(W_U \in \mathbb{R}^{V \times d_{\text{model}}}\) the unembedding matrix, the logits are \(z = W_U h_t\) and

$$p_\theta(x_t = i \mid x_{<t}) = \frac{\exp(z_i)}{\sum_{j=1}^{V} \exp(z_j)}$$

so the per-token loss \(-\log p_\theta(x_t \mid x_{<t})\) is the cross-entropy between the model's distribution and the one-hot observed token. Reported quality is usually perplexity, the exponentiated mean per-token loss:

$$\mathrm{PPL} = \exp\!\left(\frac{\mathcal{L}}{T}\right)$$

Perplexity is the effective number of equally likely continuations the model chooses among: a perfect predictor scores 1, a uniform predictor over the vocabulary scores \(V\).

Worked example: loss and perplexity over a vocabulary of 4

Take vocabulary \(\{A, B, C, D\}\) and suppose the model emits logits \(z = (2, 1, 0, -1)\) at some position. The exponentials are \(e^2 \approx 7.389\), \(e^1 \approx 2.718\), \(e^0 = 1\), and \(e^{-1} \approx 0.368\), summing to \(11.475\), so

$$p = \left(\tfrac{7.389}{11.475},\, \tfrac{2.718}{11.475},\, \tfrac{1}{11.475},\, \tfrac{0.368}{11.475}\right) \approx (0.644,\, 0.237,\, 0.087,\, 0.032)$$

If the observed token is \(A\), the loss here is \(-\ln 0.644 \approx 0.440\) and the perplexity \(e^{0.440} \approx 1.55\). A uniform model would score loss \(\ln 4 \approx 1.386\) and perplexity exactly 4; a certain, correct model would score 0 and 1.

Empirically, the mean loss of a well-trained model is a smooth function of scale. Kaplan et al. reported power-law decreases in test loss with parameters, data, and compute in 2020,⁶ and Hoffmann et al.'s 2022 compute-optimal fit takes the form

$$L(N, D) = aN^{-\alpha} + bD^{-\beta} + E$$

where \(N\) is the parameter count, \(D\) the training tokens, \(E\) an irreducible loss floor, and \(a, b, \alpha, \beta\) positive fitted constants. Its corollary, the Chinchilla rule, holds parameters and tokens in proportion at roughly 20 training tokens per parameter: the 70-billion-parameter Chinchilla was trained on about 1.4 trillion tokens (\(70 \times 10^{9} \times 20 = 1.4 \times 10^{12}\)) and outperformed the 280-billion-parameter Gopher, trained on far less text.⁷ Karpathy's ''Neural Networks: Zero to Hero'' series walks through this construction in code.⁸

Intuition

Statistically, an LLM is a lossy compression of its training distribution: since predicting the next token is easier with a model of how the text came to be, the standard argument runs, prediction at scale forces the network to internalize syntax, facts, and reasoning patterns as byproducts.¹ The softmax makes each prediction a competition: every vocabulary entry bids for mass with its logit, and perplexity converts the result into an effective branching factor: a model at perplexity 10 chooses among about 10 plausible continuations per token.

The same conditioning machinery yields in-context learning: a prompt is only a prefix, so a model trained to continue text continues the patterns the prompt contains (question-answer and few-shot formats), with no weight update.¹ And the scaling law turns capability into a purchasing decision: if loss is smooth in parameters and data, larger models are bought rather than discovered, the economic content of scaling laws.

Variants and extensions

The decoder-only form is dominant but not exclusive. Encoder-only models, trained with masked language modeling, remain standard for classification and retrieval; encoder-decoder models such as T5 persist in translation-style tasks; and diffusion language models replace left-to-right generation with iterative denoising. Sparse mixture-of-experts decoders such as Mixtral and DeepSeek-V3 route each token through a few of many feed-forward experts, decoupling parameters from per-token compute.⁹¹⁰

Post-training has its own variant stack: instruction tuning on demonstrations, preference optimization through RLHF or direct preference optimization,²¹¹ and, from late 2024, reasoning-focused post-training that spends additional inference-time compute on long chains of thought before answering, trained with reinforcement learning on verifiable rewards: OpenAI's o1 series and DeepSeek-R1.¹²¹³ At the small end, distillation transfers a large model's behavior into a compact student,¹⁴ and paired encoders extend the stack to images and audio, as in CLIP and LLaVA.¹⁵

Adoption

Frontier LLM development is concentrated in a handful of laboratories (OpenAI and Google DeepMind among the most prominent) alongside a broad open-weight ecosystem.¹ As of mid-2026 the frontier set also includes Anthropic, Meta AI, and xAI in the United States and DeepSeek, Moonshot AI, and Zhipu AI in China, while the Llama, Qwen, and DeepSeek open-weight lines, distributed through Hugging Face, support a secondary ecosystem of fine-tunes and local deployment. The field's earlier encoder-based branch, exemplified by BERT, remains in production for understanding tasks but ceded the generative frontier to decoder-only scaling.¹

LLMs are compared on standardized benchmark suites measuring knowledge, reasoning, coding, and safety behavior; MMLU served as the field's headline knowledge metric for most of the 2020s.¹⁶

Limitations and critiques

The most visible reliability failure is hallucination: fluent, confident output that does not match the facts, documented across neural text generation and inherited at scale.¹⁷ Benchmark saturation and contamination are persistent methodological complaints, and developer-reported scores are conventionally read with an "as reported" caveat.¹⁶ Models can also regurgitate near-verbatim training passages, a demonstrated privacy and copyright exposure.¹⁸

A second critique line targets the paradigm itself. Bender, Gebru, McMillan-Major, and Shmitchell's "stochastic parrots" paper argued that ever-larger models amplify encoded biases and environmental costs while remaining ungrounded in meaning,¹⁹ and Epoch AI projected that high-quality public text could be exhausted in the late 2020s at then-current scaling rates, pushing laboratories toward synthetic and licensed data.²⁰ Training and serving costs concentrate frontier capability in the few organizations able to pay.⁶

History

Statistical language modeling long predates neural networks: Shannon's 1948 formalization of information theory already illustrated English with n-gram approximations,²¹ and smoothed n-gram models remained standard in speech recognition and machine translation into the 2000s.²² Bengio, Ducharme, Vincent, and Jauvin's 2003 neural probabilistic language model learned distributed word vectors jointly with the probability model,²² and word2vec made such embeddings a transferable commodity in 2013.²³ The sequence-to-sequence framework of 2014 and attention in 2014 and 2015 supplied the architectural substrate,²⁴²⁵ and 2018 brought the pretraining-transfer recipe: ELMo's deep contextual features,²⁶ ULMFiT's fine-tuned recurrent model,²⁷ OpenAI's generatively pretrained Transformer decoder (GPT, June 2018),²⁸ and Google's bidirectional BERT (October 2018).²⁹

The decoder line then scaled: GPT-2 (February 2019) reached 1.5 billion parameters and showed zero-shot task behavior,³⁰ Kaplan et al.'s scaling laws (January 2020) made the returns quantitative,⁶ and GPT-3 (May 2020) demonstrated in-context learning at 175 billion parameters.¹ Stanford researchers coined "foundation model" for the resulting paradigm in 2021,³¹ and Chinchilla's compute-optimal analysis rebalanced parameters against data in 2022.⁷ Instruction tuning and RLHF converted base models into assistants (InstructGPT, March 2022),² ChatGPT (November 2022) brought the paradigm to mass use, and GPT-4 (March 2023) set the frontier template.³ Meta's LLaMA (February 2023) opened competitive weights to researchers and seeded the open-weight ecosystem,³² and from late 2024 the reasoning line (OpenAI's o1 series, then the open DeepSeek-R1 in January 2025) extended the recipe with inference-time compute.¹²¹³

See also

• Transformer (architecture)
• Next-token prediction
• Scaling laws
• GPT (model family)

References