Category: Concepts

This category contains 55 pages.

  • Activation steering · Adding vectors to a transformer's residual stream at inference time to shift model behavior; used both as a control knob and as causal evidence in interpretability.
  • Activation verbalizer · Anthropic technique in which a language model is trained to translate another model's token-level activations into natural-language summaries.
  • Adam (optimizer) · The adaptive optimization algorithm introduced by Kingma and Ba in 2014, combining momentum with per-parameter learning rates, used to train nearly every modern language model.
  • Analytic distillation · The extraction of what a trained model knows into smaller, cheaper, or more explicit form, spanning classic knowledge distillation, rationale distillation, and mechanistic interpretability.
  • Attention sink · The tendency of autoregressive transformers to concentrate attention mass on initial tokens; exploited by StreamingLLM and built into gpt-oss as learned biases.
  • Backpropagation · The chain-rule algorithm that computes a neural network's loss gradient in a single backward pass, developed by Werbos in the early 1970s and popularized by Rumelhart, Hinton, and Williams in 1986.
  • Boltzmann machine · The stochastic recurrent network of Ackley, Hinton, and Sejnowski (1985) that learns distributions by contrasting clamped and free-running statistics; its restricted form seeded deep belief networks.
  • Chain-of-thought prompting · Eliciting step-by-step reasoning before the answer; discovered as a prompting trick in 2022, later internalized by trained reasoning models.
  • Circuits (interpretability) · Subnetworks of a neural network that implement identifiable algorithms; the central object of study in mechanistic interpretability.
  • Compositional generalization · The capacity to understand and produce novel combinations of known components, the systematicity property long contested in neural networks and a central open question for large language models.
  • Cross-entropy loss · The standard training loss of language modeling: the negative log probability assigned to the correct token, the quantity plotted on every scaling-law curve.
  • Diffusion language model · Text generation by iterative parallel denoising of masked or corrupted token sequences, an alternative to left-to-right autoregression.
  • Dropout · The regularization technique that randomly zeroes hidden units during training, standard in early Transformers and largely retired as pretraining shifted to single-epoch, web-scale data.
  • Eliciting Latent Knowledge · Open alignment problem posed by ARC in 2021: training a model to report what it internally knows rather than what looks correct to human overseers.
  • Embedding (machine learning) · The mapping from discrete tokens to continuous vectors that forms the input and output interface of neural language models and encodes semantic similarity as geometry.
  • Feed-forward network · The position-wise two-layer perceptron inside every Transformer block, holding most of the stack's parameters and complementing attention's routing with per-token computation.
  • Hopfield network · John Hopfield's 1982 recurrent neural network that stores patterns as minima of an Ising-like energy function and retrieves them by relaxation, the model that made associative memory rigorous.
  • Hybrid architecture (LLM) · Model designs that interleave self-attention with state-space, recurrent, or linear-attention layers, or mix sliding-window and full attention, to cut KV-cache and compute costs.
  • In-context learning · The ability of large language models to perform a new task from instructions or examples in the prompt alone, without any weight update, demonstrated at scale by GPT-3 in 2020.
  • Knowledge distillation · Training a smaller student model to imitate a larger teacher; the mechanism behind small frontier-adjacent models and a persistent IP controversy.
  • Large language model · A neural language model, typically a decoder-only Transformer with billions of parameters, pretrained on internet-scale text.
  • Layer normalization · Per-token normalization of activations across the feature dimension, introduced by Ba, Kiros, and Hinton in 2016 and built into every Transformer block as part of the add-and-norm pattern.
  • Logit lens · Decoding a transformer's intermediate residual-stream states through its output unembedding, revealing layer-by-layer refinement of next-token predictions.
  • LSTM · Long short-term memory: the 1997 recurrent architecture that dominated sequence modeling for two decades before the Transformer.
  • Masked language modeling · The pretraining objective of predicting hidden tokens from bidirectional context, introduced at scale by BERT in 2018 and the training signal of the encoder-only lineage.
  • Mechanistic interpretability · The research program of reverse-engineering neural networks into human-understandable features and circuits, from the 2020 Distill circuits thread to routine white-box analyses in frontier system cards.
  • Mixture of experts · Sparse architecture in which a router activates only a few expert subnetworks per token, decoupling parameter count from per-token compute.
  • MMLU · Massive Multitask Language Understanding, a 57-subject multiple-choice knowledge benchmark introduced in 2020 and long the standard headline eval for language models.
  • Multi-head attention · Running attention in parallel subspaces within each layer; its head-sharing variants (MQA, GQA, MLA) set the size of the KV cache and much of the cost of inference.
  • Multi-head latent attention · Attention variant that caches a low-rank latent compression of keys and values instead of full per-head tensors; introduced in DeepSeek-V2 (2024).
  • Next-token prediction · The objective of predicting each token from all preceding ones, the training signal behind GPT-style language models and the foundation of autoregressive generation.
  • PagedAttention · Virtual-memory-style paging for the KV cache, introduced with vLLM in 2023; cut serving memory waste to near zero and became standard across inference stacks.
  • Perceptron · Frank Rosenblatt's 1957 learning machine: the first artificial neural network shown to learn its weights from data, with a provably convergent rule and a famous linear-separability limitation.
  • Positional encoding · The mechanism that injects token order into the otherwise permutation-invariant Transformer, evolving from fixed sinusoidal vectors to the rotary embeddings used by most modern LLMs.
  • Pretraining · The first phase of language model development: self-supervised training over large text corpora that produces a base model, before post-training adapts it for use.
  • Reward model · A model trained to score another model's outputs against human or AI preferences, supplying the training signal for RLHF and, in process form, for reasoning search.
  • RMSNorm · Root mean square layer normalization: a simplification of layer normalization that rescales activations by their root mean square alone, the default normalizer of modern open-weight LLMs.
  • Scaling laws · Empirical power-law relationships between model performance and parameters, data, and compute; the planning instrument of the LLM era.
  • Self-attention · Mechanism by which each position in a sequence computes its representation as an attention-weighted sum over all positions; the core operation of the Transformer.
  • Seq2seq · Sequence-to-sequence learning: the 2014 encoder-decoder paradigm that mapped variable-length input to output through a fixed context vector, the direct ancestor of the attention era.
  • Sliding-window attention · Restricting each token's attention to a fixed-size local window, cutting attention cost from quadratic to linear in context length and bounding KV-cache memory.
  • Softmax · The function that turns a vector of raw scores into a probability distribution, used in attention weighting, vocabulary output layers, and mixture-of-experts routing.
  • Software 2.0 · Andrej Karpathy's November 2017 essay framing neural networks as a new programming paradigm in which code is written by optimization over datasets rather than by hand.
  • Sparse attention · Attention variants in which each token attends to a selected subset of positions instead of the full sequence, cutting the quadratic cost of self-attention.
  • Sparse autoencoder · Dictionary learning on language model activations; decomposes superposed representations into sparse, often monosemantic features.
  • State-space model · Sequence architecture family (S4, Mamba) that replaces attention with a fixed-size recurrent state, giving linear-time processing and constant-memory generation.
  • Superposition (interpretability) · The hypothesis that neural networks represent more features than they have neurons by packing them into overlapping directions, explaining polysemantic neurons and motivating sparse autoencoders.
  • Teacher forcing · The training protocol that feeds a sequence model the true previous tokens instead of its own predictions, enabling fully parallel next-token training at the cost of exposure bias.
  • Test-time compute · Improving answers by spending more computation at inference (longer reasoning chains, search, sampling) rather than more pretraining; the axis behind reasoning models.
  • The Bitter Lesson · Rich Sutton's March 2019 essay arguing that seventy years of AI research show general methods leveraging computation beating methods built on human knowledge.
  • The Scaling Hypothesis · Gwern Branwen's May 2020 essay on GPT-3: meta-learning, the blessings of scale, and the theory that intelligence is simple learning algorithms applied at sufficient scale.
  • Tokenization · The conversion of raw text into the integer token ids that language models read and write, dominated by byte-pair encoding variants with vocabularies of tens to hundreds of thousands of tokens.
  • Transformer (architecture) · Neural network architecture based on self-attention, introduced by Vaswani et al. in 2017; the basis of virtually all modern large language models.
  • Vision Transformer · Transformer applied to images by treating fixed-size patches as tokens; introduced by Dosovitskiy et al. in 2020, now the standard vision encoder in multimodal LLMs.
  • Word2vec · The 2013 word-embedding method whose vector arithmetic ('king - man + woman = queen') made distributed representations the foundation of neural NLP.