Category: Architectures

This category contains 28 pages.

  • Attention sink · The tendency of autoregressive transformers to concentrate attention mass on initial tokens; exploited by StreamingLLM and built into gpt-oss as learned biases.
  • Boltzmann machine · The stochastic recurrent network of Ackley, Hinton, and Sejnowski (1985) that learns distributions by contrasting clamped and free-running statistics; its restricted form seeded deep belief networks.
  • Diffusion language model · Text generation by iterative parallel denoising of masked or corrupted token sequences, an alternative to left-to-right autoregression.
  • Embedding (machine learning) · The mapping from discrete tokens to continuous vectors that forms the input and output interface of neural language models and encodes semantic similarity as geometry.
  • Feed-forward network · The position-wise two-layer perceptron inside every Transformer block, holding most of the stack's parameters and complementing attention's routing with per-token computation.
  • FlashAttention · IO-aware exact attention algorithm that tiles the computation to fit in GPU on-chip memory; the de facto standard attention kernel in training and serving stacks.
  • Hopfield network · John Hopfield's 1982 recurrent neural network that stores patterns as minima of an Ising-like energy function and retrieves them by relaxation, the model that made associative memory rigorous.
  • Hybrid architecture (LLM) · Model designs that interleave self-attention with state-space, recurrent, or linear-attention layers, or mix sliding-window and full attention, to cut KV-cache and compute costs.
  • Layer normalization · Per-token normalization of activations across the feature dimension, introduced by Ba, Kiros, and Hinton in 2016 and built into every Transformer block as part of the add-and-norm pattern.
  • LSTM · Long short-term memory: the 1997 recurrent architecture that dominated sequence modeling for two decades before the Transformer.
  • Mixture of experts · Sparse architecture in which a router activates only a few expert subnetworks per token, decoupling parameter count from per-token compute.
  • Multi-head attention · Running attention in parallel subspaces within each layer; its head-sharing variants (MQA, GQA, MLA) set the size of the KV cache and much of the cost of inference.
  • Multi-head latent attention · Attention variant that caches a low-rank latent compression of keys and values instead of full per-head tensors; introduced in DeepSeek-V2 (2024).
  • PagedAttention · Virtual-memory-style paging for the KV cache, introduced with vLLM in 2023; cut serving memory waste to near zero and became standard across inference stacks.
  • Perceptron · Frank Rosenblatt's 1957 learning machine: the first artificial neural network shown to learn its weights from data, with a provably convergent rule and a famous linear-separability limitation.
  • Positional encoding · The mechanism that injects token order into the otherwise permutation-invariant Transformer, evolving from fixed sinusoidal vectors to the rotary embeddings used by most modern LLMs.
  • RMSNorm · Root mean square layer normalization: a simplification of layer normalization that rescales activations by their root mean square alone, the default normalizer of modern open-weight LLMs.
  • RWKV · Community-built linear-attention RNN trained like a Transformer; the open ecosystem's longest-running alternative-architecture project.
  • Self-attention · Mechanism by which each position in a sequence computes its representation as an attention-weighted sum over all positions; the core operation of the Transformer.
  • Seq2seq · Sequence-to-sequence learning: the 2014 encoder-decoder paradigm that mapped variable-length input to output through a fixed context vector, the direct ancestor of the attention era.
  • Sliding-window attention · Restricting each token's attention to a fixed-size local window, cutting attention cost from quadratic to linear in context length and bounding KV-cache memory.
  • Softmax · The function that turns a vector of raw scores into a probability distribution, used in attention weighting, vocabulary output layers, and mixture-of-experts routing.
  • Sparse attention · Attention variants in which each token attends to a selected subset of positions instead of the full sequence, cutting the quadratic cost of self-attention.
  • State-space model · Sequence architecture family (S4, Mamba) that replaces attention with a fixed-size recurrent state, giving linear-time processing and constant-memory generation.
  • The Illustrated Transformer · Jay Alammar's June 2018 visual walkthrough of the Transformer architecture, which became the standard first explanation used in university courses worldwide.
  • The Unreasonable Effectiveness of Recurrent Neural Networks · Andrej Karpathy's May 2015 tutorial-essay whose character-level LSTM experiments gave a generation of researchers their first look at neural text generation.
  • Transformer (architecture) · Neural network architecture based on self-attention, introduced by Vaswani et al. in 2017; the basis of virtually all modern large language models.
  • Vision Transformer · Transformer applied to images by treating fixed-size patches as tokens; introduced by Dosovitskiy et al. in 2020, now the standard vision encoder in multimodal LLMs.