Category: Inference

This category contains 13 pages.

  • Activation steering · Adding vectors to a transformer's residual stream at inference time to shift model behavior; used both as a control knob and as causal evidence in interpretability.
  • Attention sink · The tendency of autoregressive transformers to concentrate attention mass on initial tokens; exploited by StreamingLLM and built into gpt-oss as learned biases.
  • Chain-of-thought prompting · Eliciting step-by-step reasoning before the answer; discovered as a prompting trick in 2022, later internalized by trained reasoning models.
  • Cursor · AI code editor built by Anysphere, one of the largest commercial deployers of frontier coding models and, since late 2025, trainer of its own Composer coding model.
  • Diffusion language model · Text generation by iterative parallel denoising of masked or corrupted token sequences, an alternative to left-to-right autoregression.
  • FlashAttention · IO-aware exact attention algorithm that tiles the computation to fit in GPU on-chip memory; the de facto standard attention kernel in training and serving stacks.
  • In-context learning · The ability of large language models to perform a new task from instructions or examples in the prompt alone, without any weight update, demonstrated at scale by GPT-3 in 2020.
  • Multi-head attention · Running attention in parallel subspaces within each layer; its head-sharing variants (MQA, GQA, MLA) set the size of the KV cache and much of the cost of inference.
  • Multi-head latent attention · Attention variant that caches a low-rank latent compression of keys and values instead of full per-head tensors; introduced in DeepSeek-V2 (2024).
  • PagedAttention · Virtual-memory-style paging for the KV cache, introduced with vLLM in 2023; cut serving memory waste to near zero and became standard across inference stacks.
  • Sliding-window attention · Restricting each token's attention to a fixed-size local window, cutting attention cost from quadratic to linear in context length and bounding KV-cache memory.
  • Sparse attention · Attention variants in which each token attends to a selected subset of positions instead of the full sequence, cutting the quadratic cost of self-attention.
  • Test-time compute · Improving answers by spending more computation at inference (longer reasoning chains, search, sampling) rather than more pretraining; the axis behind reasoning models.