Category: Training methods

This category contains 23 pages.

  • Adam (optimizer) · The adaptive optimization algorithm introduced by Kingma and Ba in 2014, combining momentum with per-parameter learning rates, used to train nearly every modern language model.
  • Analytic distillation · The extraction of what a trained model knows into smaller, cheaper, or more explicit form, spanning classic knowledge distillation, rationale distillation, and mechanistic interpretability.
  • Backpropagation · The chain-rule algorithm that computes a neural network's loss gradient in a single backward pass, developed by Werbos in the early 1970s and popularized by Rumelhart, Hinton, and Williams in 1986.
  • Compositional generalization · The capacity to understand and produce novel combinations of known components, the systematicity property long contested in neural networks and a central open question for large language models.
  • Cross-entropy loss · The standard training loss of language modeling: the negative log probability assigned to the correct token, the quantity plotted on every scaling-law curve.
  • Direct preference optimization · Preference-alignment method that optimizes a policy directly on comparison data via a closed-form objective, skipping the reward model and RL loop.
  • Distributed training · Techniques for training models across thousands of accelerators: data, tensor, pipeline, and expert parallelism, plus sharding schemes like ZeRO/FSDP.
  • Dropout · The regularization technique that randomly zeroes hidden units during training, standard in early Transformers and largely retired as pretraining shifted to single-epoch, web-scale data.
  • FlashAttention · IO-aware exact attention algorithm that tiles the computation to fit in GPU on-chip memory; the de facto standard attention kernel in training and serving stacks.
  • Instruction tuning · Supervised fine-tuning of pretrained language models on instruction-response pairs, teaching them to follow natural-language tasks zero-shot.
  • Knowledge distillation · Training a smaller student model to imitate a larger teacher; the mechanism behind small frontier-adjacent models and a persistent IP controversy.
  • Masked language modeling · The pretraining objective of predicting hidden tokens from bidirectional context, introduced at scale by BERT in 2018 and the training signal of the encoder-only lineage.
  • Mixture of experts · Sparse architecture in which a router activates only a few expert subnetworks per token, decoupling parameter count from per-token compute.
  • Next-token prediction · The objective of predicting each token from all preceding ones, the training signal behind GPT-style language models and the foundation of autoregressive generation.
  • Pretraining · The first phase of language model development: self-supervised training over large text corpora that produces a base model, before post-training adapts it for use.
  • Reinforcement learning from human feedback · Post-training method that optimizes a language model against a reward model trained on human preference comparisons; the technique behind ChatGPT-style assistants.
  • Reinforcement learning with verifiable rewards · Post-training with programmatic reward signals (test suites, answer checkers) instead of learned reward models; the engine of the reasoning-model era.
  • Scaling laws · Empirical power-law relationships between model performance and parameters, data, and compute; the planning instrument of the LLM era.
  • Supervised fine-tuning · Post-training method that teaches a pretrained language model desired behavior by supervised next-token training on curated prompt-response demonstrations.
  • Teacher forcing · The training protocol that feeds a sequence model the true previous tokens instead of its own predictions, enabling fully parallel next-token training at the cost of exposure bias.
  • Tokenization · The conversion of raw text into the integer token ids that language models read and write, dominated by byte-pair encoding variants with vocabularies of tens to hundreds of thousands of tokens.
  • ULMFiT · Howard and Ruder's 2018 three-stage transfer-learning recipe for text, a direct precursor to the pretrain-and-fine-tune paradigm of BERT and GPT.
  • Word2vec · The 2013 word-embedding method whose vector arithmetic ('king - man + woman = queen') made distributed representations the foundation of neural NLP.