Category: Alignment and safety

This category contains 14 pages.

• Activation steering · Adding vectors to a transformer's residual stream at inference time to shift model behavior; used both as a control knob and as causal evidence in interpretability.
• Activation verbalizer · Anthropic technique in which a language model is trained to translate another model's token-level activations into natural-language summaries.
• Circuits (interpretability) · Subnetworks of a neural network that implement identifiable algorithms; the central object of study in mechanistic interpretability.
• Constitutional AI · Anthropic's post-training method: models critique and revise their own outputs against written principles, then train on AI-generated preferences.
• Eliciting Latent Knowledge · Open alignment problem posed by ARC in 2021: training a model to report what it internally knows rather than what looks correct to human overseers.
• InstructGPT · OpenAI's 2022 RLHF-aligned GPT-3 variants; demonstrated that alignment training beats scale on human preference and set the template for ChatGPT.
• Logit lens · Decoding a transformer's intermediate residual-stream states through its output unembedding, revealing layer-by-layer refinement of next-token predictions.
• Mechanistic interpretability · The research program of reverse-engineering neural networks into human-understandable features and circuits, from the 2020 Distill circuits thread to routine white-box analyses in frontier system cards.
• Reinforcement learning from human feedback · Post-training method that optimizes a language model against a reward model trained on human preference comparisons; the technique behind ChatGPT-style assistants.
• Simulators (essay) · janus's September 2022 essay proposing that language models are best understood not as agents or oracles but as simulators that instantiate agentic and non-agentic simulacra.
• Sparse autoencoder · Dictionary learning on language model activations; decomposes superposed representations into sparse, often monosemantic features.
• Superposition (interpretability) · The hypothesis that neural networks represent more features than they have neurons by packing them into overlapping directions, explaining polysemantic neurons and motivating sparse autoencoders.
• There's No Fire Alarm for Artificial General Intelligence · Eliezer Yudkowsky's October 2017 essay arguing that no clear, socially agreed signal of imminent AGI should be expected before the end, so waiting for one is a plan to wait forever.
• What Failure Looks Like · Paul Christiano's March 2019 essay replacing the caricature of sudden AI takeover with two realistic failure modes: slow proxy collapse and influence-seeking systems.