Self-attention (also intra-attention) is the mechanism by which each position in a sequence computes its new representation as a weighted combination of the representations at every position of the same sequence, with the weights produced by a learned compatibility function between positions. It is the defining operation of the Transformer and, through it, of essentially every modern large language model.1 Because each layer connects all pairs of positions directly, self-attention gives a model a content-addressable working memory over its input, at a compute and memory cost that grows quadratically with sequence length. That quadratic cost, together with the growth of the key-value cache during generation, shapes most of the engineering around modern inference, from FlashAttention kernels and PagedAttention memory management to architectural revisions such as sliding-window, sparse, and latent attention. The mechanism predates the Transformer: attention was introduced for machine translation in 2014, and attention within a single sequence appeared in machine-reading and inference models during 2016 and 2017.24

Role in the network

Self-attention is the sublayer of a Transformer block that mixes information across positions. Each block pairs one attention sublayer with a position-wise feed-forward network, which transforms every position in isolation, and like every sublayer the attention output sits inside a residual connection followed by normalization.1

Neural sequence models before attention compressed a whole input into fixed-size state. In the recurrent encoder-decoder architectures used for machine translation around 2014, the encoder read the source sentence into a single vector from which the decoder had to reconstruct everything, a bottleneck that degraded visibly on long sentences. Bahdanau, Cho, and Bengio removed it by letting the decoder compute, at every output step, a soft alignment over all encoder states and read from them directly, the first influential neural attention mechanism.2 Attention here still connected two different sequences; the recurrent networks underneath retained their structural problems. Information relating two distant positions in the same sequence had to traverse a chain of recurrent steps proportional to their distance, gradients attenuated along that chain, and the computation was inherently serial, position by position, which wasted the parallelism of modern accelerators.

Self-attention addresses both problems at once. Any two positions interact through a single learned step regardless of their distance, so the maximum path length between dependencies is constant rather than linear, and the entire layer reduces to dense matrix multiplications that parallelize across the whole sequence during training. Vaswani et al. gave exactly this argument, comparing per-layer cost, sequential operations, and path length against recurrence and convolution.1 The training parallelism is what made it economical to fit large Transformers on web-scale corpora, the regime later characterized by scaling laws; the constant path length is why Transformers pick up long-range structure, from coreference to code scope, that recurrent models handled unreliably.

The mathematics

Each input vector is linearly projected into three roles: a query, a key, and a value. With \(X \in \mathbb{R}^{n \times d_{\text{model}}}\) the matrix whose rows are the \(n\) token representations at one layer, the projections are three learned matrices:

$$Q = X W^{Q}, \qquad K = X W^{K}, \qquad V = X W^{V}$$
Symbol Shape Meaning
\(X\) \(n \times d_{\text{model}}\) input representations, one row per position
\(W^{Q}, W^{K}\) \(d_{\text{model}} \times d_k\) learned query and key projections
\(W^{V}\) \(d_{\text{model}} \times d_v\) learned value projection
\(Q, K\) \(n \times d_k\) per-position queries and keys
\(V\) \(n \times d_v\) per-position values

Every pair of positions then receives a raw compatibility score, the dot product of one position's query with the other's key, divided by the square root of the key dimension:

$$S = \frac{Q K^{\top}}{\sqrt{d_k}}$$

with \(S \in \mathbb{R}^{n \times n}\). The scaling term keeps the dot products from growing with dimension and saturating the softmax, which is why the operation is called scaled dot-product attention.1

In decoder-style models such as the GPT family, a causal mask restricts each position to attend only to earlier ones by overwriting the upper triangle of the score matrix before normalization:

$$S_{ij} \leftarrow -\infty \quad \text{for } j > i$$

which sets the weight of every future position to zero; this is what permits autoregressive generation and next-token training.

Each row of the (possibly masked) score matrix is then turned into a probability distribution, and the output is the corresponding weighted sum of the value rows:

$$A = \operatorname{softmax}(S), \qquad O = A V$$
Symbol Shape Meaning
\(S\) \(n \times n\) scaled (and possibly masked) compatibility scores
\(A\) \(n \times n\) attention weights; softmax applied row-wise, so each row sums to 1
\(O\) \(n \times d_v\) output representations, one row per position

In words, the attention weight between positions \(i\) and \(j\) is the scaled dot product of query \(i\) with key \(j\), normalized with a softmax across all \(j\), and position \(i\)'s output is the weighted sum of the values.1 In practice the computation runs as multi-head attention: several attention operations execute in parallel over separate, lower-dimensional learned projections, and the concatenated head outputs are projected back to the model dimension, letting different heads specialize in different relations.

The layer's cost splits into two terms. Forming \(S\) costs \(n^2 d_k\) multiply-adds per head and forming \(AV\) the same, so with the head dimensions summing to \(d_{\text{model}}\) the attention proper grows quadratically in sequence length while the four projections grow only linearly in it:

$$T_{\text{attn}}(n) = \Theta\!\left(n^{2}\, d_{\text{model}}\right) \quad\text{versus}\quad T_{\text{proj}}(n) = \Theta\!\left(n\, d_{\text{model}}^{2}\right)$$

with the quadratic term overtaking the linear one once \(n\) reaches roughly \(2\, d_{\text{model}}\). In training, computing the attention matrix for a sequence of length \(n\) therefore takes time and memory proportional to \(n\) squared, per layer and head.1

In autoregressive inference, each generated token needs attention over everything before it; rather than recompute, implementations cache the keys and values of all previous tokens (the KV cache), making each new token's attention linear in context length. The cache of one sequence occupies

$$M_{\text{KV}} = 2 \times L \times h_{\text{kv}} \times d_{\text{head}} \times n \times b$$
Symbol Shape Meaning
\(L\) scalar number of layers
\(h_{\text{kv}}\) scalar key-value heads per layer, reduced by head-sharing schemes
\(d_{\text{head}}\) scalar per-head key and value dimension
\(n\) scalar cached tokens (context length so far)
\(b\) scalar bytes per element; 2 at fp16

The leading 2 counts keys and values. A Llama 3 70B-class configuration (\(L = 80\), \(h_{\text{kv}} = 8\), \(d_{\text{head}} = 128\)) at fp16 occupies \(2 \times 80 \times 8 \times 128 \times 2 = 327{,}680\) bytes per token, about 320 KiB, so a full 128,000-token context needs roughly 42 GB of cache per sequence.24

Worked example: three tokens through a masked layer

Take a three-token sequence with \(d_{\text{model}} = d_k = d_v = 2\) and all three projections equal to the identity, so \(Q = K = V = X\), with

$$X = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{pmatrix}$$

The unscaled score matrix is \(Q K^{\top} = X X^{\top}\), and dividing through by \(\sqrt{d_k} = \sqrt{2} \approx 1.414\):

$$QK^{\top} = \begin{pmatrix} 1 & 0 & 1 \\ 0 & 1 & 1 \\ 1 & 1 & 2 \end{pmatrix} \qquad\Longrightarrow\qquad S \approx \begin{pmatrix} 0.707 & 0 & 0.707 \\ 0 & 0.707 & 0.707 \\ 0.707 & 0.707 & 1.414 \end{pmatrix}$$

The causal mask overwrites every entry above the diagonal with \(-\infty\), so row 1 keeps only its first entry, row 2 its first two, and row 3 all three. Carry the third query row \((0.707,\, 0.707,\, 1.414)\) through the softmax: the first two entries exponentiate to \(\exp(0.707) \approx 2.028\) each, the third to \(\exp(1.414) \approx 4.113\), the normalizer is \(2.028 + 2.028 + 4.113 = 8.169\), and the weights are \((2.028/8.169,\ 2.028/8.169,\ 4.113/8.169) \approx (0.248,\ 0.248,\ 0.503)\). Position 3's output is the weighted sum of the three value rows:

$$O_3 = 0.248\,(1, 0) + 0.248\,(0, 1) + 0.503\,(1, 1) \approx (0.752,\ 0.752)$$

About half of position 3's new representation comes from its own value, a quarter each from the two earlier tokens. The remaining rows are degenerate by comparison: row 1 concentrates entirely on token 1, giving \(O_1 = (1, 0)\), and row 2's weights over tokens 1 and 2 are \((1/3.028,\ 2.028/3.028) \approx (0.330,\ 0.670)\), giving \(O_2 \approx (0.330,\ 0.670)\). This pipeline, projections, scaled scores, masked softmax, weighted sum, is walked through visually in Alammar's Illustrated Transformer.25

Intuition

Informally, the query expresses what a position is looking for, the key advertises what a position contains, and the value is the information it passes along if selected. The dot product \(Q_i \cdot K_j\) is large exactly when what position \(i\) is looking for matches what position \(j\) advertises, so attention retrieves by content rather than by location: the same learned projections can implement coreference, agreement, or bracket matching with no built-in receptive field, which is the sense in which the layer acts as a content-addressable memory.

Self-attention by itself is permutation-equivariant: it treats the input as a set, so word order must be injected separately. The original Transformer added fixed sinusoidal or learned absolute position embeddings to the inputs.1 Later models moved to relative schemes. Rotary position embedding (RoPE) rotates each query and key by an angle proportional to its position so that their dot product depends only on the relative offset between tokens, a formulation that has become the default in open-weight models such as the Llama line;5 ALiBi instead biases attention scores by a linear penalty on distance, which improves extrapolation to sequences longer than those seen in training.6 The choice of positional scheme now interacts directly with long-context engineering, since methods that rescale or interpolate RoPE frequencies can stretch a pretrained model's usable context with modest fine-tuning.

Variants and extensions

Most variation in deployed models concerns how many key-value heads exist and which positions are attended to, and much of it exists chiefly to shrink the KV cache. Multi-query attention shares a single key and value head across all query heads, shrinking the KV cache by the head count at some quality cost;9 grouped-query attention interpolates between the two extremes and became the mainstream compromise, used in Llama 2's larger variants, Llama 3.1, and Mistral 7B.10 Multi-head latent attention, introduced with DeepSeek-V2 in 2024, instead compresses keys and values into a low-rank latent vector that is expanded on the fly, cutting the cache far below grouped-query levels while reportedly preserving quality.11

A second family restricts the attention pattern. Sparse attention in the sense of Child et al. factorizes full attention into strided and local components so that cost grows as \(n\) times the square root of \(n\);12 Longformer and BigBird combined local windows with a few global tokens for long documents.1323 Sliding-window attention limits every token to a fixed-size window of recent positions, relying on layer stacking to propagate information further, and was used in Mistral 7B and, interleaved with dense layers, in OpenAI's gpt-oss according to its model card. Related is the Attention sink phenomenon: softmax attention tends to park probability mass on the first few tokens, and retaining those sink tokens alongside a rolling window lets models stream over effectively unbounded input without cache growth.14

More radical proposals replace the softmax entirely. Linear attention rewrites the operation with kernel feature maps so that it runs in linear time and admits a recurrent form,15 historically at a quality cost for retrieval-heavy tasks; modern descendants appear mostly inside hybrid architectures that interleave a few full-attention layers with linear or state-space layers such as Mamba,16 keeping exact retrieval ability while paying quadratic cost only occasionally. Systems work extends rather than replaces the mechanism: ring attention shards the sequence dimension across devices, enabling contexts in the millions of tokens, and quantization of the KV cache trades precision for capacity. Cross-attention, finally, is the same arithmetic with queries drawn from one sequence and keys and values from another; it connects encoder to decoder in models like T5 and injects text conditioning into image generators.

Adoption

Self-attention is used, in some configuration, by essentially every Transformer descendant in deployment. Causal self-attention underlies the GPT, Claude, and Gemini lines and their open-weight counterparts; bidirectional self-attention underlies encoder models such as BERT; encoder-decoder models such as T5 and Whisper combine self-attention in each stack with cross-attention between them. Outside text, the Vision Transformer applies unmodified self-attention to sequences of image patches,17 CLIP uses Transformer towers for both modalities, and Stable Diffusion runs self-attention over spatial positions inside its denoising network. Diffusion language models depend on bidirectional self-attention over the whole canvas, since every position may be revised at every step. Mixture-of-experts models such as Mixtral 8x7B route only the feed-forward blocks; their attention layers remain dense and shared.

The variants map onto live systems directly. Grouped-query attention ships in Llama 3.1 and most open-weight models of its generation; sliding-window attention in Mistral 7B and gpt-oss; multi-head latent attention in DeepSeek-V2, DeepSeek-V3, and reportedly Kimi K2; MiniMax-Text-01 interleaves linear "lightning" attention with periodic softmax layers to reach a million-token context, as does the broader hybrid wave. Gemini 1.5 demonstrated production million-token contexts in 2024. On the serving side, FlashAttention-style kernels and PagedAttention-based engines such as vLLM are close to universal in open-model deployment.78 Benchmark performance on knowledge suites such as MMLU depends in part on what attention layers learn to retrieve from context.

Limitations and critiques

The quadratic barrier remains the defining limitation. Training cost grows with the square of context length, and although the KV cache makes generation linear per token, the cache grows with context length, batch size, layer count, and head count, capping concurrency in serving and pushing long-context inference toward specialized attention variants; at long contexts it, not the model weights, dominates accelerator memory. This makes the KV cache the central object of inference economics: PagedAttention manages it in non-contiguous virtual-memory-style blocks to eliminate fragmentation in serving,8 and FlashAttention attacks the training-side cost from the systems direction, computing exact attention in tiles that never materialize the full \(n\)-by-\(n\) matrix in slow GPU memory.7 Efficiency aside, models do not use long contexts uniformly: Liu et al. documented the "lost in the middle" effect, in which retrieval accuracy is high for information near the beginning or end of the context and drops for the middle,18 and length generalization beyond the training window remains fragile, entangled with the positional scheme. The attention-sink behavior is itself a critique target, an artifact of forcing softmax weights to sum to one even when no token deserves attention.14

Interpretability claims about attention are contested. Early work read attention maps as explanations of model behavior; Jain and Wallace argued that attention weights often fail standard tests for explanation, since alternative weightings can yield identical predictions,19 and the subsequent literature settled on treating raw attention maps as suggestive rather than faithful. Finally, the mechanism's dominance is periodically challenged on efficiency grounds by state-space models and linear-attention recurrences, which offer constant-memory inference; as of 2026 these have displaced full attention only partially, largely because exact softmax attention remains stronger on precise long-range retrieval, and the field's compromise position is the hybrid architecture.

History

Attention entered neural sequence modeling through machine translation. Bahdanau, Cho, and Bengio's alignment mechanism, posted in September 2014 from Jacobs University Bremen and the University of Montreal, let a decoder attend over encoder states and became the template for neural attention generally.2 Luong, Pham, and Manning at Stanford simplified and systematized the design in 2015 with global and local variants.3 The step to attention within a single sequence came in January 2016, when Cheng, Dong, and Lapata at the University of Edinburgh used what they called intra-attention to let a memory-augmented LSTM relate each word to its predecessors during reading.4 In June 2016, Parikh and colleagues at Google showed that a lightweight model built almost entirely from attention, with optional intra-sentence attention and no recurrence over word order, matched far heavier recurrent models on natural language inference.20 Lin et al. published a structured self-attentive sentence embedding in March 2017, extracting interpretable sentence representations with multiple attention heads over an LSTM.21

The consolidating event was "Attention Is All You Need," posted in June 2017 by Vaswani and seven co-authors at Google Brain and Google Research and presented at NeurIPS that year. It discarded recurrence and convolution entirely, introduced the scaled dot-product formulation and multi-head attention, and demonstrated state-of-the-art translation at a fraction of prior training cost.1 Within eighteen months self-attention was the substrate of both major pretraining paradigms, causal in GPT-2's line and bidirectional in BERT. The years 2019 through 2021 produced the efficient-attention wave (Sparse Transformers, Reformer, Longformer, Performer, and many others)1213 along with rotary embeddings;5 GPT-3 itself interleaved dense and locally banded sparse layers.22 The systems era followed: FlashAttention from Stanford in 2022,7 PagedAttention and vLLM from Berkeley in 2023,8 and attention-sink streaming the same year.14 Architectural revision of the mechanism's memory footprint continued with multi-head latent attention at DeepSeek in 2024,11 while Mamba-style state-space models from late 2023 onward16 reopened the question of whether full self-attention is necessary at all. As of 2026 it remains, in original or revised form, the core operation of nearly every frontier model.

See also

References


  1. Vaswani, A., et al., "Attention Is All You Need," arXiv:1706.03762, June 2017. ↩↩↩↩↩↩↩↩

  2. Bahdanau, D., Cho, K., and Bengio, Y., "Neural Machine Translation by Jointly Learning to Align and Translate," arXiv:1409.0473, September 2014. ↩↩↩

  3. Luong, M.-T., Pham, H., and Manning, C., "Effective Approaches to Attention-based Neural Machine Translation," arXiv:1508.04025, August 2015. ↩

  4. Cheng, J., Dong, L., and Lapata, M., "Long Short-Term Memory-Networks for Machine Reading," arXiv:1601.06733, January 2016. ↩↩

  5. Su, J., et al., "RoFormer: Enhanced Transformer with Rotary Position Embedding," arXiv:2104.09864, April 2021. ↩↩

  6. Press, O., Smith, N., and Lewis, M., "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation," arXiv:2108.12409, August 2021. ↩

  7. Dao, T., et al., "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness," arXiv:2205.14135, May 2022. ↩↩↩

  8. Kwon, W., et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention," arXiv:2309.06180, September 2023. ↩↩↩

  9. Shazeer, N., "Fast Transformer Decoding: One Write-Head is All You Need," arXiv:1911.02150, November 2019. ↩

  10. Ainslie, J., et al., "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints," arXiv:2305.13245, May 2023. ↩

  11. DeepSeek-AI, "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model," arXiv:2405.04434, May 2024. ↩↩

  12. Child, R., et al., "Generating Long Sequences with Sparse Transformers," arXiv:1904.10509, April 2019. ↩↩

  13. Beltagy, I., Peters, M., and Cohan, A., "Longformer: The Long-Document Transformer," arXiv:2004.05150, April 2020. ↩↩

  14. Xiao, G., et al., "Efficient Streaming Language Models with Attention Sinks," arXiv:2309.17453, September 2023. ↩↩↩

  15. Katharopoulos, A., et al., "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention," arXiv:2006.16236, June 2020. ↩

  16. Gu, A., and Dao, T., "Mamba: Linear-Time Sequence Modeling with Selective State Spaces," arXiv:2312.00752, December 2023. ↩↩

  17. Dosovitskiy, A., et al., "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale," arXiv:2010.11929, October 2020. ↩

  18. Liu, N., et al., "Lost in the Middle: How Language Models Use Long Contexts," arXiv:2307.03172, July 2023. ↩

  19. Jain, S., and Wallace, B., "Attention is not Explanation," arXiv:1902.10186, February 2019. ↩

  20. Parikh, A., Täckström, O., Das, D., and Uszkoreit, J., "A Decomposable Attention Model for Natural Language Inference," arXiv:1606.01933, June 2016. ↩

  21. Lin, Z., et al., "A Structured Self-attentive Sentence Embedding," arXiv:1703.03130, March 2017. ↩

  22. Brown, T., et al., "Language Models are Few-Shot Learners," arXiv:2005.14165, May 2020. ↩

  23. Zaheer, M., et al., "Big Bird: Transformers for Longer Sequences," arXiv:2007.14062, July 2020. ↩

  24. Llama Team, Meta AI, "The Llama 3 Herd of Models," arXiv:2407.21783, July 2024. ↩

  25. Alammar, J., "The Illustrated Transformer", June 2018. ↩