RMSNorm (root mean square layer normalization) is a normalization technique, a variant of layer normalization that rescales each token's activation vector by the root mean square of its components without re-centering them around their mean. Introduced by Zhang and Sennrich in 2019, it matched layer normalization's training stability in the authors' experiments while cutting the overhead of the normalization step, and it has since become the standard normalizer of open-weight large language models.1

Role in the network

RMSNorm occupies exactly the slot layer normalization does in a Transformer block. In the pre-norm arrangement used by modern stacks, an RMSNorm layer precedes each attention and feed-forward sublayer, re-standardizing the branch input while the residual stream passes untouched, and a final RMSNorm precedes the output projection.2 The problem it solves is the same as layer normalization's: as residual additions accumulate through dozens of layers, the scale of the activations drifts, and re-standardizing each token's vector keeps deep training stable. RMSNorm's contribution is to do that job with less work, dropping the re-centering half of the operation that the authors argued contributes little.1 Like layer normalization, the operation acts independently per token position, so it behaves identically at any batch size and requires no statistics collected across examples, which suits autoregressive inference.

The mathematics

For one token's activation vector \(x \in \mathbb{R}^{d}\), with \(d\) the model dimension, RMSNorm divides the vector by its root mean square and applies a learned per-feature scale:

$$\mathrm{RMS}(x) = \sqrt{\frac{1}{d} \sum_{i=1}^{d} x_i^{2} + \epsilon}, \qquad y = \frac{x}{\mathrm{RMS}(x)} \odot \gamma$$
Symbol Shape Meaning
\(x\) \(d\) activation vector at one token position
\(\mathrm{RMS}(x)\) scalar quadratic mean of the \(d\) features
\(\epsilon\) scalar small constant for numerical stability
\(\gamma\) \(d\) learned per-feature scale
\(y\) \(d\) normalized output

The contrast with full layer normalization, which computes \(y = (x - \mu)/\sqrt{\sigma^2 + \epsilon} \odot \gamma + \beta\) from the feature mean \(\mu\) and variance \(\sigma^2\), is two missing pieces: no mean subtraction, and no learned shift \(\beta\).14 The accounting is direct. Layer normalization needs two reductions over the feature dimension (mean, then variance) and four elementwise operations (subtract, divide, scale, shift); RMSNorm needs one reduction (mean of squares) and two elementwise operations (divide, scale), and it carries half the learned parameters. The authors reported equal model quality with measurably cheaper computation across translation and language modeling benchmarks.1

Worked example: rescaling one token

Take \(d = 4\) and \(x = (2, 4, 4, 6)\), with \(\epsilon\) negligible. The squares are \((4, 16, 16, 36)\), their mean is \(72/4 = 18\), and the root mean square is \(\sqrt{18} \approx 4.2426\). Dividing component-wise:

$$y = \frac{x}{\mathrm{RMS}(x)} = \left( \frac{2}{4.2426},\ \frac{4}{4.2426},\ \frac{4}{4.2426},\ \frac{6}{4.2426} \right) \approx (0.4714,\ 0.9428,\ 0.9428,\ 1.4142)$$

The check is the quadratic mean of the output: the squares are \(0.2222\), \(0.8889\), \(0.8889\), and \(2.0000\), summing to \(4.0\), so the mean square is \(4.0/4 = 1\) and the output's RMS is exactly \(1\), which is what the normalization promises. Note the contrast with layer normalization: the output's mean is \(3.7712/4 \approx 0.9428\), not zero. RMSNorm leaves the offset where it found it and fixes only the scale.

Intuition

Zhang and Sennrich's premise was that the benefit of layer normalization comes from rescaling, not re-centering. Their invariance analysis makes the distinction precise: full layer normalization makes its output invariant to shifting and scaling of the input, while RMSNorm keeps only scale invariance, since multiplying the input by a constant multiplies the RMS by the same constant and the two cancel; a constant added to every feature passes straight through.1 In a deep residual stream, the practical pathology is scale drift, and resetting the quadratic mean to one at every branch input is enough to hold the stack together. The mean of a token's features is a readable signal the following sublayer may as well keep, and the authors' experiments supported leaving it alone: parity with layer normalization on machine translation and language modeling at lower cost.1

Statistically, the operation normalizes the second moment of the features rather than their variance, which is why one pass over the vector replaces two. Geometrically, every token is projected onto a sphere of radius \(\sqrt{d}\) before the learned scale \(\gamma\) reshapes it.

Variants and extensions

The same paper proposed a partial variant, pRMSNorm, which estimates the root mean square from only a subset of the features and rescales by that estimate, reducing the normalization's cost further while reporting comparable accuracy; the normalized fraction is a hyperparameter of the layer.1 RMSNorm is also the special case of layer normalization with the mean fixed to zero and the shift \(\beta\) fixed to zero, so implementations typically share code with the full operation.4 A closely related scale-only normalization shipped in T5 the same month (October 2019), whose text-to-text recipe rescales activations without re-centering them and applies no additive bias, evidence that the simplification was in the air.5

Adoption

RMSNorm appeared in DeepMind's Gopher in 2021,3 but its wider adoption dates to Meta's LLaMA in 2023, whose recipe of pre-normalization with RMSNorm, SwiGLU feed-forward layers, and rotary position embeddings was copied by Mistral 7B, Gemma, the Qwen and DeepSeek lines, and most subsequent open-weight Transformers.2 In these stacks an RMSNorm layer precedes each attention and feed-forward sublayer (the pre-norm arrangement), and a final RMSNorm precedes the output projection. Full layer normalization remains common chiefly in older and encoder-style models such as BERT.

Limitations and critiques

The simplification has structural consequences. Because nothing subtracts the mean, any common-mode offset shared by a token's features passes through untouched, and the operation gives up the shift invariance that full layer normalization guarantees.1 The gains are also efficiency-side only: the reported result is quality parity, not improvement, so where the normalization's cost is irrelevant, full layer normalization remains a defensible choice.14 And since normalization is a small, memory-bandwidth-bound share of a block's compute next to its large matrix multiplications, the measured speedups, while real, are bounded by how little of the step the normalizer was in the first place.

History

Layer normalization dates to Ba, Kiros, and Hinton in July 2016 and became a fixture of the Transformer in 2017.4 Zhang and Sennrich published RMSNorm in October 2019, motivated by the hypothesis that layer normalization's re-centering contributes little to its stabilizing effect, and validated the simplification on neural machine translation and language modeling.1 The T5 recipe shipped a closely related scale-only normalization within weeks, independently.5 DeepMind's Gopher adopted RMSNorm at scale in 2021,3 and LLaMA's February 2023 release fixed it as the default of the open-weight ecosystem, a position it still held as of mid-2026.2

See also

References


  1. Zhang, B., and Sennrich, R., "Root Mean Square Layer Normalization," arXiv:1910.07467, October 2019. ↩↩↩↩↩↩↩↩↩↩

  2. Touvron, H., et al., "LLaMA: Open and Efficient Foundation Language Models," arXiv:2302.13971, February 2023. ↩↩↩

  3. Rae, J. W., et al., "Scaling Language Models: Methods, Analysis and Insights from Training Gopher," arXiv:2112.11446, December 2021. ↩↩

  4. Ba, L. J., Kiros, J. R., and Hinton, G. E., "Layer Normalization," arXiv:1607.06450, July 2016. ↩↩↩↩

  5. Raffel, C., et al., "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer," arXiv:1910.10683, October 2019. ↩↩