Root Mean Square Layer Normalization

What LayerNorm actually computes

LayerNorm standardizes a vector across its feature dimension and then applies a learned affine transform, so for an input $x \in \mathbb{R}^d$ it first measures the mean and standard deviation of the $d$ coordinates.

$$\mu = \frac{1}{d}\sum_{i=1}^{d} x_i, \qquad \sigma = \sqrt{\frac{1}{d}\sum_{i=1}^{d} (x_i - \mu)^2}$$

It then re-centers by subtracting $\mu$, re-scales by dividing by $\sigma$, and finally lets the model stretch and shift each coordinate through learned parameters $\gamma$ (scale, initialized to $1$) and $\beta$ (shift, initialized to $0$).

$$\bar y_i = \frac{x_i - \mu}{\sigma}\,\gamma_i + \beta_i$$

This is two statistics, two element-wise passes, and two parameter vectors per normalization site, and a deep transformer has hundreds of these sites.

The hypothesis: only the scaling matters

The paper's claim is that LayerNorm's benefit comes almost entirely from the re-scaling — dividing by $\sigma$ to keep activation magnitudes stable across depth — and very little from the re-centering that subtracts $\mu$. The intuition is that any mean shift the network actually needs can be reabsorbed downstream: the bias terms of the following linear layers can add back whatever constant offset re-centering would have removed, so spending compute to subtract $\mu$ at every norm is largely redundant.

Dropping the mean

RMSNorm acts on that hypothesis by replacing the standard deviation with the root mean square of the raw coordinates, which never subtracts a mean.

$$\operatorname{RMS}(x) = \sqrt{\frac{1}{d}\sum_{i=1}^{d} x_i^2}$$

Normalization then divides by this quantity and applies only the scale parameter, with no shift.

$$\bar y_i = \frac{x_i}{\operatorname{RMS}(x)}\,\gamma_i$$

Compared to the LayerNorm equations above, two things are gone: the mean $\mu$ is never formed or subtracted, and the shift vector $\beta$ no longer exists.

What removing them saves

The savings are concrete. Dropping mean-centering removes one reduction over the feature dimension and one element-wise subtraction, and dropping $\beta$ removes one learned vector per norm along with the add that applies it. The net is roughly 15% fewer operations at each normalization site, and because these sites are on the critical path of every forward and backward pass, that compounds into a measurable wall-clock and memory win across a full model — for negligible change in final quality.

The invariance it keeps and the one it loses

The two norms differ in what transformations of the input leave their output unchanged. RMSNorm is invariant to re-scaling: multiply the input by any constant $c$ and the RMS scales by the same $c$, so the ratio — and therefore the output — is identical.

$$\frac{c\,x_i}{\operatorname{RMS}(c\,x)} = \frac{c\,x_i}{c\,\operatorname{RMS}(x)} = \frac{x_i}{\operatorname{RMS}(x)}$$

What RMSNorm gives up is re-centering invariance: shifting every coordinate by a constant changes its output, whereas LayerNorm — which subtracts the mean first — would absorb that shift. The paper's empirical result is that this lost invariance simply does not matter for transformer quality, which is the whole bet.

Adoption

The bet paid off completely. LLaMA 1, 2, and 3 use RMSNorm, as do Mistral, Qwen, Falcon, DeepSeek, and Gemini. Practically every major open model released after 2022 ships RMSNorm rather than LayerNorm — one of the rare architecture simplifications that became universal because it costs nothing and saves something on every single layer.