RoFormer: Enhanced Transformer with Rotary Position Embedding

Position is the one thing attention forgets

Self-attention is permutation-equivariant: the dot product between a query and a key knows nothing about where either token sits in the sequence. Order has to be supplied separately, and how you supply it turns out to matter enormously for how well a model generalizes across lengths. The original Transformer added a sinusoidal vector to each token's embedding; many models instead learned an absolute position embedding per slot. Both share a weakness — they inject absolute position, while what attention usually wants is relative position: the model cares that a verb is three tokens after its subject, not that the subject is at index 47. Relative-position schemes existed but typically modified the attention scores additively with extra learned terms, complicating the computation. RoPE's contribution is a way to get exact relative-position behavior while still injecting position into each token individually, and without adding anything to the attention formula.

Rotation instead of addition

The idea is to rotate rather than add. Take a query or key vector, split it into pairs of coordinates, and treat each pair as a point in a 2D plane. RoPE rotates each pair by an angle proportional to the token's absolute position — position $m$ rotates by $m\theta$, where each pair has its own base frequency $\theta$. Different coordinate pairs use geometrically spaced frequencies, exactly as the sinusoidal scheme spread its periods, so some pairs rotate fast and capture fine-grained local offsets while others rotate slowly and capture coarse long-range position. The rotation is applied to $Q$ and $K$ before their dot product; the values $V$ are left untouched.

The mechanism is cleanest in complex form. If a 2D pair is written as a complex number $x$, applying RoPE at position $m$ multiplies it by $e^{im\theta}$. A query at position $m$ becomes $q\,e^{im\theta}$ and a key at position $n$ becomes $k\,e^{in\theta}$.

Why the dot product becomes relative

Now form the attention score between query position $m$ and key position $n$. The inner product of the rotated vectors carries the phase factor

$$\langle q\,e^{im\theta},\; k\,e^{in\theta} \rangle \;\propto\; \mathrm{Re}\big(q\,\bar{k}\,e^{i(m-n)\theta}\big),$$

and the absolute positions $m$ and $n$ appear only through their difference $m - n$. This is the whole trick. Each vector was rotated by its own absolute position, but when two rotated vectors are dotted together, the absolute angles cancel and only the relative rotation $(m-n)\theta$ survives. RoPE encodes absolute position at the level of individual tokens yet produces attention scores that depend solely on relative position — the property relative-encoding schemes wanted, obtained for free from the algebra of rotation, with no extra terms bolted onto the score.

A second consequence falls out of the same form: because rotation preserves vector norms, RoPE never changes the magnitude of $q$ or $k$, only their direction. Position rotates the scores without rescaling them, so it does not quietly distort the softmax temperature the way an additive bias can. And as the relative offset $m-n$ grows, the phases of the many frequency components fall out of alignment and their contributions interfere destructively, so attention between very distant tokens tends to wash out — a built-in long-range decay that matches the intuition that far-apart tokens should usually interact less.

Why it took over

RoPE became the default positional scheme for large language models — LLaMA, PaLM, GPT-NeoX, and most open models since adopt it — for a combination of reasons. It delivers true relative-position behavior, which generalizes better than absolute schemes. It adds no parameters and no extra attention terms; it is just a rotation applied to $Q$ and $K$, cheap and easy to fuse into the kernel. It introduces no learned position table that caps the sequence length at training time. And it composes cleanly with the attention variants that came to dominate — grouped-query and multi-query attention, and latent-attention schemes — because it acts on the query and key vectors themselves rather than on the score matrix, so it does not interfere with how those variants share or compress the keys and values.

Extending past the training length

RoPE's most consequential practical property is how it handles contexts longer than the one a model was trained on. Because position enters as a rotation frequency, you can rescale the frequencies to make a longer sequence occupy the same range of angles the model saw in training. Linear position interpolation divides the position index so that, say, a 4× longer context rotates through the same total angle, keeping the model inside the regime it learned. Frequency-aware refinements — NTK-aware scaling and YaRN — interpolate the different frequency bands unevenly, stretching the slow long-range frequencies while leaving the fast local ones nearly intact, which preserves local precision while extending reach. These methods let a model trained at one context length be adapted to a much longer one with little or no additional training, and they are the standard route to long-context models today. The extensibility is a direct gift of encoding position as rotation: there is a continuous knob (the frequency) to turn, which a learned absolute table simply does not have.

Limitations

RoPE generalizes to longer contexts far better than absolute schemes, but not infinitely — push the context well past training length without interpolation and quality still degrades, which is exactly why the YaRN/NTK family of frequency-rescaling tricks exists rather than RoPE extrapolating perfectly on its own. The base frequency (often written as the wavelength constant, commonly 10000) is a hyperparameter that interacts with the maximum useful context, and getting long-context behavior right can require tuning it or the interpolation schedule. And like any fixed positional inductive bias, RoPE's built-in distance decay is a prior that mostly helps but can work against tasks that genuinely need strong attention across very long spans. None of this has dislodged it; rotary embeddings remain the positional encoding the field converged on, and the reason is the elegant cancellation at the center of the method — rotate by absolute position, and relative position is what the dot product sees.