Attention Is All You Need

The bottleneck recurrence couldn't escape

Before 2017 the dominant sequence models were recurrent — LSTMs and GRUs, usually with an attention mechanism bolted onto an encoder-decoder. Recurrence carries an unavoidable cost: to compute the hidden state at position $t$ you need the state at $t-1$, so the computation along the sequence is fundamentally serial. You cannot parallelize across the time dimension within a single example, which caps how well training utilizes a GPU and means information between two distant tokens has to survive a long chain of intermediate states to get from one to the other. Attention had already been added to these models to let the decoder look back at the whole input, but it was always an accessory to the recurrent backbone. The paper's wager — captured in the title — is that the recurrence is not just dispensable but a liability, and that attention alone is enough to model sequences.

Scaled dot-product attention

The primitive is simple. Every position emits three vectors: a query, a key, and a value. To compute the output at a given position, you take its query, dot it against the keys of every position, scale and softmax those scores into weights, and use the weights to take a convex combination of the values. Packed into matrices $Q$, $K$, $V$:

$$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V.$$

The $\sqrt{d_k}$ in the denominator is not decoration. For large key dimension $d_k$, the raw dot products grow in magnitude, pushing the softmax into regions where its gradient vanishes; dividing by $\sqrt{d_k}$ keeps the scores at a scale where the softmax stays responsive. Because the whole operation is two matrix multiplications and a softmax, it runs in parallel across every position at once — the serial dependency of recurrence is simply gone.

Many heads, many subspaces

A single attention operation forces every position to mix information through one weighting scheme. Multi-head attention runs $h$ of them in parallel, each with its own learned projections of $Q$, $K$, and $V$ into a lower-dimensional subspace. One head can track syntactic agreement, another can resolve a pronoun's antecedent, a third can attend to positional neighbors — each in its own representational subspace — and their outputs are concatenated and projected back to the model dimension. The original Transformer uses $h = 8$ heads. This is the operation every modern attention variant descends from: grouped-query and multi-query attention are exactly multi-head attention with the key/value heads shared or collapsed to save memory.

Where attention is used three ways

The architecture is an encoder-decoder, and attention appears in three distinct roles. In the encoder, self-attention lets every input token attend to every other input token, building context-aware representations of the source. In the decoder, masked self-attention lets each output position attend only to positions at or before it — the mask sets the scores for future positions to $-\infty$ before the softmax, enforcing the causal constraint that you cannot peek at tokens you have not generated yet. And encoder-decoder attention lets each decoder position query the full encoded source, the role attention played in the older RNN models. Every block also contains a position-wise feed-forward network — two linear layers with a nonlinearity — applied identically to each position, and the whole thing is wrapped in residual connections and layer normalization.

Telling the positions apart

Attention is permutation-invariant: shuffle the input tokens and the set of attention outputs is just shuffled the same way, because nothing in the dot-product operation knows where a token sits. Order has to be injected explicitly. The paper adds sinusoidal positional encodings to the input embeddings — fixed vectors whose entries are sines and cosines of the position at geometrically spaced frequencies:

$$PE_{(pos, 2i)} = \sin\!\left(\frac{pos}{10000^{2i/d}}\right), \qquad PE_{(pos, 2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d}}\right).$$

The geometric spread of frequencies means relative offsets are expressible as linear functions of the encodings, which the authors conjectured would help the model attend by relative position. This particular scheme was later superseded — rotary position embeddings became the standard in large language models — but the problem it solves, and the idea of encoding position as a function of frequency, traces directly back to here.

What it bought, immediately

On the WMT 2014 English-to-German translation task, the big Transformer reached a BLEU of 28.4, a new state of the art that beat every previously published single model and even the best ensembles, while on English-to-French it set a single-model record. The result that mattered as much as the score was the cost: it reached these numbers after a fraction of the training compute the best recurrent and convolutional models required, because the absence of recurrence let it saturate the available hardware. Better quality and cheaper training at the same time is the combination that makes a method spread.

Why it became the foundation

The Transformer's real legacy is not the translation record but its architecture's two structural properties. First, it parallelizes across sequence positions, which is precisely what lets it scale to enormous models and datasets on modern accelerators — the entire scaling-laws research program assumes an architecture you can train efficiently at scale. Second, self-attention gives a constant path length between any two positions: distant tokens interact in a single step rather than through a long recurrent chain, so long-range dependencies are learned far more readily. Strip away the encoder, keep only the masked decoder, and you have the decoder-only language model that underlies GPT and every model in its lineage. The components introduced here — scaled dot-product attention, multi-head projections, residual-plus-norm blocks, position-wise feed-forwards — are still the components of a frontier model in 2025. What changed since was scale, normalization placement, the positional scheme, and the attention variant; the skeleton is the one this paper drew.

Limitations the field spent years on

The attention operation is quadratic in sequence length — every position attends to every other, so cost grows as $O(n^2)$ in both compute and memory — which is the constraint that motivated FlashAttention's IO-aware tiling, sliding-window and sparse attention, and the KV-cache compression literature. The original sinusoidal encodings extrapolate poorly past the training length. And the encoder-decoder framing has largely given way to decoder-only models for general-purpose language modeling. None of these revisions touched the core claim. Attention really was all you needed.