Lost in the Middle: How Language Models Use Long Contexts

The experiment

The setup is deliberately clean: multi-document question answering where exactly one document contains the answer and the rest are distractors from the same retrieval corpus. The researchers vary a single variable — the position of the answer document within the context — while holding everything else fixed: the question, the answer document, and the distractor documents. Evaluated models include GPT-3.5-Turbo, Claude 1.3, LongChat-13B, and MPT-30B-Chat, all tested across context lengths from 10 to 30 documents.

The evaluation is exact-match accuracy against a reference answer, so the metric is crisp. A model that actually reads the full context uniformly should show flat accuracy across positions.

The finding: a U-shaped curve

None of the models show flat accuracy. Every model shows a U-shaped performance curve: accuracy is highest when the answer document appears at position 1 (beginning of context) or at position $N$ (end of context), and lowest when the answer document appears in the middle. The degradation from best to worst position is large — typically 20 to 40 percentage points — and consistent across model families, sizes, and context lengths. Larger models and models with longer stated context windows show the same shape; they shift the curve up uniformly but do not flatten it.

The result is the same whether the documents are ordered by retrieval rank or shuffled, ruling out the hypothesis that models are simply following a retrieval heuristic. Position is the variable that explains the variance.

Why the middle loses

Two structural causes, rooted in how causal transformers are trained, explain the bias.

Primacy bias. In causal attention, every token at position $k$ can attend to all previous tokens $1, \ldots, k-1$. The earliest tokens in a sequence are attended to by the largest number of subsequent tokens — token 1 is a potential attender-to for every later position. During training, the gradient signal that flows through the attention weights is proportional to how often a position is attended to. Early tokens accumulate large gradient updates because they are in the attended-to position for nearly every training step. The model's attention weights develop a systematic bias toward early tokens because those tokens were the most frequently rewarded signal source during pretraining.

Recency bias. The next-token prediction objective makes nearby context the most directly useful: predicting token $k+1$ usually requires attending to tokens near $k$ more than tokens near 1. Models trained predominantly on short sequences — where the entire context is within close range — develop strong local attention patterns. When these models are evaluated at long context, the local attention bias persists, which means the final tokens in the context remain strongly attended-to even at long range. The last document in a 30-document context is close to the generation position; it benefits from recency the way the first document benefits from primacy.

Middle tokens have neither advantage. They are not early enough to benefit from primacy and not close enough to the generation position to benefit from recency. During training they were the least-attended positions; at inference time the model has not learned to systematically retrieve from them.

Implications for retrieval-augmented generation

The finding directly contradicts a common RAG construction pattern. Retrieval systems typically return documents ranked by similarity score and concatenate them in rank order: most relevant first, then progressively less relevant. If the most relevant document is at position 1, the primacy bias actually helps. But many RAG pipelines produce long context windows where the top-ranked document is followed immediately by lower-ranked documents, pushing the query-critical content into a middle block — exactly where the model attends least.

The actionable implication: place the most relevant retrieved documents at the beginning or end of the context, not in the middle. In practice this means putting the top-1 and top-2 retrieved documents at positions 1 and $N$, with lower-ranked documents filling the middle. This ordering — directly motivated by the U-shaped attention curve — measurably improves answer accuracy over flat ranked concatenation, without changing the retrieval model, the generation model, or the number of documents.

Context length and utilization are orthogonal

The most important framing the paper establishes is the separation of context length (how many tokens a model can process without error) from context utilization (whether the model actually attends to information uniformly across that length). These are different properties, and one does not imply the other.

A model with a 100K token context window may have been extended from a 4K training context via positional interpolation — the mechanics of long-context extension ensure that the model does not crash on long inputs, but they say nothing about whether the attention weights have been trained to retrieve from middle positions. The paper shows empirically that this gap is real and large. Fine-tuning with synthetic retrieval tasks where the answer is deliberately placed in the middle of the context — not just increasing the context window — is what closes it. This distinction became a standard lens for evaluating long-context models: benchmarks like RULER and Needle-in-a-Haystack test positional retrieval explicitly, not just perplexity at long range, precisely because the two properties can diverge.