Efficient Streaming Language Models with Attention Sinks

The streaming problem

Picture a model that has to keep generating indefinitely — a chat assistant in a session that never ends, a system summarizing a feed as it arrives. Two things break. First, the KV cache grows without bound: every token ever seen leaves a key and value behind, so memory climbs until the GPU is full. Second, the model was trained on sequences up to some fixed length, and once the context runs past that length, quality falls apart even if you had the memory to hold it. You need a way to bound the cache and stay inside the model's effective range, forever.

The obvious fix that doesn't work

The intuitive bound is a sliding window: keep only the most recent $L$ tokens' keys and values, evict the rest. The cache is capped at $L$, the math is trivial, and you would expect graceful forgetting of distant context. Instead, the moment the very first tokens are evicted from the window, the model's output quality collapses — perplexity spikes dramatically. This is the puzzle the paper opens with, because nothing about the recent-window content has changed; the only thing lost was a handful of tokens at the start of the sequence, tokens that by now are far in the past and seemingly irrelevant to predicting the next word.

The attention sink

The explanation is a phenomenon the authors name the attention sink. If you look at where the softmax attention mass actually goes in a trained transformer, a strikingly large fraction lands on the first few tokens of the sequence — and it does so across layers and heads regardless of whether those initial tokens carry any semantic relevance to the current position. The opening token of a document gets enormous attention weight even when it is something content-free like a start-of-sequence marker.

The reason is structural. The softmax forces the attention weights to sum to one — every query must distribute a full unit of attention mass somewhere, even when it has no informative token it actually wants to attend to. The model needs a place to dump that obligatory mass, and during training it learns to use the initial tokens as that outlet: they are visible to every position (causal masking never hides them) and present in every sequence, so they make a reliable, always-available sink. The attention sink is the softmax's pressure-release valve, not a content lookup.

This immediately explains the sliding-window collapse. Evict the first tokens and you remove the sink. The attention mass that those positions were absorbing has nowhere to go, gets forced onto tokens that were never meant to carry it, and the whole distribution distorts. The model breaks not because it lost information but because it lost its normalization outlet.

The fix: keep the sinks, slide the rest

The remedy follows directly. StreamingLLM keeps a small number of initial tokens — typically four — permanently in the cache as sink tokens, and runs a sliding window of recent tokens alongside them. Everything between the sinks and the window is discarded. The cache is bounded at sinks-plus-window, the attention mass keeps its outlet, and generation continues stably over streams far longer than the training length — effectively without limit. The paper shows models running over millions of tokens with stable perplexity, where the plain sliding window had collapsed almost immediately.

There is one subtlety in the positional encoding. Because the cache holds a non-contiguous slice — a few tokens from the very start, then a gap, then a recent window — positions are assigned by location within the cache rather than absolute position in the original stream. The recent window is treated as following right after the sink tokens, so the rotary or relative positions stay inside the range the model was trained on no matter how long the underlying stream has run. This keeps the model from ever seeing a position index beyond its training horizon.

A pre-trainable sink token

The paper goes one step further with a training-time observation. If you deliberately prepend a single dedicated, learnable sink token to every sequence during pretraining, the model learns to route its surplus attention there specifically, concentrating the sink behavior into one designated slot rather than smearing it across the first several real tokens. A model trained this way needs only that one sink token retained at streaming time, and the effect is cleaner. This reframes the attention sink from a quirk you discover and work around into a mechanism you can design for.

What it buys

The headline efficiency number is a 22.2× speedup over the natural baseline for unbounded generation — the sliding-window-with-recomputation approach, which re-encodes the context window on each step to stay within length. StreamingLLM avoids that recomputation entirely: it keeps a fixed, small cache and never re-processes old tokens, so per-token cost stays flat as the stream grows. Crucially, all of this is inference-time only and tuning-free for existing models (the dedicated sink token is the one optional training-time refinement); it applies to LLaMA, Falcon, Pythia, MPT, and others as-is.

The honest limitation

StreamingLLM does not give a model infinite memory — it gives it infinite operation. Anything that scrolled out of the recent window is gone; the sink tokens are an attention outlet, not a summary of the discarded context. A question about something said a million tokens ago, long since evicted, cannot be answered from the cache. The method is the right tool when the task is genuinely streaming — staying coherent and responsive over an unbounded run where recent context dominates — and the wrong tool when you need true long-range recall, which is the province of full long-context attention or retrieval. Within that scope, the contribution is both a practical mechanism and a piece of mechanistic understanding: it named the attention sink, explained it from the softmax's sum-to-one constraint, and turned that explanation into a cache policy that just works.