KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

The cache, not the weights, is the problem

For a model serving long contexts and large batches, the KV cache routinely outgrows the weights as the dominant consumer of GPU memory. Every token, at every layer, deposits a key vector and a value vector that must be retained for the rest of the sequence, and that footprint scales with batch size and context length while the weights stay fixed. Once the cache is the binding constraint, the lever that buys the most is storing each cached entry in fewer bits. Casting from FP16 to INT8 halves it; the prize is going lower still, to 2 bits, which would cut the cache to an eighth. The obstacle is that naive low-bit quantization of the KV cache wrecks model quality — and KIVI's contribution is a careful diagnosis of why, and a quantization scheme shaped around the answer.

Keys and values are not the same tensor

The paper's central observation is that keys and values have different statistical structure, so quantizing them the same way is a mistake. When you inspect the key cache, you find persistent outliers concentrated in specific channels — certain feature dimensions carry consistently large magnitudes across many tokens. The value cache shows no such per-channel pattern; its magnitudes are comparatively uniform across channels.

This matters because of how quantization error propagates. A key feeds the softmax: it is dotted against a query, scaled, and exponentiated, so an error in a key is amplified through a nonlinearity and reshapes the entire attention distribution downstream. A value, by contrast, is only linearly combined into the output via the attention weights, so its error stays local and contributes proportionally. Keys are the sensitive tensor, and the outlier channels are exactly where the damage concentrates.

Quantize along the axis that hides the outliers

Quantization always groups numbers and assigns each group a shared scale. The grouping axis is the design choice, and KIVI picks it differently for each tensor — this is the "asymmetric" in the title, and it refers to granularity, not bit-width. Both keys and values are quantized to the same 2 bits; what differs is the direction along which the grouping runs.

Keys are quantized per-channel — each feature dimension gets its own scale, computed across the tokens. Because the key outliers live in particular channels, isolating each channel keeps a high-magnitude channel from inflating the scale of its well-behaved neighbors; the outlier is contained within its own group rather than blowing up the dynamic range of everything quantized alongside it. Values are quantized per-token — each token's value vector gets its own scale across its channels — which fits the value cache's flatter, token-wise structure. Matching the grouping axis to where each tensor's variation lives is what lets 2 bits survive where a uniform scheme would collapse.

The streaming complication

Per-token value quantization is natural in a streaming decoder: a token arrives, you quantize its value vector, you are done, and nothing about a new token disturbs an old one. Per-channel key quantization is harder, because a channel's scale is computed across tokens — and in autoregressive decoding the tokens arrive one at a time, so the full set is never available at once. Recomputing every channel scale on each new token would defeat the purpose.

KIVI's answer is to keep a small residual window of the most recent keys in full precision and quantize only the older keys in groups. As the window fills, a completed group of keys is quantized per-channel once and frozen; the newest tokens stay in FP16 until they age out of the window. This keeps the per-channel scheme compatible with token-by-token generation, bounds the full-precision overhead to a fixed window regardless of sequence length, and has the side benefit that the most recent tokens — which attention weights most heavily — are never degraded at all.

Tuning-free, and that is the point

The scheme requires no fine-tuning, no calibration pass, and no changes to the model weights. It is applied at inference time to whatever checkpoint you already have. The paper demonstrates this directly on LLaMA-2, Falcon, and Mistral models, dropping 2-bit KIVI in front of the existing weights and measuring quality on language modeling and a battery of downstream tasks. Across these, 2-bit KIVI holds accuracy close to the FP16 baseline while shrinking the KV cache substantially — the memory freed translates into either larger batches or longer contexts at fixed hardware, and the paper reports correspondingly higher peak throughput because more sequences fit in flight at once.

Why it matters

KIVI is part of a broader shift in how the field thinks about the KV cache: not as a monolithic block of numbers to be compressed uniformly, but as a structured object whose keys and values, channels and tokens, recent and old positions, all warrant different treatment. The specific insight — that key outliers are per-channel and value magnitudes are per-token, so the quantization granularity should follow suit — is a clean example of letting the data's structure dictate the algorithm rather than applying a one-size scheme and hoping. It also stacks cleanly with the other cache-reduction techniques, since quantization shrinks each retained entry while eviction methods like H2O and StreamingLLM decide which entries to retain at all; the two operate on orthogonal axes.

Limitations and the broader trade

Two bits is aggressive, and the residual window is the safety margin that makes it tolerable — shrink the window too far and the most-attended recent tokens start to suffer; widen it and the full-precision overhead grows. The per-channel key path also asks more of the kernel than a uniform per-token scheme, since the quantization runs along a different axis than the natural memory layout, so realizing the memory savings as wall-clock speedup depends on an implementation that handles the mixed-granularity, mixed-precision cache efficiently. And like every lossy compression of the cache, KIVI trades a small, measurable quality cost for memory; the case it makes is that with the asymmetry matched correctly, that cost at 2 bits is small enough to be worth an $8\times$ reduction in the tensor that was bounding your batch size.