KV Cache Explosion Glossary

The serving-memory pressure created as an attention model stores key and value tensors for more tokens, users, and concurrent generations.

KV Cache Explosion is the Dictionary’s name for the serving-memory pressure created as an attention model stores key and value tensors for more tokens and more concurrent generations.

During autoregressive inference, a transformer can cache the key and value tensors produced for earlier tokens rather than recomputing them at every step. In a basic dynamic cache, the stored sequence dimension grows with each processed token. For a fixed architecture and numerical precision, cache memory therefore grows roughly linearly with sequence length. The total bill also depends on the number of layers, key-value heads, head dimension, precision, batch size, and the number of simultaneous requests.

Longer contexts are consequently not free. Documents, conversations, code, images represented as tokens, and tool traces all consume serving memory when they remain available to attention. Techniques such as grouped- or multi-query attention, sliding windows, cache quantization, paging, offloading, and eviction can reduce or manage the burden, usually with implementation or quality trade-offs. “More context” sounds like a software feature. At scale, it is also a memory-capacity and memory-bandwidth problem.

This matters for the Dictionary because long context is one of the places where Frontier Dependence can remain strong. Local models can do useful work, but large hosted systems can combine substantial memory with serving infrastructure designed for many concurrent users. The user experiences this as “the model can hold more.” The data center experiences it as memory pressure.

The phrase is a useful reminder: every magical long-context conversation has a physical memory bill somewhere.

See also

Sources

Return to Dictionary All Entries (A–Z) For Students Other Writing Capstone 2.0