KV cache
The cache that stores the attention mechanism's "key" and "value" vectors for already-processed tokens, so they don't have to be recomputed for every new token generated. Absolutely crucial for autoregressive text generation to be fast.
Why: an LLM generates one token at a time, and each new token needs to "look at" all previous tokens via attention. Without a cache, the entire sequence would be recomputed at every step (quadratic waste). The KV cache saves the key/value vectors for all previous tokens; at each new step only the new token's vectors are computed and compared against the cache. Problem: the cache grows linearly with the sequence length AND with the batch size and quickly becomes the dominant memory consumer at long context → a bottleneck for throughput. Solutions: PagedAttention (efficient memory management), grouped-query attention (fewer KV heads), and KV-cache quantization. The central data structure in LLM inference. Related to grouped-query attention and continuous batching.