PagedAttention
A virtual-memory trick for the KV cache that lets LLM servers batch multiple requests without wasting GPU memory.
Invented by the vLLM team at Berkeley (2023). Classically you allocate a contiguous KV cache per request, which causes internal fragmentation (you reserve for max context but use only a fraction). PagedAttention chops the cache into fixed "pages" (e.g. 16 tokens) and keeps a page table per sequence — exactly like the operating system's virtual memory. Result: 2–4× throughput gains, room for far more concurrent users.