GQA (Grouped-Query Attention)
A compromise between multi-head attention (MHA, one KV per head) and multi-query attention (MQA, one KV total) — multiple query heads share KV heads in groups.
Google Research (2023). MQA shrinks the KV cache dramatically but hurts quality; GQA with 8 groups (what Llama 2/3 uses) recovers near full-MHA quality with a 4–8× smaller KV cache, meaning larger batches and longer context on the same VRAM.
Default choice in modern decoder models (Llama 2/3, Mistral, Qwen, Gemma). Multi-Latent Attention (MLA) in DeepSeek-V2/V3 goes further by compressing KV into a single latent vector.