MQA (Multi-Query Attention)
Attention variant where all query heads share a single key/value pair — drastically shrinks the KV cache but sacrifices quality.
Noam Shazeer, 2019. Standard multi-head attention has N query heads + N KV heads; MQA has N + 1. Result: KV cache N× smaller, dramatically faster inference at long context and large batches. The lost expressiveness, however, shows up in benchmarks.
Used in PaLM, Falcon, StarCoder. For today's big models, GQA has taken over as the default — it gives nearly full MHA quality without paying MQA's price. DeepSeek's MLA goes further by compressing KV into a latent vector.