MQA Multi-Query Attention
A variant of attention in which all heads share a single set of keys and values while keeping their own queries. Proposed by Noam Shazeer in 2019 for one reason: shrinking the KV cache.
In ordinary multi-head attention each head stores its own keys and values for every token, and during generation that cache grows linearly with context length. At long contexts the memory bandwidth to the cache, not the arithmetic, becomes the real bottleneck. With one shared key-value pair per layer the cache shrinks by a factor equal to the number of heads, which can mean ten to a hundred times less. The quality loss is small but measurable, and the resulting compromise was GQA: grouping heads so that several but not all share keys. PaLM and Falcon use MQA directly; most newer models have settled on GQA.