Multi-head latent attention (MLA)
An attention architecture (DeepSeek, 2024) that compresses key/value into a small latent vector instead of caching full K/V per head — which shrinks the KV cache even more than GQA. A key behind DeepSeek's low inference cost.
Idea: instead of storing full-size key and value vectors (like MHA) or sharing heads (like GQA), MLA projects K and V down to a shared, low-rank "latent" representation that is cached; the full K/V are reconstructed on the fly when needed. Result: a much smaller KV cache (DeepSeek reports orders of magnitude less than MHA) → longer context, higher throughput, and lower memory cost, with preserved model quality. It was one of the techniques that let the DeepSeek models run unusually cheaply. Distinct from GQA (which reduces the number of KV heads) by instead compressing the KV content via low-rank projection. One of the more noted architecture advances of 2024. Related to grouped-query attention and KV cache.