Flash Attention
Smart implementation of the attention mechanism — uses GPU fast memory efficiently, 2-4x faster with the same result.
Tri Dao 2022. The key: avoid materialising the N×N attention matrix in slow GPU memory. Enabled long context windows. v2 and v3 brought further optimisations. Standard today in every serious LLM training stack.