FlashAttention-2
Memory-efficient attention implementation. Tri Dao 2023 (FA-1 2022). Reformulates self-attention for better GPU memory access pattern (HBM ↔ SRAM). 2-4x faster training, 4-8x less memory consumption vs naïve attention.
Key fix vs FA-1: better work partitioning between thread blocks, fewer non-matmul operations. Standard in modern LLM training stacks (HuggingFace, Megatron, PyTorch). FA-3 (2024): Hopper-specific (H100), 1.5-2x faster than FA-2 via asynchronous instructions + low precision (FP8). Enables 32K+ context-window training without memory OOM. Tri Dao then went to Together AI to commercialize. Standard building block in modern LLM inference (vLLM, TGI) too.