Self-attention
Mechanism where each token in a sequence "looks at" every other token and decides how much weight to give them — the heart of a transformer's expressivity.
Three learned projections per token: query (Q), key (K), value (V). Attention score = softmax(QK^T / √d) · V. Multi-head: run several self-attentions in parallel with different projections and concatenate.
Time complexity O(n²) in sequence length — the primary bottleneck in long-context LLMs. Optimisations: Flash Attention (better memory handling), Sparse Attention, Sliding Window, GQA, MLA.