Sliding-window attention
A form of sparse attention where each token only gets to look at a fixed number of nearest preceding tokens (a sliding window), instead of the entire history. Popularized by Mistral 7B for efficient handling of long context.
Mechanic: instead of letting token N attend to all tokens 1…N, it's limited to the last W (e.g. 4096) tokens. The trick: when several such layers are stacked, information can still "flow" further back — a token in layer 2 sees the window from layer 1, which in turn saw a window back, so the effective range becomes W × number of layers (like receptive fields in a CNN). Win: constant memory use per token and linear scaling, which enables long sequences cheaply. Trade-off: long-range dependencies must pass through several layers and can weaken. Mistral combined it with a "rolling buffer" KV cache. A concrete, influential example of sparse attention. Related to sparse attention and KV cache.