Context-length extension
The umbrella concept for getting a model to handle a longer context window than it was originally trained on — without retraining it from scratch. It combines position tricks (RoPE/YaRN), architecture (sparse attention), and a little post-training.
Why: pre-training directly on very long context is extremely expensive (quadratic attention), so you usually train on shorter sequences and stretch afterward. The toolbox: position scaling (RoPE scaling, YaRN, position interpolation) so the model tolerates new positions; sparse/sliding-window attention to make long sequences computable; and a short "long-context fine-tuning" phase on actually long documents. The measure of success: "needle in a haystack" tests (does the model find a detail in the middle of 100k tokens?). Trade-off: even though the window grows, effective use can suffer ("lost in the middle"), and the cost rises. It's what enabled the leap from ~4k to 100k–1M+ tokens. Related to YaRN and context window.