StreamingLLM
A method that lets a model generate indefinitely at constant memory, by keeping the first four tokens plus a sliding window of the most recent ones.
The starting point is the discovery that a plain sliding window destroys the model as soon as the opening tokens fall out of the cache. The fix is minimal: keep them permanently as anchors and let the rest of the window roll. Positions are additionally assigned by where a token sits in the cache rather than where it sat in the original text, which keeps distances inside the range the model was trained on. The important limitation is commonly misread — the method does not extend the context length. Anything that rolled out of the window is gone and the model cannot refer back to it. What it gives you is a model that can hold a conversation for days without memory growth or quality collapse.