Annealing phase Midtraining
The final stretch of pretraining, where the learning rate decays toward zero while the data mixture is swapped for substantially higher-quality material.
The observation behind it is that what a model sees last, when the steps are small and fine-grained, has disproportionate influence on how it ends up behaving. Rather than keeping the corpus identical throughout, the share of curated text, textbooks, mathematics, code and instruction-like synthetic material is raised during the final few percent of training. The technique was first described openly in the work around MiniCPM and is now used by essentially everyone — Llama 3 and OLMo document their own variants. The effect is that the boundary between pretraining and post-training has blurred, which is why the term midtraining has appeared to describe exactly this stage.