Synthetic data
Training data generated by another model — increasingly important as the open internet runs out as a data source. Phi-3, Llama 3.1, Claude all use it heavily.
Self-Instruct (2022) was first popular method. Phi-3 (Microsoft) is trained almost entirely on synthetic data. Risk: "model collapse" (Shumailov et al. 2023) — repeated generations of synthetic data may degrade quality. Mitigated by mixing in real data, distillation, RLHF/RLAIF.