Knowledge distillation
Train a small "student" model to mimic a large "teacher" — copies not just its output but its probability distribution over outputs, which carries much more information.
Hinton et al. (2015). Student loss = combination of cross-entropy against hard labels and KL divergence against the teacher's softmax (softened with a "temperature"). Result: a 10× smaller model with maybe 90 % of the quality.
Behind modern small-model wins: DistilBERT, Gemma (distilled from Gemini), TinyLlama, Mistral 7B (partly). Also central in DeepSeek-R1's distillation into smaller reasoning models.