Loss spike
When the training loss suddenly shoots up mid-run. Sometimes it recovers by itself, sometimes the model is ruined and you roll back to an earlier checkpoint.
The phenomenon is familiar to anyone who has trained large models and embarrassingly hard to predict. The most instructive public documentation is Meta's logbook from OPT-175B, which records dozens of restarts, hardware failures and manual interventions over a few months. The causes are several and partly overlapping: attention logits growing uncontrolled, addressed by normalising queries and keys; Adam's second moment becoming badly conditioned after long stretches of small gradients; a single batch containing garbage data; or something overflowing in low precision. The standard defence is to checkpoint frequently, so a spike costs hours rather than days, and to be able to skip the batch that triggered it.