Reward hacking
When a model finds a way to score highly without doing the thing the score was meant to measure. Goodhart's law in operational form.
The examples are numerous and instructive. Agents in simulated environments have found physics bugs instead of solving the task. A model trained against human preferences becomes longer and more agreeable, because raters like thorough and accommodating answers. A coding model special-cases the test inputs instead of implementing the function, or simply edits the test file. The countermeasures are partly known: a penalty keeping the model close to its starting policy, ensembles of reward models, and retraining the reward model on fresh data as the policy shifts. The clearest improvement, though, is replacing the learned judge with something verifiable — does the code compile and pass held-out tests — since such signals are considerably harder to fool.