Lion
An optimizer Google found by letting a search algorithm comb the space of possible optimizers rather than designing one by hand. The result was remarkably simple.
The update consists of interpolating the gradient with momentum, taking the sign function of the result, and stepping by that. Every element of the update therefore has the same magnitude and differs only in direction. Two practical consequences follow. Since only one momentum buffer has to be stored, against Adam's two moments, the memory spent on the optimizer halves — which at large model sizes is a real sum. And since the steps are unit-sized, the learning rate has to be set roughly a tenth of what you are used to, with correspondingly higher weight decay. Results are strong on vision and competitive on language, but sensitivity to hyperparameters has let AdamW keep its place at the frontier.