Adam
The optimization algorithm practically all deep learning uses. The name stands for adaptive moment estimation. Published by Kingma and Ba in 2014 and cited over a hundred thousand times.
The idea is that each parameter gets its own effective learning rate, computed from two running averages: the gradient's direction (first moment) and its square (second moment). Parameters with large, noisy gradients are therefore treated cautiously, while rarely updated parameters take bigger steps. The result is a method that works decently at default settings, which accounts for much of its popularity. In practice what's used is AdamW (Loshchilov and Hutter, 2017), which decouples weight decay from the gradient step — the original coupling was simply wrong. The cost is memory: two extra states per parameter, so three times the model size during training.