Batch normalization
Normalizing each feature across all samples in the minibatch. Presented by Ioffe and Szegedy in 2015 and one of the single biggest reasons deep convolutional networks became trainable at all.
The effect was dramatic: learning rates could be raised by orders of magnitude, initialization became less sensitive and networks converged far faster. The original explanation — that it reduces "internal covariate shift" — has since been questioned; a more common interpretation today is that it smooths the loss landscape. Two practical problems follow from drawing statistics from the batch: with small batches the estimates get noisy, and behaviour differs between training and inference since stored running averages are used then, a classic source of bugs. Sequence models therefore use layer normalization instead.