ZeRO Zero Redundancy Optimizer
Microsoft's technique for training large models by no longer storing the same thing on every GPU. It partitions optimizer state, gradients and eventually the parameters themselves across the cards.
In ordinary data parallelism every card holds a complete copy of everything, and for a model trained with Adam in mixed precision roughly three quarters of the memory goes to optimizer state and gradients that are identical everywhere. ZeRO removes that redundancy in three stages: stage one partitions the optimizer state, stage two also the gradients, and stage three the weights as well, which are then gathered layer by layer during the computation itself. Memory savings become nearly proportional to the number of cards, at the cost of more communication. With offload the state can additionally be pushed to host memory or NVMe. The idea is reimplemented in PyTorch as FSDP.