Zero-bubble pipeline
A schedule that removes the idle time in pipeline-parallel training by splitting the backward pass in two halves and deferring the half nobody is waiting for.
Pipeline parallelism splits the model into stages across nodes, and the price is the bubble: at the start of each batch the later stages sit idle waiting for data to arrive, and at the end the earlier stages sit idle waiting for gradients. The key observation is that the backward pass is really two independent computations — the gradient with respect to the inputs, which the upstream stage genuinely waits on, and the gradient with respect to the weights, which nobody needs until it is time to update. By computing the input gradient immediately and slotting the weight gradient into the gaps, the schedule can be filled completely. The 2023 work from Sea AI Lab demonstrated schedules reaching zero bubble under realistic assumptions, against a fifth or more of the time wasted in classic 1F1B, at the cost of more activation memory and a considerably nastier implementation.