Auxiliary-loss-free load balancing
Balancing the load between experts in an MoE model with an adjustable bias term rather than a penalty in the loss function.
The established method adds an auxiliary loss penalising uneven distribution across the experts. It works, but it competes with what you actually want to train: the gradient pulls partly toward better language modelling and partly toward more even statistics, and the latter costs measurably in quality. DeepSeek-V3 instead moved balancing outside the gradient entirely. Each expert gets a bias value added to its routing score only when the top-k list is formed, and that value is adjusted continuously during training — up for experts receiving too little, down for those receiving too much. The weights used to combine outputs are left untouched. The result was both more even load and a better model.