Expert parallelism
Distributing an MoE model's experts across different GPUs so each card holds only a few. The form of parallelism that makes trillion-parameter models runnable at all.
A model with two hundred and fifty-six experts per layer fits nowhere if every card has to hold all of them. Splitting them up creates a new problem, though: once the router has decided which expert each token goes to, the token has to be sent to whichever card happens to hold that expert, and the results sent back. That means two all-to-all operations per MoE layer, and it is that communication which dominates the time — MoE inference is in practice network-bound rather than compute-bound. Work in the area is therefore mostly about overlapping the communication with computation, as in DeepSeek's DualPipe. Normally combined with tensor and pipeline parallelism simultaneously.