Megatron-LM
Nvidia's reference implementation for training large transformer models, published in 2019. The paper that introduced tensor parallelism in the form now used everywhere.
The core idea is to split each individual matrix multiplication across several cards rather than splitting the model into layers. In a feed-forward block the first weight is split column-wise and the second row-wise, which means only a single reduction is needed per block. Attention heads are distributed correspondingly, one or more heads per card. Because the communication happens inside every layer the method demands high bandwidth and is therefore used within a node, over NVLink, while pipeline and data parallelism are used between nodes. That combination was documented in the follow-up paper on 3D parallelism and underlies essentially every large training run since.