Tensor parallelism
Splitting individual weight matrices across several GPUs so each card computes part of the same layer. Necessary when a model simply doesn't fit in one card's memory.
Each matrix multiplication is divided by columns or rows, the cards compute in parallel and the results are combined with an all-reduce. That happens several times per layer, which makes the method extremely sensitive to communication latency — so it's used almost exclusively within a single server, over NVLink, and rarely between nodes. The alternatives trade differently: pipeline parallelism puts different layers on different cards and communicates only at the boundaries, but suffers bubbles while cards wait for each other, while data parallelism copies the whole model to each card and splits the batch. Large training runs combine all three. Megatron-LM popularized the technique.