Tensor parallelism
Splitting each layer of a model across several GPUs so they compute it together, needing very fast links between them.
Also: model parallelism
Tensor parallelism divides the large matrices inside each layer among several GPUs, so each computes part of every step. The GPUs must exchange partial results many times per layer, so this is normally kept within a scale-up domain such as an eight-GPU server or an NVL72 rack, where NVLink is fast enough. It lets models too large for one GPU run and speeds up each step, and is used in both training and inference of large models.