Data parallelism
Training on many GPUs at once by giving each a copy of the model and a different slice of the data, then combining what they learn.
Also: FSDP, ZeRO
In data parallelism every GPU holds the model and processes a different batch of data; after each step the GPUs average their updates with an all-reduce operation. It is the simplest way to use many GPUs, but each must hold the whole model. Methods such as ZeRO and fully sharded data parallelism (FSDP) split the model's state across GPUs so larger models fit. Large training runs combine data parallelism with tensor, pipeline and expert parallelism, a mix sometimes called 3D or 4D parallelism.