Collective communication
Standard group operations, such as all-reduce, through which many GPUs share and combine results during training.
Also: all-reduce, NCCL
Distributed training relies on a small set of collective operations. All-reduce sums values from every GPU and gives each the total; all-gather collects pieces from every GPU; all-to-all sends each GPU a different piece from every other. Libraries such as Nvidia's NCCL and AMD's RCCL carry them out, choosing routes through NVLink and the network. Their speed often decides training efficiency, and InfiniBand switches can perform part of the sums themselves to save time.