Expert parallelism
Placing the different experts of a mixture-of-experts model on different GPUs and routing each token to the right one.
In a mixture-of-experts model, experts can be spread across many GPUs. Each token is sent to the GPUs holding its chosen experts and the results are sent back, an all-to-all exchange that happens in every expert layer. This makes heavy, uneven demands on the network, especially when some experts are more popular than others. Large NVLink domains, careful placement of experts and dedicated communication libraries, such as DeepSeek's DeepEP released in 2025, all aim to keep it fast.