Pipeline parallelism
Splitting a model's layers into stages on different GPUs and passing data along them like an assembly line.
In pipeline parallelism, the first group of layers sits on one set of GPUs, the next group on another, and so on. Batches are cut into smaller micro-batches that flow through the stages one after another, so all stages stay busy. Because only the results at stage boundaries cross between groups, it suits slower links between servers. The main cost is idle time at the start and end of each batch, called the pipeline bubble, which schedulers work to reduce.