Batching
Processing several requests or data samples together, so the chip does more useful work per read of the model.
Also: batch size, continuous batching
Reading a model's weights from memory is costly, so serving systems group many users' requests and process them together, sharing each read. Larger batches raise total throughput and lower cost per token, but each user waits a little longer. Continuous batching, used in servers such as vLLM, adds and removes requests from the batch at every step instead of waiting for a whole batch to finish. In training, batch size is the number of samples per step, which affects both speed and how well the model learns.