Prefill and decode
The two phases of answering a prompt: reading it all at once (prefill), then writing the reply one token at a time (decode).
Also: disaggregated serving
In prefill the model processes the whole prompt in parallel, which is heavy on arithmetic and sets the time to first token. In decode it produces one token after another, which is limited by memory bandwidth. Because the phases stress hardware differently, large services increasingly run them on separate groups of GPUs and pass the KV cache between them, a method called disaggregated serving, supported by tools such as Nvidia's Dynamo, released in 2025. This raises efficiency, at the price of more complex systems.