Goodput
The share of a cluster's time that produces real progress on a job, after losses to failures, restarts and waiting.
Also: effective training time
Goodput counts only useful work. On a large training job, time is lost when hardware fails and the job must restart from its last checkpoint, when a slow node holds back the rest, or while data loads. Meta reported that training Llama 3 405B on 16,384 H100 GPUs saw 419 unexpected interruptions over 54 days, about one every three hours. Providers now advertise goodput and recovery tools, because at that scale a few percentage points equal thousands of GPU-hours.