Checkpointing
Saving the full state of a training job at intervals so it can resume from there after a failure.
Also: checkpoint
A checkpoint is a snapshot of a model's parameters and training state written to storage. On large clusters, hardware failures are routine: in a job across tens of thousands of GPUs, something breaks every few hours. Without checkpoints, all progress since the start would be lost. Writing a checkpoint for a very large model can mean terabytes of data, so fast parallel storage and asynchronous saving, which writes in the background while training continues, are important. The interval is a trade-off between time spent saving and work lost after a crash.