Parallel file system
Storage spread across many servers so thousands of GPUs can read data and save checkpoints at the same time.
Also: Lustre, GPFS
A parallel file system splits files across many storage servers and lets every compute node read and write them at once, giving combined speeds of terabytes per second. Lustre and IBM Storage Scale (formerly GPFS) come from supercomputing; commercial systems from DDN, WEKA and VAST Data are common in AI clouds. Training clusters need this to feed data to GPUs and to write checkpoints quickly. Many setups add a fast local flash tier on each node and a cheaper object store behind.