Scale-out network
The network that joins thousands of servers or racks into one cluster, usually InfiniBand or Ethernet.
Also: back-end network, scale-out
The scale-out tier connects many servers or racks into one training cluster. Each GPU typically has its own 400 or 800 Gb/s network card, and traffic flows through layers of switches, so a cluster of 100,000 GPUs needs tens of thousands of switches and cables. This back-end network is kept separate from the front-end network used for storage and the internet. It runs on InfiniBand or on Ethernet with RDMA, and its design, bandwidth and reliability strongly affect how well a large training job runs.