Training clusters
Reserved clusters of thousands of GPUs on one fabric. 105 providers on EveryNeocloud so far.
A training cluster is a block of hundreds to hundreds of thousands of GPUs joined on one fast network, reserved by one customer for months or years. This is how frontier models are trained, and it is the core business of the largest neoclouds. Deals are signed as contracts worth billions of dollars, often before the data centre is built.
What makes a cluster
Inside each server, GPUs talk over NVLink; between servers they use InfiniBand or fast Ethernet in a layout called a fat tree, so any GPU can reach any other at full speed. Shared storage feeds training data, and software such as Slurm or Kubernetes schedules jobs and restarts them when a GPU fails, which in a cluster of 100,000 GPUs happens many times a day.
How it is sold
Customers sign take-or-pay contracts: they pay for the capacity whether or not they use it. Microsoft, OpenAI, Meta and Google have signed such deals with CoreWeave, Nebius, Nscale, IREN and others. The contracts let providers borrow against them to buy GPUs. Critics say this ties a small number of buyers and lenders together; providers say long contracts make the business stable.
Famous training clusters providers
Further reading
馃嚚馃嚟 SwitzerlandAlps
Swiss national supercomputer using Nvidia GH200 chips for AI and scientific research.Supercomputer 路 Other Nvidia GPUs 路 2024
馃嚭馃嚫 United StatesNCAR-Wyoming Supercomputing Center
A high-performance computing and data archival facility in Cheyenne, Wyoming.Supercomputer 路 Other Nvidia GPUs 路 2003
馃嚚馃嚟 SwitzerlandSwiss National Supercomputing Centre
Switzerland's national high-performance computing centre, run by ETH Zurich in Lugano.Supercomputer 路 1992
馃嚠馃嚦 IndiaCentre for Development of Advanced Computing
Indian government society building supercomputers and advanced computing systemsSupercomputer 路 Other and custom chips 路 1988
馃嚘馃嚭 AustraliaPawsey Supercomputing Centre
National research computing centre in Perth with supercomputers, storage and cloud.Supercomputer 路 2014
馃嚭馃嚫 United StatesSelene
Nvidia's own DGX SuperPOD supercomputer, built in three months in 2020Supercomputer 路 Nvidia Ampere 路 2020
馃嚫馃嚜 SwedenNational Supercomputer Centre in Sweden
Swedish national supercomputer centre in Link枚ping, running Triolith.Supercomputer
馃嚛馃嚜 GermanyGauss Centre for Supercomputing
Coordinates Germany's three national Tier-0 supercomputing centresSupercomputer 路 2007
馃嚞馃嚙 United KingdomDiRAC
UK supercomputing service for particle physics, astronomy and cosmology researchSupercomputer 路 2009
Photo credits on this page
The small pictures above come from Wikimedia Commons under free licences. Some are cropped to fit.
- CoreWeave: Wikideas1, CC0, source
- JUWELS: Forschungszentrum J眉lich, CC BY-SA 4.0, source
- NCAR-Wyoming Supercomputing Center: National Science Foundation, Public domain, source
- Barcelona Supercomputing Center: Herodotptlomeu, CC BY-SA 4.0, source
- CINECA: Carlo Dani, CC BY-SA 4.0, source
- Swiss National Supercomputing Centre: CSCS, CC BY-SA 3.0, source
- Alps: Marco Abram, CSCS, CC BY-SA 3.0, source
- Galileo: CINECA, CC BY 2.0, source
- Spanish Supercomputing Network: David Abi谩n, CC BY-SA 4.0, source
- National Center for Supercomputing Applications: en:User:Ragib / Site , User:Ragib, CC BY 2.5, source
- Texas Advanced Computing Center: UNM Research Data Services, CC BY-SA 2.0, source
- National Computational Infrastructure: Nick-D, CC BY-SA 3.0, source