AI compute words
146 terms. The words of GPU clouds, clusters and AI infrastructure, in plain language.
A
- Accelerator
- Any chip built to speed up a particular kind of work, such as AI, beyond what an ordinary processor can do.
- AI diffusion rule
- A US rule issued in January 2025 to cap AI chip exports by country in three tiers, withdrawn before it took effect.
- AI factory
- A term popularised by Nvidia for a data centre built to produce AI output, measured in tokens, at industrial scale.
- AI gigafactory
- The European Union's plan for very large AI compute sites, each with around 100,000 advanced chips.
- AIRAWAT
- India's national AI supercomputer at C-DAC Pune, built on Nvidia A100 GPUs and ranked 75th on the TOP500 list in 2023.
- Anchor tenant
- The large customer whose long contract makes a new cluster or data centre possible to finance and build.
B
- Bare metal
- Renting a whole physical server with no virtualisation layer, so the customer gets every bit of its performance.
- Batching
- Processing several requests or data samples together, so the chip does more useful work per read of the model.
- Behind-the-meter power
- Electricity generated at or next to a data centre and supplied directly, rather than drawn through the public grid.
- BF16
- A 16-bit number format created at Google Brain that keeps the range of 32-bit numbers with less detail, now standard for AI training.
- Burn-in testing
- Running a new cluster hard for days or weeks before handing it over, to find faulty GPUs, cables and nodes early.
C
- Capital expenditure
- Money spent on long-lasting assets such as data centres, power equipment and servers, rather than on running costs.
- Checkpointing
- Saving the full state of a training job at intervals so it can resume from there after a failure.
- Chipmaker cloud
- A cloud run by a chip company to sell access to its own accelerators, rather than selling the chips alone.
- Cluster
- Many servers joined by a fast network and shared storage so they can work together on one job.
- Co-packaged optics
- Putting the optical parts of a network link right next to the switch chip, instead of in separate plug-in modules.
- Cold start
- The delay when a service must start a new machine and load a model before it can answer the first request.
- Collective communication
- Standard group operations, such as all-reduce, through which many GPUs share and combine results during training.
- Colocation
- Renting space, power and cooling in someone else's data centre to house your own servers.
- Compute deal
- A contract in which a customer commits to buy a set amount of AI compute over several years.
- Confidential computing
- Hardware that keeps data and models encrypted even while they are being processed, so even the cloud cannot see them.
- Containers
- A packaged bundle of an application and everything it needs to run, so it behaves the same on any machine.
- Context window
- The most tokens a model can take into account at once, covering the prompt, any documents and its reply.
- CUDA
- Nvidia's software platform for programming its GPUs, released in 2007 and a key reason most AI runs on Nvidia.
- Custom accelerator
- An AI chip designed by a cloud or internet company for its own data centres, such as Trainium, Maia or MTIA.
D
- Data parallelism
- Training on many GPUs at once by giving each a copy of the model and a different slice of the data, then combining what they learn.
- Data residency
- A rule or promise that data is stored and processed inside a particular country or region.
- Decentralised compute
- A network that pools GPUs owned by many independent operators, often coordinated and paid for with a cryptocurrency token.
- Depreciation
- The accounting charge that spreads the cost of hardware over the years it is expected to stay useful.
- DGX and HGX
- Nvidia's complete AI servers (DGX) and the eight-GPU baseboard it supplies to other server makers (HGX).
- Direct-to-chip cooling
- Liquid cooling where metal plates with fluid channels sit right on top of the GPUs and processors.
- DPU
- A network card with its own processors that handles networking, storage and security work so the main server does not have to.
E
- Egress fees
- Charges a cloud makes for data leaving its network, which can make moving large datasets between clouds costly.
- Ethernet fabric
- An AI cluster network built from Ethernet switches, tuned to carry GPU traffic with little loss and delay.
- EuroHPC Joint Undertaking
- The European Union body, set up in 2018, that co-funds Europe's large supercomputers, including JUPITER, LUMI and Leonardo.
- Exaflop
- A billion billion (10 to the power 18) calculations per second, the scale of today's fastest computers.
- Expert parallelism
- Placing the different experts of a mixture-of-experts model on different GPUs and routing each token to the right one.
- Export controls
- Government rules that restrict selling advanced chips and chip-making tools to certain countries, mainly applied by the United States to China.
F
- Fine-tuning
- Further training of an existing model on a smaller, specialised dataset to adapt it to a task or a company's data.
- FLOPS
- The number of calculations with decimal numbers a chip can do each second, the basic measure of compute.
- Foundation model
- A large model trained on broad data that can be adapted to many tasks; the most capable are called frontier models.
- FP4
- A 4-bit number format introduced for AI with Nvidia's Blackwell GPUs, mainly used to make inference faster and cheaper.
- FP8
- An 8-bit number format, supported since Nvidia's Hopper GPUs in 2022, that doubles AI speed over 16-bit formats.
G
- Goodput
- The share of a cluster's time that produces real progress on a job, after losses to failures, restarts and waiting.
- GPU
- A chip with thousands of small cores working in parallel, first built for graphics and now the main engine of AI.
- GPU marketplace
- A platform that lists GPUs from many owners and matches buyers to them, usually by price.
- GPU-backed debt
- Loans secured on GPUs and on the contracts to rent them out, used by neoclouds to buy hardware at scale.
- GPU-hour
- One GPU used for one hour, the basic unit in which cloud compute is priced and training runs are measured.
- GPUDirect
- Nvidia technologies that let network cards and storage move data straight into GPU memory, skipping the CPU.
- Green500
- A ranking, published with the TOP500 since 2007, that orders supercomputers by energy efficiency rather than raw speed.
- Grid interconnection
- The process of connecting a large new power user or generator to the electricity grid, often the slowest step in building AI compute.
H
- HBM
- Memory chips stacked on top of one another and placed right next to the processor, giving AI chips their huge memory speed.
- High-performance computing
- Using clusters and supercomputers to solve problems too big for ordinary computers, such as weather, physics and AI.
- HPC hosting
- Renting space, power and cooling for high-performance computing, a business many former bitcoin miners moved into.
- HPCG
- A supercomputer benchmark that tests memory and network speed, which better reflects many real applications than HPL.
- HPL
- The benchmark used to rank the TOP500, which measures how fast a computer solves a huge system of equations.
- Hyperscaler
- One of the few very large general cloud companies, such as AWS, Microsoft Azure and Google Cloud, that run data centres across the world.
- Hypervisor
- The software that creates and runs virtual machines on a physical server and keeps them apart.
I
- Immersion cooling
- Cooling servers by submerging them entirely in a fluid that does not conduct electricity.
- IndiaAI compute portal
- The online platform through which Indian start-ups, researchers and agencies apply for subsidised GPUs under the IndiaAI Mission.
- IndiaAI Mission
- India's national AI programme, approved in March 2024, which subsidises GPU access and funds Indian models and datasets.
- Inference
- Running a trained model to produce answers, images or other output for users, as opposed to training it.
- Inference server
- Software that loads a model onto GPUs and serves requests to it efficiently, handling batching and memory.
- InfiniBand
- A high-speed, low-delay network standard long used in supercomputers and now in many AI clusters.
J
- Job scheduler
- Software that queues jobs on a shared cluster and decides which run where and when.
K
- Kubernetes
- Open-source software, first released by Google in 2014, that runs and manages containers across many servers.
- KV cache
- Memory a language model keeps of the conversation so far, so it does not recompute it for every new token.
L
- Latency
- The time something takes to happen: a network message to arrive, or a model to answer.
- Liquid cooling
- Removing heat from chips with liquid rather than air, needed for the densest AI racks.
- LPU
- Groq's chip for running language models, which keeps model weights in fast on-chip memory to generate tokens quickly.
M
- Megawatt
- A million watts; data centres and AI clusters are measured by the megawatts of power their computers can draw.
- Memory bandwidth
- How many bytes per second a chip can read from its memory, often the true limit on how fast a model answers.
- MFU
- The share of a chip's peak arithmetic speed that a training job actually turns into useful model work.
- Mixture of experts
- A model design where each token is sent to only a few of many smaller sub-networks, so a huge model runs at the cost of a smaller one.
- ML framework
- The software library, such as PyTorch or JAX, in which AI models are written, trained and run.
- MLPerf
- A set of industry benchmarks, run by the MLCommons group, that compares how fast systems train and run AI models.
- Model API
- A web service that runs a model for you and charges by the token, so you never manage GPUs yourself.
- Multi-Instance GPU
- An Nvidia feature that splits one data centre GPU into as many as seven smaller, isolated GPUs.
- Multi-tenancy
- Serving many customers on shared infrastructure while keeping their work and data separate.
N
- National Supercomputing Mission
- India's programme, launched in 2015, to build PARAM supercomputers across research institutions, run by C-DAC and IISc.
- Neocloud
- A cloud company built mainly to rent out GPUs for AI, rather than to offer the full range of services a general cloud sells.
- Node
- One server in a cluster, in AI usually a box holding eight GPUs, two processors, memory and network cards.
- Numerical precision
- How many bits a computer uses to store each number; fewer bits mean faster sums but less exact results.
- NVL72
- Nvidia's rack-scale system that joins 72 Blackwell GPUs and 36 Grace CPUs over NVLink so they act as one.
- NVLink
- Nvidia's very fast link that lets GPUs share data with each other directly, much faster than through the network.
- NVMe
- A fast standard for connecting flash drives directly to a server, used for local scratch space and storage arrays in AI clusters.
- NVSwitch
- A switch chip that connects many GPUs over NVLink so that every GPU can talk to every other at full speed.
O
- Object storage
- Cheap, very large storage that keeps data as objects reached over the web, used for datasets, models and backups.
- On-demand
- Renting GPUs by the hour or second with no commitment, at the highest list price but with the most freedom.
- Open-weight model
- A model whose trained parameters are published for anyone to download and run, such as Llama, Mistral, Qwen or DeepSeek.
- Optical transceiver
- A small module that turns electrical signals into light and back, so data can travel over fibre between switches and servers.
P
- Parallel file system
- Storage spread across many servers so thousands of GPUs can read data and save checkpoints at the same time.
- Parameters
- The numbers inside a model that training adjusts; their count, in billions, is the usual measure of a model's size.
- Petaflop
- A thousand million million (10 to the power 15) calculations per second, one thousandth of an exaflop.
- Pipeline parallelism
- Splitting a model's layers into stages on different GPUs and passing data along them like an assembly line.
- Pod
- A standard building block of a cluster: a set number of racks and GPUs with their own network layer.
- Post-training
- The phases after pre-training that shape a model's behaviour, such as instruction tuning and reinforcement learning.
- Power purchase agreement
- A long contract to buy electricity from a specific power plant or wind or solar farm, often to back clean-energy claims.
- Powered shell
- A data centre building delivered with its structure and power connection ready, which the tenant then fits out.
- Pre-training
- The first, largest phase of training a model, on trillions of words or images, before it is adapted to tasks.
- Prefill and decode
- The two phases of answering a prompt: reading it all at once (prefill), then writing the reply one token at a time (decode).
- PUE
- A data centre's total power use divided by the power that reaches its computers; 1.0 would mean no overhead at all.
Q
- Quantisation
- Storing a model's numbers in fewer bits, such as 8 or 4 instead of 16, to make it smaller and faster to run.
R
- Rack
- The standard steel frame, about two metres tall, that holds servers, switches and power units in a data centre.
- Rack density
- How much electrical power a single rack draws, the number that most shapes an AI data centre's design.
- Rail-optimised network
- A cluster wiring scheme where GPU number one in every server joins one switch group, GPU two another, and so on.
- RDMA
- A way for one computer to read or write another's memory directly over the network, without involving its processor.
- Region and availability zone
- A cloud region is a geographic area of data centres; an availability zone is one isolated site or group of sites within it.
- Research allocation
- Compute time granted to researchers through a competitive call, rather than sold at a price.
- Reserved capacity
- Compute a customer commits to use for months or years in return for a lower price and guaranteed access.
- RoCE
- A way to run RDMA, the fast memory-to-memory transfer used in AI clusters, over ordinary Ethernet networks.
- ROCm
- AMD's open-source software platform for programming its Instinct accelerators, its answer to CUDA.
S
- Sale-leaseback
- Selling an asset, such as a data centre or a fleet of servers, to an investor and renting it back at once.
- Scale-out network
- The network that joins thousands of servers or racks into one cluster, usually InfiniBand or Ethernet.
- Scale-up network
- The fastest network tier, joining a small group of GPUs, such as a server or rack, so tightly that they work as one.
- Scaling laws
- Observed rules showing that models get predictably better as compute, data and size grow, the reason labs build ever larger clusters.
- Serverless GPU
- GPUs that start when a request arrives and shut down when it ends, billed by the second, so idle time costs nothing.
- Service level agreement
- The part of a contract that sets the availability and support a provider promises and the credits owed if it falls short.
- Slurm
- The open-source job scheduler that runs most of the world's supercomputers and many AI training clusters.
- Sovereign AI
- AI compute, data and models kept under a country's own control, inside its borders and under its laws.
- Sparsity
- Skipping calculations on values that are zero; chip makers quote speeds with sparsity that real models often cannot reach.
- Speculative decoding
- Speeding up text generation by letting a small model guess several tokens ahead and a large model check them at once.
- Spine-leaf
- A network layout where every server switch connects to every core switch, so any two servers are only a few hops apart.
- Spot instances
- Spare cloud capacity sold at a deep discount that the provider can take back at short notice.
- SRAM
- Very fast memory built into the processor itself, much quicker than HBM but holding far less.
- Stargate
- A joint venture announced in January 2025 by OpenAI, SoftBank, Oracle and MGX to build AI data centres in the United States.
- Stranded power
- Electricity capacity that exists at a site but has no nearby demand, such as a remote wind farm or a gas field.
- Superchip
- Nvidia's name for a module that joins its Grace processor and one or two GPUs with a fast link on one board.
- Supercomputer
- One of the fastest computers of its time, usually a very large cluster run by a national lab, university or company.
- SuperPOD
- Nvidia's reference design for a complete AI cluster built from DGX systems, network, storage and software.
T
- Take-or-pay contract
- A contract in which the buyer pays for the agreed capacity whether or not it actually uses it.
- Tensor core
- A specialised unit inside a GPU that multiplies small blocks of numbers in one step, doing most of the work in AI.
- Tensor parallelism
- Splitting each layer of a model across several GPUs so they compute it together, needing very fast links between them.
- Test-time compute
- Letting a model spend more computation while answering, for example by reasoning step by step, to get better results.
- Time to first token
- How long a user waits between sending a prompt and seeing the first word of the reply.
- Tokens
- The small chunks of text, roughly three-quarters of an English word each, that language models read and write.
- Tokens per second
- How fast a model produces text, measured for one user or for a whole server, the key speed figure for inference.
- TOP500
- The twice-yearly ranking of the world's 500 fastest supercomputers, published since 1993.
- TPU
- Google's own AI chip, used in its data centres since 2015 and rented to others through Google Cloud.
- Training
- The process of building an AI model by showing it vast amounts of data and adjusting its weights to reduce its errors.
- Trainium
- Amazon Web Services' own AI chips: Trainium for training models and Inferentia for running them.
U
- Ultra Ethernet
- An industry effort, begun in 2023, to update Ethernet for AI and high-performance computing.
- Utilisation
- How much of the time, or capacity, a provider's GPUs are actually doing paid or useful work.
V
- Virtual machine
- A software-defined computer that runs on a share of a physical server, rented by the hour from a cloud.
W
- Wafer-scale engine
- Cerebras's design that turns a whole silicon wafer into one giant chip instead of cutting it into many small ones.
- Water usage effectiveness
- The litres of water a data centre uses for cooling per kilowatt-hour of computing, a growing concern for AI.