News
DealsCorvex agrees USD 103 million take-or-pay deal to perform AI inference with Mid-Atlantic GPUsSovereignNeysa scales AI infrastructure in India with AMD Instinct MI350X GPUs on TytanInferenceMicrosoft Foundry adds Baseten, DeepSeek AI inference (MSFT:NASDAQ)ClustersDataOne and Rowan University sign engineering co-op program agreement for DataOne's Vineland AI factoryClustersLook inside Cal Poly's $3 million AI Factory (Video)DealsOpenAI Taps AMD Chips for 457 MW of AI Capacity in Six-Year TensorWave DealClustersCal Poly launches an AI data center on campus. Take a look insideInferenceArista takes aim at the next AI network battleground: distributed inferenceDealsOpenAI Signs 457 MW AI Data Center Deal as AMD Expands Its Challenge to NvidiaClustersBehind Dell AI Factory’s NVIDIA & AMD Enterprise ExpansionDealsCorvex agrees USD 103 million take-or-pay deal to perform AI inference with Mid-Atlantic GPUsSovereignNeysa scales AI infrastructure in India with AMD Instinct MI350X GPUs on TytanInferenceMicrosoft Foundry adds Baseten, DeepSeek AI inference (MSFT:NASDAQ)ClustersDataOne and Rowan University sign engineering co-op program agreement for DataOne's Vineland AI factoryClustersLook inside Cal Poly's $3 million AI Factory (Video)DealsOpenAI Taps AMD Chips for 457 MW of AI Capacity in Six-Year TensorWave DealClustersCal Poly launches an AI data center on campus. Take a look insideInferenceArista takes aim at the next AI network battleground: distributed inferenceDealsOpenAI Signs 457 MW AI Data Center Deal as AMD Expands Its Challenge to NvidiaClustersBehind Dell AI Factory’s NVIDIA & AMD Enterprise Expansion
EveryNeocloud

AI compute words

146 terms. The words of GPU clouds, clusters and AI infrastructure, in plain language.

GPU
Node
Rack
NVLink
InfiniBand
Cluster
Training
Inference
Tokens
Reserved capacity
Spot instances
Megawatt

A

Accelerator
Any chip built to speed up a particular kind of work, such as AI, beyond what an ordinary processor can do.
AI diffusion rule
A US rule issued in January 2025 to cap AI chip exports by country in three tiers, withdrawn before it took effect.
AI factory
A term popularised by Nvidia for a data centre built to produce AI output, measured in tokens, at industrial scale.
AI gigafactory
The European Union's plan for very large AI compute sites, each with around 100,000 advanced chips.
AIRAWAT
India's national AI supercomputer at C-DAC Pune, built on Nvidia A100 GPUs and ranked 75th on the TOP500 list in 2023.
Anchor tenant
The large customer whose long contract makes a new cluster or data centre possible to finance and build.

B

Bare metal
Renting a whole physical server with no virtualisation layer, so the customer gets every bit of its performance.
Batching
Processing several requests or data samples together, so the chip does more useful work per read of the model.
Behind-the-meter power
Electricity generated at or next to a data centre and supplied directly, rather than drawn through the public grid.
BF16
A 16-bit number format created at Google Brain that keeps the range of 32-bit numbers with less detail, now standard for AI training.
Burn-in testing
Running a new cluster hard for days or weeks before handing it over, to find faulty GPUs, cables and nodes early.

C

Capital expenditure
Money spent on long-lasting assets such as data centres, power equipment and servers, rather than on running costs.
Checkpointing
Saving the full state of a training job at intervals so it can resume from there after a failure.
Chipmaker cloud
A cloud run by a chip company to sell access to its own accelerators, rather than selling the chips alone.
Cluster
Many servers joined by a fast network and shared storage so they can work together on one job.
Co-packaged optics
Putting the optical parts of a network link right next to the switch chip, instead of in separate plug-in modules.
Cold start
The delay when a service must start a new machine and load a model before it can answer the first request.
Collective communication
Standard group operations, such as all-reduce, through which many GPUs share and combine results during training.
Colocation
Renting space, power and cooling in someone else's data centre to house your own servers.
Compute deal
A contract in which a customer commits to buy a set amount of AI compute over several years.
Confidential computing
Hardware that keeps data and models encrypted even while they are being processed, so even the cloud cannot see them.
Containers
A packaged bundle of an application and everything it needs to run, so it behaves the same on any machine.
Context window
The most tokens a model can take into account at once, covering the prompt, any documents and its reply.
CUDA
Nvidia's software platform for programming its GPUs, released in 2007 and a key reason most AI runs on Nvidia.
Custom accelerator
An AI chip designed by a cloud or internet company for its own data centres, such as Trainium, Maia or MTIA.

D

Data parallelism
Training on many GPUs at once by giving each a copy of the model and a different slice of the data, then combining what they learn.
Data residency
A rule or promise that data is stored and processed inside a particular country or region.
Decentralised compute
A network that pools GPUs owned by many independent operators, often coordinated and paid for with a cryptocurrency token.
Depreciation
The accounting charge that spreads the cost of hardware over the years it is expected to stay useful.
DGX and HGX
Nvidia's complete AI servers (DGX) and the eight-GPU baseboard it supplies to other server makers (HGX).
Direct-to-chip cooling
Liquid cooling where metal plates with fluid channels sit right on top of the GPUs and processors.
DPU
A network card with its own processors that handles networking, storage and security work so the main server does not have to.

E

Egress fees
Charges a cloud makes for data leaving its network, which can make moving large datasets between clouds costly.
Ethernet fabric
An AI cluster network built from Ethernet switches, tuned to carry GPU traffic with little loss and delay.
EuroHPC Joint Undertaking
The European Union body, set up in 2018, that co-funds Europe's large supercomputers, including JUPITER, LUMI and Leonardo.
Exaflop
A billion billion (10 to the power 18) calculations per second, the scale of today's fastest computers.
Expert parallelism
Placing the different experts of a mixture-of-experts model on different GPUs and routing each token to the right one.
Export controls
Government rules that restrict selling advanced chips and chip-making tools to certain countries, mainly applied by the United States to China.

F

Fine-tuning
Further training of an existing model on a smaller, specialised dataset to adapt it to a task or a company's data.
FLOPS
The number of calculations with decimal numbers a chip can do each second, the basic measure of compute.
Foundation model
A large model trained on broad data that can be adapted to many tasks; the most capable are called frontier models.
FP4
A 4-bit number format introduced for AI with Nvidia's Blackwell GPUs, mainly used to make inference faster and cheaper.
FP8
An 8-bit number format, supported since Nvidia's Hopper GPUs in 2022, that doubles AI speed over 16-bit formats.

G

Goodput
The share of a cluster's time that produces real progress on a job, after losses to failures, restarts and waiting.
GPU
A chip with thousands of small cores working in parallel, first built for graphics and now the main engine of AI.
GPU marketplace
A platform that lists GPUs from many owners and matches buyers to them, usually by price.
GPU-backed debt
Loans secured on GPUs and on the contracts to rent them out, used by neoclouds to buy hardware at scale.
GPU-hour
One GPU used for one hour, the basic unit in which cloud compute is priced and training runs are measured.
GPUDirect
Nvidia technologies that let network cards and storage move data straight into GPU memory, skipping the CPU.
Green500
A ranking, published with the TOP500 since 2007, that orders supercomputers by energy efficiency rather than raw speed.
Grid interconnection
The process of connecting a large new power user or generator to the electricity grid, often the slowest step in building AI compute.

H

HBM
Memory chips stacked on top of one another and placed right next to the processor, giving AI chips their huge memory speed.
High-performance computing
Using clusters and supercomputers to solve problems too big for ordinary computers, such as weather, physics and AI.
HPC hosting
Renting space, power and cooling for high-performance computing, a business many former bitcoin miners moved into.
HPCG
A supercomputer benchmark that tests memory and network speed, which better reflects many real applications than HPL.
HPL
The benchmark used to rank the TOP500, which measures how fast a computer solves a huge system of equations.
Hyperscaler
One of the few very large general cloud companies, such as AWS, Microsoft Azure and Google Cloud, that run data centres across the world.
Hypervisor
The software that creates and runs virtual machines on a physical server and keeps them apart.

I

Immersion cooling
Cooling servers by submerging them entirely in a fluid that does not conduct electricity.
IndiaAI compute portal
The online platform through which Indian start-ups, researchers and agencies apply for subsidised GPUs under the IndiaAI Mission.
IndiaAI Mission
India's national AI programme, approved in March 2024, which subsidises GPU access and funds Indian models and datasets.
Inference
Running a trained model to produce answers, images or other output for users, as opposed to training it.
Inference server
Software that loads a model onto GPUs and serves requests to it efficiently, handling batching and memory.
InfiniBand
A high-speed, low-delay network standard long used in supercomputers and now in many AI clusters.

J

Job scheduler
Software that queues jobs on a shared cluster and decides which run where and when.

K

Kubernetes
Open-source software, first released by Google in 2014, that runs and manages containers across many servers.
KV cache
Memory a language model keeps of the conversation so far, so it does not recompute it for every new token.

L

Latency
The time something takes to happen: a network message to arrive, or a model to answer.
Liquid cooling
Removing heat from chips with liquid rather than air, needed for the densest AI racks.
LPU
Groq's chip for running language models, which keeps model weights in fast on-chip memory to generate tokens quickly.

M

Megawatt
A million watts; data centres and AI clusters are measured by the megawatts of power their computers can draw.
Memory bandwidth
How many bytes per second a chip can read from its memory, often the true limit on how fast a model answers.
MFU
The share of a chip's peak arithmetic speed that a training job actually turns into useful model work.
Mixture of experts
A model design where each token is sent to only a few of many smaller sub-networks, so a huge model runs at the cost of a smaller one.
ML framework
The software library, such as PyTorch or JAX, in which AI models are written, trained and run.
MLPerf
A set of industry benchmarks, run by the MLCommons group, that compares how fast systems train and run AI models.
Model API
A web service that runs a model for you and charges by the token, so you never manage GPUs yourself.
Multi-Instance GPU
An Nvidia feature that splits one data centre GPU into as many as seven smaller, isolated GPUs.
Multi-tenancy
Serving many customers on shared infrastructure while keeping their work and data separate.

N

National Supercomputing Mission
India's programme, launched in 2015, to build PARAM supercomputers across research institutions, run by C-DAC and IISc.
Neocloud
A cloud company built mainly to rent out GPUs for AI, rather than to offer the full range of services a general cloud sells.
Node
One server in a cluster, in AI usually a box holding eight GPUs, two processors, memory and network cards.
Numerical precision
How many bits a computer uses to store each number; fewer bits mean faster sums but less exact results.
NVL72
Nvidia's rack-scale system that joins 72 Blackwell GPUs and 36 Grace CPUs over NVLink so they act as one.
NVLink
Nvidia's very fast link that lets GPUs share data with each other directly, much faster than through the network.
NVMe
A fast standard for connecting flash drives directly to a server, used for local scratch space and storage arrays in AI clusters.
NVSwitch
A switch chip that connects many GPUs over NVLink so that every GPU can talk to every other at full speed.

O

Object storage
Cheap, very large storage that keeps data as objects reached over the web, used for datasets, models and backups.
On-demand
Renting GPUs by the hour or second with no commitment, at the highest list price but with the most freedom.
Open-weight model
A model whose trained parameters are published for anyone to download and run, such as Llama, Mistral, Qwen or DeepSeek.
Optical transceiver
A small module that turns electrical signals into light and back, so data can travel over fibre between switches and servers.

P

Parallel file system
Storage spread across many servers so thousands of GPUs can read data and save checkpoints at the same time.
Parameters
The numbers inside a model that training adjusts; their count, in billions, is the usual measure of a model's size.
Petaflop
A thousand million million (10 to the power 15) calculations per second, one thousandth of an exaflop.
Pipeline parallelism
Splitting a model's layers into stages on different GPUs and passing data along them like an assembly line.
Pod
A standard building block of a cluster: a set number of racks and GPUs with their own network layer.
Post-training
The phases after pre-training that shape a model's behaviour, such as instruction tuning and reinforcement learning.
Power purchase agreement
A long contract to buy electricity from a specific power plant or wind or solar farm, often to back clean-energy claims.
Powered shell
A data centre building delivered with its structure and power connection ready, which the tenant then fits out.
Pre-training
The first, largest phase of training a model, on trillions of words or images, before it is adapted to tasks.
Prefill and decode
The two phases of answering a prompt: reading it all at once (prefill), then writing the reply one token at a time (decode).
PUE
A data centre's total power use divided by the power that reaches its computers; 1.0 would mean no overhead at all.

Q

Quantisation
Storing a model's numbers in fewer bits, such as 8 or 4 instead of 16, to make it smaller and faster to run.

R

Rack
The standard steel frame, about two metres tall, that holds servers, switches and power units in a data centre.
Rack density
How much electrical power a single rack draws, the number that most shapes an AI data centre's design.
Rail-optimised network
A cluster wiring scheme where GPU number one in every server joins one switch group, GPU two another, and so on.
RDMA
A way for one computer to read or write another's memory directly over the network, without involving its processor.
Region and availability zone
A cloud region is a geographic area of data centres; an availability zone is one isolated site or group of sites within it.
Research allocation
Compute time granted to researchers through a competitive call, rather than sold at a price.
Reserved capacity
Compute a customer commits to use for months or years in return for a lower price and guaranteed access.
RoCE
A way to run RDMA, the fast memory-to-memory transfer used in AI clusters, over ordinary Ethernet networks.
ROCm
AMD's open-source software platform for programming its Instinct accelerators, its answer to CUDA.

S

Sale-leaseback
Selling an asset, such as a data centre or a fleet of servers, to an investor and renting it back at once.
Scale-out network
The network that joins thousands of servers or racks into one cluster, usually InfiniBand or Ethernet.
Scale-up network
The fastest network tier, joining a small group of GPUs, such as a server or rack, so tightly that they work as one.
Scaling laws
Observed rules showing that models get predictably better as compute, data and size grow, the reason labs build ever larger clusters.
Serverless GPU
GPUs that start when a request arrives and shut down when it ends, billed by the second, so idle time costs nothing.
Service level agreement
The part of a contract that sets the availability and support a provider promises and the credits owed if it falls short.
Slurm
The open-source job scheduler that runs most of the world's supercomputers and many AI training clusters.
Sovereign AI
AI compute, data and models kept under a country's own control, inside its borders and under its laws.
Sparsity
Skipping calculations on values that are zero; chip makers quote speeds with sparsity that real models often cannot reach.
Speculative decoding
Speeding up text generation by letting a small model guess several tokens ahead and a large model check them at once.
Spine-leaf
A network layout where every server switch connects to every core switch, so any two servers are only a few hops apart.
Spot instances
Spare cloud capacity sold at a deep discount that the provider can take back at short notice.
SRAM
Very fast memory built into the processor itself, much quicker than HBM but holding far less.
Stargate
A joint venture announced in January 2025 by OpenAI, SoftBank, Oracle and MGX to build AI data centres in the United States.
Stranded power
Electricity capacity that exists at a site but has no nearby demand, such as a remote wind farm or a gas field.
Superchip
Nvidia's name for a module that joins its Grace processor and one or two GPUs with a fast link on one board.
Supercomputer
One of the fastest computers of its time, usually a very large cluster run by a national lab, university or company.
SuperPOD
Nvidia's reference design for a complete AI cluster built from DGX systems, network, storage and software.

T

Take-or-pay contract
A contract in which the buyer pays for the agreed capacity whether or not it actually uses it.
Tensor core
A specialised unit inside a GPU that multiplies small blocks of numbers in one step, doing most of the work in AI.
Tensor parallelism
Splitting each layer of a model across several GPUs so they compute it together, needing very fast links between them.
Test-time compute
Letting a model spend more computation while answering, for example by reasoning step by step, to get better results.
Time to first token
How long a user waits between sending a prompt and seeing the first word of the reply.
Tokens
The small chunks of text, roughly three-quarters of an English word each, that language models read and write.
Tokens per second
How fast a model produces text, measured for one user or for a whole server, the key speed figure for inference.
TOP500
The twice-yearly ranking of the world's 500 fastest supercomputers, published since 1993.
TPU
Google's own AI chip, used in its data centres since 2015 and rented to others through Google Cloud.
Training
The process of building an AI model by showing it vast amounts of data and adjusting its weights to reduce its errors.
Trainium
Amazon Web Services' own AI chips: Trainium for training models and Inferentia for running them.

U

Ultra Ethernet
An industry effort, begun in 2023, to update Ethernet for AI and high-performance computing.
Utilisation
How much of the time, or capacity, a provider's GPUs are actually doing paid or useful work.

V

Virtual machine
A software-defined computer that runs on a share of a physical server, rented by the hour from a cloud.

W

Wafer-scale engine
Cerebras's design that turns a whole silicon wafer into one giant chip instead of cutting it into many small ones.
Water usage effectiveness
The litres of water a data centre uses for cooling per kilowatt-hour of computing, a growing concern for AI.

The weekly digest

Every Monday: the provider of the day, new pages on EveryNeocloud, and the conferences and anniversaries of the week ahead.