Pre-training
The first, largest phase of training a model, on trillions of words or images, before it is adapted to tasks.
Pre-training is where a foundation model learns general patterns from a huge dataset, usually by predicting the next token of text from the internet, books and code. It accounts for most of the compute in a model's life before deployment: frontier models are reported to use on the order of 10 to the power 25 floating-point operations or more. Pre-training needs the largest and most reliable clusters, and is the phase for which the biggest compute deals are signed.