Memory bandwidth
How many bytes per second a chip can read from its memory, often the true limit on how fast a model answers.
Memory bandwidth is the rate at which a processor can move data in and out of its memory, given in terabytes per second. When a language model generates text, every new token requires reading most of the model's weights from memory, so the speed of generation is often set by bandwidth rather than by arithmetic. This is why each new GPU generation advertises its HBM speed alongside its FLOPS, and why some inference chips keep weights in on-chip SRAM. Training, which processes many tokens at once, leans more on arithmetic.