Quantisation
Storing a model's numbers in fewer bits, such as 8 or 4 instead of 16, to make it smaller and faster to run.
Also: quantization
Quantisation converts a trained model's weights, and sometimes its working values, to a lower-precision format. A 70-billion-parameter model needs about 140 GB in 16-bit form but about 35 GB in 4-bit form, so it can fit on fewer GPUs and generate tokens faster. The cost is some loss of accuracy, which careful methods keep small. Most inference providers serve quantised versions of open models, and some publish which precision they use, since it affects output quality.