KV cache
Memory a language model keeps of the conversation so far, so it does not recompute it for every new token.
Also: key-value cache
When a transformer model generates text, each new token needs to look back at all earlier tokens. Rather than recompute them, the model stores intermediate values, called keys and values, in GPU memory. This KV cache grows with the length of the conversation and the number of users, and for long contexts can take more memory than the model itself. Managing it well, by paging it like computer memory, compressing it, or moving it to cheaper storage and back, is one of the main ways inference providers raise throughput.