Inference server
Software that loads a model onto GPUs and serves requests to it efficiently, handling batching and memory.
Also: vLLM, TensorRT-LLM, SGLang
An inference server turns a model file into a running service. It loads the weights, splits them across GPUs if needed, batches incoming requests, manages the KV cache and streams tokens back. Open-source options include vLLM, which began at the University of California, Berkeley, in 2023, and SGLang; Nvidia offers TensorRT-LLM and Dynamo. Much of the speed difference between inference providers serving the same open model comes from how well they tune this layer.