TensorRT-LLM
Nvidia's inference library for language models. It compiles the model into an optimized engine locked to a particular GPU generation, batch size and precision — and is in return very hard to beat on throughput.
The gain comes from compilation seeing the whole graph: attention, normalization and activation are fused into fewer kernel launches, weights are quantized to FP8 or INT4, and tensor parallelism across multiple cards is baked into the engine. The runtime does in-flight batching, meaning new requests are slotted into batches already running instead of waiting for the slowest one to finish. Serving is usually done through Triton Inference Server. The drawback is the workflow: every model, quantization level and hardware combination requires its own build step that takes time, and the engine is not portable. vLLM is often chosen instead simply because it starts.