vLLM
More info
- Creator
- UC Berkeley Sky Lab
- Released
- Owner
- PyTorch Foundation
- Type
- LLM inference server
- License
- Apache 2.0
- Website
- docs.vllm.ai
- Source
- github.com/vllm-project/vllm
High-throughput inference server for LLMs — the core innovation is PagedAttention, which manages the KV cache like virtual memory instead of a contiguous buffer.
From UC Berkeley (2023). PagedAttention splits the KV cache into fixed blocks and references them via a lookup table — enabling continuous batching and prefix sharing without wasting GPU memory on padding. Result: 2–10× higher throughput than HuggingFace TGI on the same hardware.
Supports most open models (Llama, Mistral, Qwen, DeepSeek), tensor parallelism across multiple GPUs, quantisation (AWQ, GPTQ, FP8) and speculative decoding. OpenAI-compatible API. The default choice for on-prem LLM serving in 2025.