vLLM
High-throughput LLM inference and serving engine
Highlights
- PagedAttention memory management
- Leading continuous-batch throughput
- OpenAI-compatible server
Use cases
- Production LLM API serving
- Private high-concurrency inference
- Model-serving gateway
Review
vLLM is the de facto performance benchmark for open-source LLM serving. With PagedAttention it manages KV cache like virtual memory, thoroughly solving memory fragmentation under long contexts and concurrent requests. It directly competes with TGI and TensorRT-LLM and leads throughput on most public benchmarks.
Core capabilities include PagedAttention, continuous batching, tensor and pipeline parallelism, quantized inference (AWQ, GPTQ, FP8), an OpenAI-compatible HTTP server, and broad adaptation to mainstream open models (Llama, Qwen, Mistral, DeepSeek). Recent versions add multimodal inference and hot-loaded LoRA adapters.
The highlight is that throughput and memory utilization are nearly unmatched in open source, with a deployment form close to start a server and use it. The downside is high hardware requirements (A100/H100-class GPUs recommended), the advantage shrinks on small-VRAM cards, and adaptation for niche models often lags behind official weight releases.
Use cases: production-grade LLM API services, private high-concurrency inference, model-serving proxy gateways. For consumer-grade GPUs at home, llama.cpp or Ollama are better picks.