Open source · ★ 93,741 GitHub stars (as of 2026-10-14)
vLLM
vLLM is an open-source library for fast, efficient LLM inference and serving.
Overview
vLLM is a library for LLM inference and serving that started at the UC Berkeley Sky Computing Lab and is now maintained jointly by contributors from academia and industry. It supports a wide range of model architectures from Hugging Face.
It manages attention key-value memory efficiently with PagedAttention and groups incoming requests with continuous batching. It includes prefix caching, a variety of quantization formats, speculative decoding, and distributed inference features such as tensor, pipeline and expert parallelism. It provides an OpenAI-compatible API server and the Anthropic Messages API, and supports NVIDIA, AMD and Intel GPUs, various CPUs, and hardware plugins.
Suited to developers and ML engineers who want to serve open models on their own infrastructure.
What it is used for
- Serve open models through an OpenAI-compatible API server.
- Serve quantized models with less memory.
- Split a model across multiple GPUs for distributed inference.
- Build applications that need structured output, tool calling and multi-LoRA serving.
Pricing and license
Check the official site
Getting started
- Install with uv pip install vllm (pip also works).
- Download precompiled kernels with vllm download-kernels to speed up startup.
- Follow the Quickstart Guide in the official documentation to run a model.
Links
Related terms and companies
Recent mentions in briefings
- vLLM Semantic Router team open-sources 'Decision 3.0,' multimodal decision models that also analyze images (report) 2026-10-14
- Precision Numerics releases Boole-1.7B, a 247MB open-weight 1-bit quantization of Qwen3-1.7B (per community post) 2026-10-13
- vLLM v0.31.0 released, focused on DeepSeek-V4.1-Flash inference performance 2026-10-07
Sources
Compiled from public sources (official pages, READMEs). Pricing and features change, so check the official site.