vLLM
vLLM is an open-source library under Apache-2.0 for fast, efficient serving of language models, started at UC Berkeley's Sky Computing Lab and built by over 2,000 contributors. It serves an OpenAI-compatible API plus the Anthropic Messages API, supports more than 200 model architectures from Hugging Face, handles tool calling, and spreads large models across many GPUs with tensor, pipeline, data, expert and context parallelism.
What does vLLM do?
vLLM is "a fast and easy-to-use library for LLM inference and serving", with the tagline "Easy, fast, and cheap LLM serving for everyone." It serves an "OpenAI-compatible API server, plus Anthropic Messages API and gRPC support", with tool calling, reasoning parsers and structured outputs. The README says it supports more than 200 model architectures on Hugging Face, including multimodal and embedding models.
Its throughput comes from PagedAttention, continuous batching, chunked prefill, prefix caching and speculative decoding. Quantisation covers FP8, MXFP8 and MXFP4, NVFP4, INT8, INT4, GPTQ and AWQ, GGUF and more. Distributed inference uses "tensor, pipeline, data, expert, and context parallelism", and multi-LoRA serving handles many fine-tuned adapters at once. It runs on NVIDIA, AMD and Intel GPUs and x86, ARM and PowerPC CPUs, with plugins for Google TPUs, Intel Gaudi, Huawei Ascend and Apple Silicon.
Key facts
- Licence: Apache-2.0, with no added conditions.
- An OpenAI-compatible API server, plus the Anthropic Messages API and gRPC; tool calling, reasoning parsers and structured outputs.
- 200+ model architectures on Hugging Face, including multimodal and embedding models.
- Quantisation: FP8, MXFP8 and MXFP4, NVFP4, INT8, INT4, GPTQ and AWQ, GGUF, compressed-tensors and others.
- Tensor, pipeline, data, expert and context parallelism for distributed inference; multi-LoRA for dense and mixture-of-experts layers.
- PagedAttention, continuous batching, chunked prefill, prefix caching and speculative decoding.
- Hardware: NVIDIA, AMD and Intel GPUs; x86, ARM and PowerPC CPUs; plugins for TPUs, Gaudi, Ascend, Apple Silicon and others.
- Install: uv pip install vllm. Over 2,000 contributors, begun at UC Berkeley's Sky Computing Lab. Latest tagged release when read: v0.29.0 on 2026-09-09; 92,045 stars on 2026-09-17.
What does it replace, and where does it fall short?
vLLM replaces the OpenAI API for an organisation serving open models to many users at once from its own GPUs, where throughput and cost per request decide the choice. It speaks both the OpenAI and Anthropic APIs, so existing code moves with a URL change. It is the pick for serving at scale on the open-source alternatives to the OpenAI API page.
Where it falls short: it is built for servers with GPUs and for people who operate them, not for a laptop; the README states no operating system list or install requirements beyond pip. It serves language models, not images or speech. For one machine and a quick start, Ollama; for every modality in one server, LocalAI.
How does vLLM run?
Install with uv pip install vllm, or pip, or build from source, then start the OpenAI-compatible server with a Hugging Face model. Multi-GPU and multi-node serving are configured with the parallelism options. The README describes no hosted or paid tier; partnership enquiries go to a project address.
Who is vLLM for?
A team with GPUs serving an open model to an application or many users, that needs throughput, batching and multi-GPU support behind an OpenAI-compatible API. One developer on a laptop should use Ollama or llama.cpp.
What limits does the README state?
None stated in the README.
Questions people ask
Is vLLM open source?
Yes. vLLM is Apache-2.0 with no added conditions. It began at UC Berkeley's Sky Computing Lab and, by its README, has more than 2,000 contributors. No paid tier is described.
Does vLLM work with the OpenAI API?
Yes. It serves an OpenAI-compatible API server, plus the Anthropic Messages API and gRPC, with tool calling and structured outputs. Code written against the OpenAI SDK can point at a vLLM server by changing the base URL.
Can vLLM run on a single GPU or a CPU?
It supports NVIDIA, AMD and Intel GPUs and x86, ARM and PowerPC CPUs, and plugins cover TPUs, Gaudi, Ascend and Apple Silicon. Its design, batching and parallelism across GPUs, is aimed at serving many requests; on one laptop, Ollama or llama.cpp is simpler.
How does vLLM compare with Ollama?
Ollama gets one model running on one machine in a minute. vLLM serves models to many users with continuous batching, quantisation and parallelism across GPUs, behind OpenAI and Anthropic-compatible APIs. The open-source alternatives to the OpenAI API page compares both with LocalAI and llama.cpp.
Which models can vLLM serve?
More than 200 model architectures on Hugging Face, by the README, including multimodal models such as LLaVA and Qwen-VL and embedding and retrieval models. Quantised formats including GPTQ, AWQ, FP8 and GGUF are supported.
Sources
- vLLM README and Apache-2.0 LICENSE: github.com/vllm-project/vllm, read 2026-09-17.
Compared with the others
On the open-source alternatives to the OpenAI API page, vLLM is the pick for serving many users from gpus. Apache-2.0, OpenAI and Anthropic-compatible APIs, 200+ architectures, wide quantisation, batching and parallelism across many GPUs.
Also on that page: Ollama for the quickest local start, LocalAI for every modality behind openai-compatible apis, llama.cpp for the engine on any hardware.
More local model runtimes
An MIT tool for downloading and running open models on your own machine with one command, built on llama.cpp, with a REST API, Python and JavaScript libraries, and a launcher that connects coding agents to local models.
The MIT C and C++ inference engine behind much of local AI, running GGUF models with 1.5 to 8-bit quantisation on CPUs, Apple Silicon and 17 hardware backends, with an OpenAI-compatible server and a web interface built in.
An MIT engine that runs text, vision, voice, image and video models on any hardware, no GPU required, behind drop-in OpenAI, Anthropic and ElevenLabs-compatible APIs, over 60+ backends including llama.cpp, vLLM and whisper.cpp.
Added September 17, 2026. Every claim above comes from the project's README, LICENSE or model card, read on September 17, 2026, or from the GitHub API on the date shown in the panel. Found an error? Write to reveneau@licheo.com and it is fixed in the next weekly pass. Repository: github.com/vllm-project/vllm.
Running one of these inside your own environment, with your own data and your own security rules, is the kind of work Reveneau does. Read how a forward deployed engagement works.