vLLM

vLLM is a high-throughput inference and serving engine for large language models, maintained by the vLLM Project and originally developed at UC Berkeley’s Sky Computing Lab. Released in 2023 under the Apache 2.0 license, it has quickly become the dominant open-source engine for production-grade GPU serving, with over 84,500 GitHub stars, 2,900+ contributors, and 5.6 million monthly PyPI downloads.

This engine is built for teams that need to serve open models at scale on NVIDIA and AMD GPUs. It competes directly with SGLang, TensorRT-LLM, and TGI (Text Generation Inference), but distinguishes itself through two core innovations: PagedAttention and continuous batching. These aren’t academic niceties — they directly translate to more requests per second and lower cost per token on the same hardware.

vLLM is not a desktop toy. It is a serving engine designed for Linux, GPUs, and production deployments. If you are running Ollama for a personal assistant, vLLM is overkill. If you are serving a model to thousands of concurrent users, vLLM is the default choice.

vLLM exposes a Python library and a command-line server that loads a Hugging Face model and serves it over HTTP with an OpenAI-compatible API. The simplest path to a running endpoint is:

1pip install vllm
2python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-3.1-8B-Instruct

This spins up a server on localhost:8000 that accepts the same /v1/chat/completions and /v1/completions endpoints as OpenAI’s API. Any client written for OpenAI — LangChain, LlamaIndex, custom apps — works with zero code changes by swapping the base URL.

Under the hood, vLLM loads the model into GPU memory, partitions it across available devices if needed, and applies its custom memory manager. The server uses an async event loop, so requests queue efficiently without blocking. You can control the batch size, max model length, and scheduling policy via environment variables or a configuration file.

For programmatic use, vLLM also provides an LLM class:

1from vllm import LLM, SamplingParams
2
3llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")
4outputs = llm.generate(["Tell me a joke."], SamplingParams(temperature=0.7))

This is the same engine behind the server, so throughput characteristics match.

• NVIDIA GPUs: Full support with CUDA kernels, FlashAttention, and CUDA graphs. This is where vLLM is fastest.
• AMD GPUs: Supported via ROCm. Performance is competitive but lags behind NVIDIA on certain kernel optimizations.
• CPU: Limited support. vLLM can run on x86, ARM, and PowerPC CPUs, but it is not optimized for CPU-only inference. Use llama.cpp or Ollama for that.
• Apple Silicon: Not supported in practice. The engine is built for discrete GPUs, not unified memory.

vLLM’s PagedAttention eliminates KV cache fragmentation. Traditional serving reserves a fixed-size KV cache per request, wasting VRAM. PagedAttention allocates memory in fixed-size pages, so the same physical VRAM can hold many more concurrent sequences. This is the primary reason vLLM achieves 2-4x throughput over naive Hugging Face Transformers serving.

Continuous batching keeps the GPU busy: as one request finishes, a new one slots in immediately. Combined with chunked prefill and prefix caching, vLLM can handle long prompts and high concurrency without dropping throughput.

Quantization support is extensive: FP8, INT8, INT4, GPTQ, AWQ, GGUF, and more. This lets you fit larger models on fewer GPUs or increase batch sizes. vLLM also supports speculative decoding (EAGLE, n-gram) and tensor/pipeline parallelism for multi-GPU deployments.

vLLM is not fast on CPU. It is not designed for single-user interactive use on laptops. The overhead of its memory management and server stack is wasted on low-concurrency workloads. For those cases, Ollama or LM Studio are faster to set up and run.

This is the engine’s signature feature. Borrowed from operating system virtual memory, it pages the KV cache in 16- or 32-page blocks. Memory utilization goes from roughly 60% (naive caching) to near 100%. More concurrent requests fit on the same GPU, raising throughput.

Requests enter the inference batch as soon as a slot opens, rather than waiting for the entire batch to finish. This cuts tail latency and keeps utilization high. It is standard in modern serving engines, but vLLM implemented it first and remains one of the best-tuned.

The server exposes a drop-in replacement for the OpenAI chat and completions endpoints. Features like streaming, function/tool calling, and structured output (via xgrammar or guidance) work out of the box. This is the fastest path to a self-hosted OpenAI replacement.

vLLM supports FP8, INT8, INT4, AWQ, GPTQ, and GGUF. It also handles compressed-tensors, ModelOpt, TorchAO, and NVFP4. This flexibility lets you choose the right trade-off between speed and model quality.

Tensor parallelism splits model layers across GPUs; pipeline parallelism splits the model by layer depth. vLLM handles both automatically. For models like Llama 3.1 70B or 405B, you can run across multiple NVIDIA H100s or AMD MI250s.

You can enforce JSON or grammar-constrained generation. This is critical for agentic workflows where the model must produce parseable output. vLLM integrates with xgrammar and guidance.

Output tokens stream back via Server-Sent Events (SSE) or chunked transfer encoding. Latency to first token (TTFT) is low, typically under 1 second on 10k-token prompts with modern GPUs.

The primary use case. Teams serving chatbots, code assistants, or content generation APIs at scale choose vLLM to maximize throughput per dollar. It has benchmarked at over 230 tokens per second per user on DeepSeek V3.2 across multiple providers.

Enterprises that must keep data in-house point their existing OpenAI clients at a vLLM server. The same code, the same API contracts, but running on their own GPU cluster. No vendor lock-in, no data egress.

When a model exceeds single-GPU VRAM, vLLM’s tensor parallelism distributes it across multiple cards. This is the standard way to serve models like Llama 3.1 405B or Qwen 3.5 397B.

Researchers running experiments with large models use vLLM’s Python API for batch inference on thousands of prompts. The throughput improvements cut experiment time from hours to minutes.

• Single-user desktop inference: Use Ollama or LM Studio for a simpler setup.
• CPU-only or Apple Silicon: Use llama.cpp.
• Fine-tuning: vLLM is for inference, not training. Use Hugging Face Transformers or Axolotl.

1pip install vllm

Requires Python 3.10 or later, a Linux system with an NVIDIA GPU (or AMD with ROCm). Docker images are available for quick deployment.

1. Install vLLM.
2. Start the server:

1python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-3.1-8B-Instruct --dtype auto --max-model-len 8192

1. Query it with curl:

1curl http://localhost:8000/v1/chat/completions \
2  -H "Content-Type: application/json" \
3  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","messages":[{"role":"user","content":"Hello!"}]}'

That’s it. The model is served, the API is live, and any OpenAI client can point to it.

• A GPU with at least 16 GB VRAM for 7B models, 48 GB for 70B (or multi-GPU).
• A quantized model if VRAM is tight.
• An existing OpenAI client or curl for testing.

• Full docs: [docs.vllm.ai](https://docs.vllm.ai)
• GitHub: [github.com/vllm-project/vllm](https://github.com/vllm-project/vllm)
• Community Slack and user forum linked from the docs.

SGLang focuses on structured generation and a Python-native programming model. It is competitive on throughput for many benchmarks. Choose vLLM if you need the most battle-tested engine with the largest hardware support (NVIDIA + AMD) and the widest quantization ecosystem. Choose SGLang if you prioritize Python-based control over generation flow and are okay with narrower GPU support.

Ollama is built for desktop ease: one command to run a model, no GPU tuning required. It uses llama.cpp under the hood. Choose Ollama for local experimentation, quick demos, or CPU/Apple Silicon. Choose vLLM for production serving at scale on GPUs. They are not interchangeable — vLLM is an order of magnitude more complex to tune but delivers 3-5x the throughput on the same NVIDIA GPU.

TensorRT-LLM is NVIDIA’s proprietary engine. It can beat vLLM on raw throughput in some configurations, especially on Hopper and Blackwell GPUs with custom kernels. But it is harder to set up, requires model conversion to TRT format, and lacks the community breadth of vLLM. Choose TensorRT-LLM if you must squeeze every last token per second and can invest in NVIDIA-only tooling. Choose vLLM if you want open-source, multi-vendor hardware support, and a faster iteration cycle.