LMDeploy and vLLM compared side by side on GitHub stars, downloads, language, license, capabilities, strengths and trade-offs.
TL;DR
Choose vLLM if you need broad hardware flexibility, expansive model architecture support, and a massive production ecosystem. Pick LMDeploy if you run dedicated NVIDIA hardware and want built-in weight and KV cache quantization coupled with the high-throughput TurboMind C++ backend.
Add an Engine
Slot 3 of 3
Comparing inference engines is the process of evaluating two or three competing tools side by side on live community signals, technical capabilities, language, license, and trade-offs so a team can pick the one that fits its hardware and workload.
Start with the constraints that are not negotiable: the hardware you already run on, the license your legal team will sign off on, and the kind of workload you need to serve. An engine that cannot use your GPU or fit your model is not really an option, no matter how popular it is.
Then look at live signals: GitHub stars and contributors show whether the project is gathering momentum, PyPI downloads show whether teams are actually shipping with it, and last-commit date shows whether the maintainers are still around. Capability flags like OpenAI-compatible API, GPU support, quantization, continuous batching, and multi-GPU narrow the field to engines that match your job-to-be-done.
Finish by reading the strengths and trade-offs columns side by side. The simplest engine that covers your real requirements almost always beats the most powerful one. Copy the share link once you have a comparison you trust so you can revisit it during planning.
Decisions about inference engines rarely happen in isolation. Pair this comparator with the directories and benchmarks that ground the rest of the stack.
Live data from GitHub and the package registries, refreshed every day.
GitHub stars
Contributors
Forks
PyPI downloads per month
Star history sampled from the GitHub stargazer timeline and extended by our daily sync.
LMDeploy and vLLM are two open-source inference frameworks designed to tackle the primary operational bottleneck in production AI: serving large language models with high throughput and low memory waste. Both frameworks use continuous batching and paged memory algorithms to maximize GPU compute, and both expose OpenAI-compatible web servers. However, they approach architecture and ecosystem scale from very different angles.
While both platforms target production LLM serving, their engineering trade-offs diverge across core runtimes, hardware targets, and ecosystem adoption.
| Feature | LMDeploy | vLLM |
|---|---|---|
| Maintainer | InternLM Team | vLLM Project (originated at UC Berkeley) |
| Primary Runtime | Mixed (C++/CUDA TurboMind + PyTorch engine) | Python / C++ (PagedAttention, CUDA/HIP) |
| Hardware Focus |
Pick LMDeploy if you need: High-throughput serving with built-in quantization
Full LMDeploy profileOpen and closed model rankings with benchmarks, context windows, modalities, and live API prices.
Straight answers to the questions we hear most often.
Can't find what you're looking for? Book a Discovery Call
It depends on the workload and configuration. LMDeploy's published benchmarks state that its TurboMind engine can deliver up to 1.8x higher request throughput than vLLM on supported NVIDIA setups, particularly when using combined 4-bit weight and KV cache quantization. However, vLLM performance is constantly improving across architectures, and results vary depending on prompt lengths and batch sizes.
Yes, in most cases. Both LMDeploy and vLLM provide OpenAI-compatible HTTP servers for chat completions and streaming. You can switch between them by changing your client's base URL and model parameter.
LMDeploy's primary high-performance engine, TurboMind, is tuned for NVIDIA GPUs. While its PyTorch backend can target other platforms like Huawei Ascend, it does not offer the broad native AMD ROCm support found in vLLM.
LMDeploy is generally easier for end-to-end quantization workflows. It bundles weight quantization (AWQ, GPTQ) and KV cache quantization directly inside the toolkit, whereas vLLM focuses on serving pre-quantized models generated by external libraries like AutoAWQ, llm-compressor, or TorchAO.
vLLM has a significantly larger community. Measured at the time of writing, vLLM has over 93,000 GitHub stars and nearly 2 million monthly PyPI downloads, compared to LMDeploy's roughly 8,100 stars and 27,000 monthly downloads. vLLM also features broader third-party platform integrations.
Both engines are free and open-source under the Apache 2.0 license, so software costs are zero. Infrastructure costs depend on GPU utilization. If LMDeploy's TurboMind engine or INT4/INT8 KV-cache quantization lets you pack your target workload onto fewer or smaller GPUs, it can reduce compute costs; otherwise, vLLM's broad hardware support allows you to choose cheaper alternative hardware like AMD GPUs.
vLLM is more popular on GitHub with 97.0K stars, against 8.1K for LMDeploy. Stars measure developer interest, so also compare downloads and contributors above.
LMDeploy has 165 contributors and its last commit was Today. vLLM has 4.1K contributors and its last commit was 2 days ago.
vLLM has AMD GPU built in. LMDeploy does not, based on each project’s documentation. See the capabilities table above for the full list.
LMDeploy is a Mixed engine from InternLM, released under the Apache 2.0 license. vLLM is a Python engine from vLLM Project, released under the Apache 2.0 license. The table above shows how they differ on hardware support, APIs and serving features.
Start with your hardware and your traffic. Check which engine supports your GPUs, whether you need an OpenAI-compatible API, and how many users it must serve at once. Then compare community activity, since an active project ships fixes faster.
| NVIDIA GPUs (primary) |
| NVIDIA, AMD ROCm, CPUs, and accelerators |
| Built-in Quantization | AWQ, GPTQ, SmoothQuant, INT4/INT8 KV Cache | FP8, INT4, INT8, AWQ, GPTQ, GGUF, TorchAO |
| GitHub Stars | 8,106 | 93,102 |
| Monthly PyPI Downloads | 27,213 | 1,942,109 |
| GitHub Contributors | 157 | 3,628 |
vLLM centers around the PagedAttention algorithm introduced by UC Berkeley researchers in 2023. It treats KV-cache memory like virtual memory pages in an operating system, virtually eliminating internal memory fragmentation. Its Python-first architecture makes it fast to adapt to emerging model topologies, supported by a contributor community of over 3,600 developers.
LMDeploy, maintained by the InternLM team, provides two execution engines: a PyTorch engine and TurboMind, an inference engine written in C++ and CUDA. TurboMind is tuned for NVIDIA GPUs. LMDeploy bundles native model compression tools, allowing teams to quantize both model weights and KV caches directly within the same toolkit without external quantization libraries.
Both frameworks achieve orders-of-magnitude improvements over naive Hugging Face Transformers serving by using continuous batching and paged memory. However, their throughput characteristics depend heavily on prompt length, batch size, and quantization schemes.
According to LMDeploy documentation, LMDeploy delivers up to 1.8x higher request throughput than vLLM by combining persistent batching, blocked KV cache, dynamic split-and-fuse scheduling, and tuned CUDA kernels. The LMDeploy project also reports that its 4-bit inference performance is 2.4x higher than FP16, with quantization quality verified via OpenCompass evaluations. It achieves this by simultaneously combining AWQ, automatic prefix caching, and INT4 or INT8 KV cache quantization.
In contrast, vLLM draws its performance from PagedAttention, chunked prefill, and multi-backend attention kernels like FlashAttention, FlashInfer, and FlashMLA. Because vLLM processes batch scheduling and model orchestration with piecewise CUDA graphs, it handles mixed-length concurrent requests with low scheduling overhead. In community issue discussions, actual throughput comparisons vary depending on the benchmark setup: whether throughput measures generated tokens only or includes input prompt processing, and which sequence lengths are tested. For high-concurrency production deployments on standard Hugging Face checkpoints, vLLM remains the industry baseline against which inference speeds are evaluated.
vLLM provides broad hardware support. While it was built primarily on NVIDIA GPUs, it natively supports AMD GPUs via ROCm, x86/ARM CPUs, and accelerator hardware like Google TPUs and Intel Gaudi. Its architecture catalog covers over 200 model architectures on Hugging Face. This includes standard decoder-only models (Llama, Qwen), Mixture-of-Experts architectures (Mixtral, DeepSeek-V3), multimodal systems (Qwen-VL, Pixtral), state-space models (Mamba), and embedding architectures.
LMDeploy is tightly focused on NVIDIA GPUs through its high-performance TurboMind engine, though its secondary PyTorch engine provides an alternate path. Its documentation details support for standard LLMs (like Llama, InternLM, and Qwen) and vision-language models (VLMs). While it covers the most popular production models, its catalog is narrower than vLLM's library of hundreds of specialized model types.
Both systems offer drop-in replacements for standard proprietary endpoints, reducing the engineering cost of migration:
openai Python or TypeScript SDKs) can point to either server simply by changing the base URL.Neither tool is a zero-configuration desktop runner; both are built for cloud and server infrastructure. Users install both tools through standard PyPI wheels or container images.
Installing vLLM is generally simple on common Linux distributions with standard CUDA drivers. However, compiling it from source or running on non-NVIDIA hardware can require managing complex kernel dependencies. Its massive user base means common installation errors are thoroughly documented on forums and GitHub issues.
LMDeploy can be installed via PyPI, but working with its TurboMind backend requires careful alignment with compatible CUDA runtime environments. If you want to use its built-in quantization pipelines, LMDeploy simplifies workflow management by removing the need to run third-party tools like AutoAWQ before loading models into the serving engine.
Both projects share open, enterprise-friendly licensing terms:
Both licenses allow free commercial modification, self-hosting, and internal distribution without proprietary license fees.
LMDeploy is a strong fit if your operations match these needs:
vLLM is the better fit if your requirements depend on these factors: