LMDeploy and SGLang compared side by side on GitHub stars, downloads, language, license, capabilities, strengths and trade-offs.
TL;DR
Choose SGLang if you run multi-turn agents, structured JSON workloads, or need hardware flexibility across NVIDIA and AMD GPUs. Choose LMDeploy if you operate strictly on NVIDIA hardware and prioritize maximum raw throughput on quantized models using an end-to-end C++ and CUDA runtime.
Add an Engine
Slot 3 of 3
Comparing inference engines is the process of evaluating two or three competing tools side by side on live community signals, technical capabilities, language, license, and trade-offs so a team can pick the one that fits its hardware and workload.
Start with the constraints that are not negotiable: the hardware you already run on, the license your legal team will sign off on, and the kind of workload you need to serve. An engine that cannot use your GPU or fit your model is not really an option, no matter how popular it is.
Then look at live signals: GitHub stars and contributors show whether the project is gathering momentum, PyPI downloads show whether teams are actually shipping with it, and last-commit date shows whether the maintainers are still around. Capability flags like OpenAI-compatible API, GPU support, quantization, continuous batching, and multi-GPU narrow the field to engines that match your job-to-be-done.
Finish by reading the strengths and trade-offs columns side by side. The simplest engine that covers your real requirements almost always beats the most powerful one. Copy the share link once you have a comparison you trust so you can revisit it during planning.
Decisions about inference engines rarely happen in isolation. Pair this comparator with the directories and benchmarks that ground the rest of the stack.
Live data from GitHub and the package registries, refreshed every day.
GitHub stars
Contributors
Forks
Star history sampled from the GitHub stargazer timeline and extended by our daily sync.
LMDeploy and SGLang are two high-performance inference frameworks designed to replace or outperform default serving options like vLLM. LMDeploy, developed by the InternLM team, relies on its C++ TurboMind runtime and deeply integrated quantization pipeline to squeeze maximum speed out of NVIDIA GPUs. SGLang, originating from the LMSYS team, focuses on execution graphs, RadixAttention prefix caching, and fast constrained decoding for agentic workloads. Both deliver production-grade serving, but they optimize for different deployment constraints.
At a fundamental level, LMDeploy is an end-to-end deployment toolkit, whereas SGLang is an execution engine tuned for complex prompt interactions and structured generation. LMDeploy includes built-in model compression tools like AWQ, SmoothQuant, and KV cache quantization alongside its TurboMind inference engine. SGLang differentiates itself with RadixAttention, which maintains an automatic radix tree of KV caches across requests so multi-turn agent calls and common system prompts do not recompute token attention.
Ecosystem momentum is another clear differentiator. At the time of writing, LMDeploy has 8,106 GitHub stars, 157 contributors, and 27,213 monthly PyPI downloads since its release on June 15, 2023. SGLang, released on January 17, 2024, has reached 36,744 GitHub stars, 1,992 contributors, and 2,556,024 monthly PyPI downloads. SGLang enjoys a wider contributor community and rapid adoption across production clusters, while LMDeploy maintains a smaller footprint concentrated in the InternLM and OpenDataLab ecosystem.
Pick LMDeploy if you need: High-throughput serving with built-in quantization
Full LMDeploy profileOpen and closed model rankings with benchmarks, context windows, modalities, and live API prices.
Straight answers to the questions we hear most often.
Can't find what you're looking for? Book a Discovery Call
It depends on the workload and quantization. In published tests on quantized models (such as Qwen2.5-AWQ on an RTX 2080 Ti), LMDeploy's TurboMind engine achieved higher raw throughput than SGLang. However, on multi-turn agent workflows with shared prefixes, SGLang is often faster overall because RadixAttention avoids recomputing prompt attention.
Both engines require Python environments with compatible GPU drivers and offer pre-built Docker containers. SGLang has an advantage in practical troubleshooting because its larger user base and contributor community mean that setup bugs and driver issues are documented and resolved more quickly.
Yes. SGLang natively supports AMD Instinct accelerators through AMD ROCm alongside NVIDIA GPUs. LMDeploy focuses primarily on NVIDIA CUDA environments and does not offer production-grade AMD GPU support.
Yes. Both LMDeploy and SGLang provide OpenAI-compatible HTTP servers supporting standard chat completion and completion endpoints. You can switch between them by changing the base URL port and server configuration without modifying your client-side application logic.
Operational costs depend on your hardware and traffic pattern. For standard quantized generation on NVIDIA GPUs, LMDeploy can yield lower hardware costs due to high throughput per card. For agent pipelines with repeated prefixes or multi-turn interactions, SGLang reduces GPU compute costs by reusing cached prompt tokens.
Yes. LMDeploy includes a standalone quantization module that supports AWQ, GPTQ, SmoothQuant, and KV cache quantization. You can use LMDeploy strictly to quantize model weights and KV caches, and then evaluate or export them as needed.
Yes, both engines support structured outputs and constrained decoding. However, SGLang was built from the ground up around high-speed grammar-based decoding and regex constraints, making it especially well suited for complex JSON schema enforcement in production.
SGLang is more popular on GitHub with 39.0K stars, against 8.1K for LMDeploy. Stars measure developer interest, so also compare downloads and contributors above.
LMDeploy has 165 contributors and its last commit was Today. SGLang has 2.3K contributors and its last commit was Today.
LMDeploy was first released in 2023 and SGLang in 2024. A longer track record usually means more examples, integrations and answered questions.
SGLang has AMD GPU built in. LMDeploy does not, based on each project’s documentation. See the capabilities table above for the full list.
LMDeploy is a Mixed engine from InternLM, released under the Apache 2.0 license. SGLang is a Python engine from SGLang Project, released under the Apache 2.0 license. The table above shows how they differ on hardware support, APIs and serving features.
Start with your hardware and your traffic. Check which engine supports your GPUs, whether you need an OpenAI-compatible API, and how many users it must serve at once. Then compare community activity, since an active project ships fixes faster.
PyPI downloads per month
| Dimension | LMDeploy | SGLang |
|---|---|---|
| Primary Maintainer | InternLM | SGLang Project (LMSYS) |
| First Released | June 15, 2023 | January 17, 2024 |
| Core Architecture | TurboMind (C++/CUDA) and PyTorch | RadixAttention runtime (Python/CUDA/HIP) |
| GitHub Stars (at writing) | 8,106 | 36,744 |
| Monthly PyPI Downloads | 27,213 | 2,556,024 |
| Supported Accelerators | NVIDIA GPUs | NVIDIA, AMD ROCm, Intel Xeon, TPU, Ascend |
| Built-in Model Quantization | AWQ, GPTQ, SmoothQuant, INT4/INT8 KV | Integrates with external quantization formats |
| Prefix Caching Strategy | Automatic prefix caching | RadixAttention (tree-structured LRU cache) |
| Structured Output | Supported (constrained decoding) | Native high-speed regex and JSON grammar |
Both frameworks consistently beat baseline runtimes across published benchmarks, but their relative advantages depend on quantization, concurrency, and workload structure.
In a standalone benchmark published by Loouis Blog testing short-input, long-output generation with Qwen2.5-14B-Instruct-AWQ on an Intel Xeon E5-2680v4 CPU with a single NVIDIA RTX 2080 Ti (22GB), LMDeploy demonstrated higher raw decoding throughput than SGLang across all concurrency tiers:
On 16-bit precision evaluations on data center hardware, the performance gap narrows significantly. In an evaluation conducted on an NVIDIA A100 GPU (80GB) running Llama-3-8B and Llama-3.1-8B in BF16 (documented in SGLang GitHub Issue #834), both SGLang 0.2.6 and LMDeploy 0.5.2.post1 hit approximately 5,000 output tokens per second under high-concurrency offline workloads. Both engines sustained identical request-per-second ceilings during online latency tests, far outpacing vLLM 0.5.2 under the same conditions.
Where SGLang pulls ahead in practice is multi-turn agent workflows. Its RadixAttention maintains an active cache tree across independent requests. In workloads with shared system prompts, multi-candidate tree searches, or tool-use loops, SGLang avoids prefill recomputation almost entirely, significantly lowering time-to-first-token (TTFT).
Hardware flexibility is a key divergence between the two runtimes. LMDeploy focuses its performance tuning on NVIDIA GPUs. While it offers an experimental PyTorch engine backend, its primary production engine, TurboMind, requires NVIDIA hardware with CUDA. It does not provide production-grade support for AMD ROCm, Apple Silicon, or pure CPU execution.
SGLang provides broader accelerator compatibility. In addition to NVIDIA GPUs, SGLang maintains production support for AMD Instinct accelerators via ROCm. The project documentation also lists support for Intel Xeon processors, Google TPUs, Huawei Ascend NPUs, and Moore Threads MUSA accelerators. Teams running mixed fleet infrastructure or deploying on AMD hardware can use SGLang without rewriting their serving pipeline.
Both engines maintain extensive coverage for current open-weight transformer architectures, including Llama 3, Qwen 2.5, DeepSeek, and Mistral. Both also support Vision-Language Models (VLMs) like Qwen-VL and InternVL. LMDeploy has deep native support for InternLM model families, while SGLang provides broad community support for new model variants within days of release.
Both engines make drop-in replacement straightforward by shipping OpenAI-compatible HTTP servers. Each provides /v1/chat/completions, /v1/completions, and /v1/models endpoints that integrate directly with LangChain, LlamaIndex, LiteLLM, or native OpenAI client libraries.
LMDeploy also adds Anthropic-compatible endpoints out of the box, allowing teams using Claude client conventions to redirect requests without intermediate translation layers. Additionally, LMDeploy includes a dedicated Request Distributor Server, which routes incoming requests across multi-machine and multi-GPU nodes to manage model replicas.
SGLang differentiates its API through native grammar-guided generation and its SGLang frontend language. Constrained decoding with JSON schemas or regular expressions runs directly within the batching loop with minimal throughput penalty. LMDeploy also supports structured outputs, but SGLang makes constrained generation a core design priority. Both systems support continuous batching, tensor parallelism for multi-GPU setups, speculative decoding, and streaming responses.
Neither engine is a simple one-line single-binary desktop utility like Ollama. Both require setting up Python environments with compatible CUDA or ROCm runtimes, NVIDIA drivers, and GPU dependencies.
LMDeploy distributes pre-built wheels via PyPI and provides official Docker images. Setting up TurboMind requires matching your host CUDA driver versions carefully, as the C++ extensions depend on specific CUDA toolchains. Once installed, launching a server is simple via the lmdeploy serve api_server command.
SGLang also installs via PyPI and Docker. SGLang uses a Python-centric architecture with FlashInfer and custom CUDA/HIP kernels. Starting the server requires running python3 -m sglang.launch_server. Because SGLang has a community of over 1,900 contributors and millions of monthly downloads, troubleshooting dependency conflicts and driver quirks is often faster due to extensive GitHub discussion threads.
Both projects use the permissive Apache 2.0 open-source license. You can deploy both engines in commercial products, modify the source code, and run them on private infrastructure without royalty fees or open-source copyleft restrictions. LMDeploy is copyrighted and maintained by the InternLM team. SGLang is hosted by the open-source LMSYS organization, which also maintains Chatbot Arena.
LMDeploy is the right choice when your infrastructure runs entirely on NVIDIA GPUs and you want an integrated pipeline for both quantization and serving. If you plan to compress models down to AWQ, GPTQ, or 4-bit and 8-bit KV caches, LMDeploy provides those tools natively without requiring external quantization scripts. Its TurboMind C++ backend is heavily optimized for quantized weights, delivering leading throughput on consumer and enterprise NVIDIA cards alike. Teams that want Anthropic-compatible routing or already use the InternLM ecosystem will find LMDeploy well aligned with their operational stack.
SGLang is the right choice for agentic applications, structured data extraction, and diverse hardware environments. If your application sends requests with shared system prompts, document contexts, or multi-turn conversational history, SGLang's RadixAttention saves substantial compute and reduces latency compared to conventional prefix caching. It is also the better choice if your team requires strict JSON schema compliance at high generation speeds or runs on non-NVIDIA hardware such as AMD Instinct accelerators. Finally, SGLang's rapid development pace, larger contributor base, and widespread adoption make it a safer long-term bet for general-purpose inference.