LMDeploy

The runtime model is a persistent batch (continuous batching) with blocked KV cache management. When a request arrives, it joins the running batch if a slot is free; otherwise it queues. The engine dynamically splits and fuses attention computations across tokens in the batch, maximizing GPU utilization. Streaming responses are supported via server-sent events.

To serve a model, you point LMDeploy at a Hugging Face model ID or a local path. It downloads the weights automatically if needed, applies any configured quantization (AWQ, GPTQ, SmoothQuant, or INT4/INT8 KV cache), and starts the server. The entire flow from install to a running endpoint can be under five minutes on a modern GPU rig.

LMDeploy is optimized for NVIDIA GPUs. The TurboMind engine uses high-performance CUDA kernels, tensor parallelism, and dynamic split-and-fuse attention to deliver strong throughput. The maintainers report up to 1.8x higher request throughput than vLLM in their benchmarks, and 4-bit quantized inference at 2.4x the tokens per second of FP16. These numbers are workload-dependent, but the engine consistently ranks among the fastest for batch serving on H100 and A100-class hardware.

Hardware support (confirmed capabilities): NVIDIA GPUs (V100 and newer). The engine also has experimental support for AMD ROCm, Ascend NPUs, and Cambricon accelerators via community contributions, but these are not covered in the confirmed capabilities and should be evaluated case by case. There is no CPU-only inference path; LMDeploy requires a GPU for any meaningful performance.

Memory management: LMDeploy supports weight-only quantization (AWQ, GPTQ, SmoothQuant) and KV cache quantization (INT4/INT8). This allows running larger models on smaller GPUs — for example, a 70B parameter model quantized to 4-bit can fit on a single 80GB A100 or H100. The blocked KV cache manager pages memory efficiently, and automatic prefix caching further reduces redundant computation for repeated prompt prefixes.

Scaling: Multi-GPU inference is handled via tensor parallelism across up to 8 GPUs on a single node. For multi-node deployments, the request distributor server can route across machines, though this adds operational complexity. Continuous batching is the default, and it works with all quantization modes simultaneously.

• TurboMind engine: A CUDA-optimized inference backend that implements persistent batch (continuous batching), blocked KV cache, and dynamic split-and-fuse attention. It is the core reason for LMDeploy’s throughput advantage.
• Built-in quantization: Supports weight-only quantization (AWQ, GPTQ, SmoothQuant) and INT4/INT8 KV cache quantization. You can apply quantization during model loading or pre-quantize offline. The quantization quality is verified by OpenCompass evaluations.
• OpenAI-compatible API: The server exposes endpoints identical to OpenAI’s /v1/chat/completions and /v1/completions. This means any OpenAI client library (Python, JavaScript, curl) can be pointed at LMDeploy with only a URL change. It also supports tool calling, streaming, and structured output (JSON mode).
• Structured output: LMDeploy can constrain generation to follow a JSON schema or regex pattern, useful for agentic workflows and API integrations.
• Vision-language model support: Beyond text-only LLMs, LMDeploy can serve VLMs like InternVL, LLaVA, and Qwen-VL through the same OpenAI-compatible server, accepting multimodal inputs (images + text).
• LoRA serving: Load and swap LoRA adapters at runtime without restarting the server, enabling multi-tenant fine-tuned model serving.
• Automatic prefix caching: Reduces latency for repeated prompt prefixes, common in chat applications and RAG pipelines.

• High-throughput production serving: Deploy a 70B model behind LMDeploy to serve a busy chat application or API endpoint. The combination of continuous batching, quantization, and tensor parallelism keeps per-token costs low.
• Memory-constrained GPU deployments: Run a 70B model on a single 48GB GPU (e.g., A6000) using 4-bit weight quantization and INT8 KV cache. This is a common pattern for teams that cannot access H100s.
• Self-hosted OpenAI replacement: Point existing OpenAI clients (e.g., LangChain, LlamaIndex, custom apps) at an LMDeploy server running a model like Llama 3 or Qwen. No code changes required — just swap the base URL and API key.
• Offline batch inference: Use the Python pipeline to process large datasets (e.g., synthetic data generation, evaluation) with maximum throughput.
• Multi-model gateways: The request distributor server can route to different models on different GPUs, enabling a unified endpoint for multiple fine-tuned variants.

LMDeploy is a poor fit for CPU-only deployments, edge devices without NVIDIA GPUs, or teams that need tight integration with AMD or Intel hardware. It also has a smaller ecosystem of community extensions compared to vLLM.

Install: pip install lmdeploy

Minimum path to a running server:

1. Ensure you have an NVIDIA GPU with at least 16GB VRAM (e.g., RTX 4090, A10, A100).
2. Run: lmdeploy serve api_server internlm/internlm2-chat-7b
3. The server starts on http://localhost:23333. Send a request:

What you need: A GPU with CUDA 11.8 or later. For quantized models, you may need additional packages (lmdeploy[awq] or autoawq). The documentation at [lmdeploy.readthedocs.io](https://lmdeploy.readthedocs.io) covers supported models, quantization recipes, and advanced configuration.

Community: GitHub issues and discussions are active. The project has a WeChat group, Discord server, and Twitter account for updates. The documentation is thorough and includes quickstart guides, benchmark instructions, and deployment recipes.

LMDeploy vs vLLM: vLLM has broader hardware support (AMD, Intel, Apple Silicon via community forks) and a larger ecosystem of model architectures. LMDeploy tends to win on raw throughput for NVIDIA workloads, especially with quantization enabled. If you are on NVIDIA and want the fastest tokens per second with minimal setup, LMDeploy is the stronger choice. If you need multi-platform support or a larger community for troubleshooting, vLLM is safer.

LMDeploy vs Ollama: Ollama focuses on ease of use for local experimentation on a single machine, including Mac and CPU support. LMDeploy is built for production serving with high throughput and multi-GPU scaling. For a single developer running models on a laptop, Ollama is simpler. For a team serving a model to thousands of users, LMDeploy is more appropriate.

LMDeploy vs SGLang: SGLang introduces a custom frontend language for structured generation and has strong support for agentic workflows. LMDeploy’s structured output is simpler (JSON schema via the API) but less flexible. SGLang also supports a wider range of hardware. LMDeploy wins on throughput and quantization maturity; SGLang wins on programmable generation and multi-hardware support.

Choose LMDeploy when your primary goal is maximum throughput on NVIDIA GPUs with built-in quantization and an OpenAI-compatible API. Avoid it if you need CPU inference, AMD-first hardware, or a large English-language community.

• TurboMind engine

  A fast inference backend tuned for throughput on NVIDIA GPUs.

• Built-in quantization

  Shrink models with weight and KV cache quantization to save memory.

• OpenAI-compatible server

  Serves a familiar API so existing clients connect without changes.