LMDeploy

Key Features

What the engine gives you out of the box, in plain language.

• OpenAI-Compatible API
• NVIDIA GPU
• AMD GPU
• Apple Silicon
• CPU Inference
• Quantization
• Continuous Batching
• Multi-GPU
• Desktop GUI
• One-Line Install
• Structured Output
• Streaming

• TurboMind engine

  A fast inference backend tuned for throughput on NVIDIA GPUs.

• Built-in quantization

  Shrink models with weight and KV cache quantization to save memory.

• OpenAI-compatible server

  Serves a familiar API so existing clients connect without changes.

Where It Shines

The jobs this engine is best suited for.

• Production model serving

  Serve an open model to a busy app with low cost per token.

• Memory-constrained GPUs

  Use quantization to run a bigger model on a smaller card.

• Self-hosted OpenAI replacement

  Point existing OpenAI clients at your own GPU box.

Frequently Asked Questions