TensorRT-LLM

• `trtllm-serve`: a CLI that starts an OpenAI-compatible HTTP server. You point it at the engine directory, and it exposes /v1/chat/completions, /v1/completions, and /v1/responses endpoints. This drops in as a replacement for any application written against the OpenAI API.
• LLM Python API: a programmatic interface for offline inference, batch processing, or custom integration. The LLM class loads the engine and runs inference with an API similar to Hugging Face’s pipeline but optimized for performance.

Both runtimes support the same core features: continuous batching, quantization, multi-GPU parallelism, and streaming. The mental model for an engineer is: you compile a model once per GPU target, then treat the engine file as a deployable artifact. No Python dependencies are needed at runtime for the C++ server path.

TensorRT-LLM only runs on NVIDIA GPUs. No CPU, AMD, or Apple Silicon support. It is designed to wring every token out of H100, H200, B200, GB200, and RTX 6000 Pro Blackwell cards. It uses CUDA graphs, fused attention kernels, and FP4/FP8/INT4 quantization to reduce memory and boost throughput.

The engine achieves its speed through:

• Compiled engines: graph optimizations eliminate Python overhead during inference, making it one of the fastest options on NVIDIA hardware.
• In-flight batching: requests are added to and removed from the running batch dynamically. This keeps GPU utilization high, especially under variable load.
• Multi-GPU support: you can split a model across GPUs using tensor parallelism or pipeline parallelism. This is critical for models that do not fit on a single GPU (e.g., Llama 3.1 405B).
• KV cache optimization: TensorRT-LLM manages the KV cache with paged attention-like techniques to maximize memory capacity for longer sequences.

NVIDIA publishes official throughput benchmarks for key models across its GPU lineup. For example, on B200 with FP4 quantization, Llama 3.3 70B achieves over 10,600 tokens per second per GPU at 128 input and 128 output tokens. At longer context lengths (128 input, 4096 output), throughput drops to 6,276 t/s per GPU, still competitive. These numbers are reproducible using the trtllm-bench tool.

The trade-off: you must compile for each GPU target and batch configuration. Changing batch size or model version requires recompilation. This makes it less flexible than dynamic runtimes like vLLM but delivers higher throughput in stable production environments.

• OpenAI-compatible server: trtllm-serve exposes an API that matches the OpenAI chat, completion, and responses schemas. You can point any existing OpenAI client (LangChain, LlamaIndex, custom app) at it without code changes. Supports streaming, structured output (JSON schema), and function calling.
• Continuous batching: also called in-flight batching. Requests are not queued in a fixed batch; the engine dynamically reorders and packs sequences to maximize GPU utilization. This is essential for latency-sensitive serving because it keeps the GPU busy without waiting for batch completion.
• Quantization: support for FP8, INT8, INT4, and FP4 weight and activation quantization. NVIDIA provides pre-quantized models on Hugging Face (e.g., Llama-3.3-70B-Instruct-FP4). Quantization reduces memory footprint and increases throughput, often with minimal accuracy loss.
• Multi-GPU parallelism: tensor parallelism (splitting layers across GPUs) and pipeline parallelism (splitting layers across stages). You can run models that require hundreds of GB of memory across multiple DGX nodes.
• Structured output: guaranteed JSON or regex-constrained generation via guided decoding. Useful for applications that require parseable output (e.g., API responses, code generation).
• Streaming: token-by-token streaming via Server-Sent Events (SSE) on the OpenAI-compatible API, or via Python generators for the LLM API.

Latency-sensitive serving: If you serve an LLM-powered assistant where users expect sub-second first-token latency, TensorRT-LLM’s compiled kernels and minimal overhead make it a strong choice. It is commonly used in NVIDIA’s own services (e.g., NIM).

High-throughput NVIDIA deployments: Teams running thousands of queries per second on a fleet of H100 or B200 GPUs choose TensorRT-LLM to maximize tokens per watt and per dollar. The compilation step pays off at scale.

Large multi-GPU models: When a model like Llama 3.1 405B (or a MoE model like Mixtral) does not fit on a single GPU, TensorRT-LLM’s multi-GPU support with tensor parallelism is production-grade. It is often the default for internal NVIDIA clusters.

Self-hosted OpenAI API: Many organizations replace OpenAI with a self-hosted model behind the same API surface. TensorRT-LLM’s trtllm-serve makes this straightforward.

Poor fit for: teams running on AMD, Intel, or Apple hardware; developer laptops without NVIDIA GPUs; rapid prototyping where compile times are unacceptable; heterogeneous hardware environments.

The install command is:

Prerequisites:

• An NVIDIA GPU with compute capability 8.0 or later (Ampere, Ada, Hopper, Blackwell).
• CUDA 12.x and cuDNN 8.x installed.
• At least 20 GB of free disk space for the build and engine files.

The smallest path to a running model:

1. Install TensorRT-LLM and its dependencies.
2. Download a pre-quantized model from NVIDIA’s Hugging Face collection (e.g., nvidia/Llama-3.2-3B-Instruct-FP8).
3. Build an engine: trtllm-build --checkpoint_dir ./model --output_dir ./engine.
4. Serve: trtllm-serve ./engine.
5. Query: curl http://localhost:8000/v1/chat/completions -d '{"model":"default","messages":[{"role":"user","content":"hello"}]}'.

For offline inference using the Python API:

Documentation is at [nvidia.github.io/TensorRT-LLM](https://nvidia.github.io/TensorRT-LLM). The GitHub repository ([NVIDIA/TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM)) contains example recipes, benchmarking tools (trtllm-bench), and Slurm deployment scripts. Community support is primarily through GitHub issues and NVIDIA developer forums.

TensorRT-LLM vs vLLM: vLLM uses PagedAttention and is architecture-agnostic (NVIDIA, AMD, CPU). TensorRT-LLM is faster on NVIDIA GPUs (10-30% higher throughput in many benchmarks) but requires compilation per GPU target. vLLM is easier to iterate with and supports more hardware. Choose TensorRT-LLM for production fleets of homogeneous NVIDIA GPUs where peak throughput matters more than quick deployment. Choose vLLM for dynamic environments, multi-vendor hardware, or when you need to swap models frequently without recompilation.

TensorRT-LLM vs Ollama: Ollama is designed for local development and single-user use on consumer GPUs. It abstracts away compilation and offers a simple ollama run experience. TensorRT-LLM requires manual setup and compilation, making it inappropriate for quick experimentation. Ollama also supports CPU and Apple Silicon. If you are running open models locally on a gaming GPU, use Ollama. If you are deploying to a production cluster with dozens of H100s, use TensorRT-LLM.

TensorRT-LLM vs SGLang: SGLang emphasizes programming language-like control over LLM outputs (structured generation, function calls). TensorRT-LLM supports these features but does not match SGLang’s flexibility in composing calls. However, on NVIDIA GPUs, TensorRT-LLM typically delivers superior throughput. SGLang also supports AMD GPUs. Your choice depends on whether you need advanced output control (SGLang) or raw speed (TensorRT-LLM).

TensorRT-LLM vs TGI: Hugging Face’s TGI offers broad model support and simple deployment. It is slower than TensorRT-LLM on equivalent hardware. TensorRT-LLM is the better choice if you are already committed to NVIDIA and need every token of performance. TGI wins on model coverage and ease of use.

• Compiled inference engines

  Turns a model into a hardware-tuned engine for faster runs on NVIDIA GPUs.

• In-flight batching

  Adds and removes requests from the batch on the fly to keep the GPU busy.

• OpenAI-compatible server

  trtllm-serve exposes a local endpoint that mirrors the OpenAI API.