d3-nano

The d3-nano model from vLLM Semantic Router is a specialized 2.21B parameter dense decision model built for deterministic classification, intent routing, and structured evaluation. Rather than functioning as an autoregressive text generator that predicts next tokens, d3-nano processes states (text, structured JSON, images, or video) against predefined questions and outputs probability distributions across target classes in a single forward pass. It sits in the Decision 3.0 family, providing an open-source, non-generative alternative to using heavy generalist large language models for triage and tool routing.

For engineers running local agent architectures, routing has historically presented an awkward tradeoff. Using a standard 7B or 8B generative model with structured JSON output introduces latency penalties, token generation overhead, and parsing failures. Conversely, traditional zero-shot cross-encoders often struggle with nuanced multimodal contexts. The d3-nano model targets this gap directly, scoring 38.9 on the Jev Decision Index 0.3.2 text leaderboard while running with a median text latency of 13.4 ms on enterprise accelerators like the AMD Instinct MI325X.

Released under an Apache-2.0 license, d3-nano serves as an ultra-fast System 1 triage layer. It allows developers to classify incoming queries, enforce policy checks, score input urgency, and route tasks across local execution graphs without adding prompt-formatting latency or consuming the memory footprint of a generative system.

The d3-nano model is built on a 2.21B parameter dense backbone derived from Qwen3.5-2B, split between a 1.88B language foundation and an integrated 0.33B vision encoder.

Its text processing engine uses 24 layers configured in a hybrid attention setup: three linear-attention layers for every standard full-attention layer. This hybrid design significantly minimizes the computational bottleneck of quadratic self-attention on long contexts. The language backbone features a hidden dimension of 2,048, an intermediate feed-forward network (FFN) dimension of 6,144, 8 query heads over 2 key-value heads (Grouped-Query Attention) with a head dimension of 256, and an attention output gate. The linear-attention layers implement 16 value and 16 key heads with a dimension of 128 and a convolution kernel size of 4.

Key architectural specifications include:

• Total Parameters: 2,213,241,664 (2.21B)
• Vision Tower: 24 layers, 1,024 hidden dimension, patch size 16 with spatial merge factor 2, projecting to 2,048 dimensions (accounting for 0.33B parameters)
• Attention Mechanism: Hybrid linear-attention and full-attention with partial rotary factor (0.25)
• Vocabulary Size: 248,320 tokens with tied embeddings
• Inference Mode: Non-autoregressive forward pass (system_one runtime) returning Softmax-normalized probability arrays across candidate targets

Because d3-nano does not generate tokens sequentially, it bypasses the autoregressive decode phase entirely. Inference time is determined almost exclusively by prefill throughput. Multiple questions (choice, noul, and score) evaluated against the same input state can be processed simultaneously against shared activations, maximizing GPU compute density.

The model is optimized strictly for zero-shot decision classification. It handles three core question primitives:

• Choice: Multi-class classification where the model distributes probabilities across an arbitrary set of categorical routes or tools.
• Yes / No (`noul`): Binary gating used for guardrails, policy enforcement, hallucination checks, and PII detection.
• Score: Ordinal ranking along a continuous scale, such as assessing ticket urgency, sentiment polarity, or retrieval relevance.

In production environments, d3-nano replaces brittle prompt-based routing pipelines. Concrete implementations include:

Instead of sending every raw user prompt to a large reasoning model, d3-nano evaluates input difficulty, required tools, and domain requirements in sub-20ms runtimes. Easy queries route to local small language models (SLMs), while complex reasoning queries route to frontier models.

Because d3-nano natively embeds images and video (sampled at 2 frames per second), it can evaluate media inputs directly. An automation pipeline can feed a customer support ticket containing text and a photo of a broken part to d3-nano, simultaneously asking whether a receipt is present (binary), which department owns the ticket (choice), and how severe the physical damage appears (score).

Running input-output safety checks through generative LLMs doubles system latency. Serving d3-nano as an inline guardrail allows teams to classify safety violations, prompt injection attempts, and context drift at line speed before downstream models execute.

Because d3-nano is an efficient local AI model with 2.21B parameters, running it on consumer hardware requires minimal system resources.

The memory footprint of d3-nano is dictated by its weights and the working memory needed for batch prefill:

• BF16 / FP16 (Unquantized): ~4.5 GB to 5.0 GB VRAM. Recommended total GPU memory: 6 GB.
• INT8 / FP8: ~2.5 GB to 3.0 GB VRAM. Recommended total GPU memory: 4 GB.
• INT4 / Q4_K_M: ~1.5 GB to 2.0 GB VRAM. Recommended total GPU memory: 4 GB.

How to run a 2.21B model on a consumer GPU comes down to framework support. For standard desktop setups:

• Entry-Level Consumer GPUs: An NVIDIA GeForce RTX 3060 (12GB) or RTX 4060 (8GB) can comfortably run d3-nano locally in full BF16 precision with substantial headroom left over for concurrent local SLMs or embeddings.
• Enthusiast GPUs: On an RTX 4090 or RTX 5090, d3-nano runs almost entirely in L2/device cache, achieving latencies in the single-digit millisecond range for text requests.
• Apple Silicon: Mac devices equipped with M2, M3, or M4 chips (even base 8GB or 16GB configurations) run d3-nano efficiently under unified memory via PyTorch or MLX.
• Enterprise Hardware: The best GPU for d3-nano deployment at scale includes the AMD Instinct MI300X/MI325X or NVIDIA H100/L40S. In vLLM Semantic Router production benchmarks, a single MI325X achieves 318 requests per second under batch serving.

Traditional generative metrics like "d3-nano tokens per second" do not apply in the typical sense because the model does not generate output tokens. Its d3-nano performance is measured in request latency and decisions per second:

• Text Request Latency: 10 ms to 15 ms median response time on high-end hardware; 20 ms to 40 ms on mid-range consumer GPUs.
• Image Processing Latency: ~90 ms per image request.
• Video Processing Latency: ~300 ms for a 10-second clip evaluated at 2 fps.

For practitioners selecting the best quantization for d3-nano:

• BF16: Run unquantized if you have 6 GB of available VRAM. At 2.21B parameters, weight compression is rarely necessary on modern discrete GPUs, and native precision preserves exact probability calibration across edge-case routing categories.
• Q4_K_M or AWQ 4-bit: Use this if you are co-locating d3-nano alongside a larger 8B or 14B model on a single 16GB or 24GB consumer GPU. The model consumes less than 2 GB of VRAM, leaving the remainder of the pool for your primary generation engine.

You can load and test the weights directly via Hugging Face Transformers:

1pip install "transformers==5.17.0" torch torchvision pillow safetensors accelerate

1from transformers import AutoModel
2
3model = AutoModel.from_pretrained("vllm-sr/d3-nano", trust_remote_code=True)
4
5result = model.system_one(
6    state="Customer asking for an immediate refund on an order delivered three weeks ago.",
7    questions={
8        "route": {
9            "type": "choice",
10            "instructions": "Determine handling tier",
11            "criteria": {
12                "support_l1": "Standard inquiries and tracking",
13                "returns": "Refund processing and RMA creation",
14                "escalations": "Policy disputes and high-value orders"
15            }
16        },
17        "eligible": {
18            "type": "noul",
19            "instructions": "Is the order within the standard 14-day return window?"
20        }
21    }
22)
23print(result["answers"])

Understanding d3-nano vs alternative routing setups comes down to task specialization versus generalist overhead.

• d3-nano vs. Generative SLMs (Qwen2.5-3B / Llama-3.2-3B): Running a general-purpose 3B model with guided decoding (such as Outlines or instructor) to return a JSON decision requires sampling 10 to 50 tokens per request. Even at high generation speeds, this incurs a latency cost of 100 ms to 300 ms. The d3-nano model performs the same classification task in 13 ms to 20 ms without JSON validation failures or syntax drift.
• d3-nano vs. LiquidAI d1-3B: On the Jev Decision Index 0.3.2 text benchmark, d3-nano registers a score of 38.9, matching LiquidAI d1-3B while using fewer active weights and offering native multi-modal image/video ingest.
• d3-nano vs. Decision 2.0 (2B): The previous generation Decision 2.0 2B model scored 27.1 on the official Jev index. Version 3.0 represents an 11.8 point jump on the leaderboard, driven largely by the hybrid linear-attention stack and architectural improvements inherited from the Qwen3.5 base.

If your pipeline requires creative synthesis, complex free-form code generation, or agentic dialogue, run a general-purpose autoregressive model. If your architecture demands a low-overhead, deterministic decision engine for classification, filtering, and traffic routing, d3-nano offers optimal parameter efficiency and latency.