Decision-2.0-Lux-9B

Decision-2.0-Lux-9B is a specialized 7.94B parameter dense decision model built by the vLLM Semantic Router team, released under the Apache-2.0 license. Unlike general-purpose autoregressive text generators that emit tokens sequentially, Decision-2.0-Lux-9B is designed for deterministic, low-latency evaluation. It accepts an input state (raw text or structured JSON) alongside a set of targeted questions, returning normalized probability distributions across all potential answers in a single forward pass.

This non-generative "System 1" design solves a primary bottleneck in autonomous agent architectures: decision latency and schema hallucination. Running a standard generative 8B model to classify a prompt or route an agent requires generating tens or hundreds of tokens, introducing parsing overhead and variability. Decision-2.0-Lux-9B eliminates token generation entirely, achieving a median latency of 18.4 ms for single-question requests on a single GPU. It established state-of-the-art results for its weight class on the Jev Decision Index (46.3) and JevArena (68.1), making it an ideal local router for complex multi-model pipelines.

As local inference setups shift toward composite systems, having a dedicated local AI model with 7.94B parameters dedicated to triage, guardrailing, and evaluation allows practitioners to offload critical control-flow tasks from heavier reasoning engines.

Decision-2.0-Lux-9B relies on a dense transformer backbone containing 7.94B parameters. Because it is a dense architecture rather than a Mixture of Experts (MoE), all 7.94B parameters remain active during every execution pass. This yields predictable compute requirements and consistent memory access patterns across diverse workloads.

Key architectural characteristics include:

• Non-Generative Single-Pass Inference: Instead of iteratively decoding tokens via an autoregressive loop, the model maps the concatenated input context and question definitions into an internal representation, decoding probability heads directly. Multiple choice, boolean (yes/no), and rating scale questions are solved simultaneously in one forward step.
• 16,384-Token Context Window: The native 16k context window gives the model ample headroom to ingest extensive conversational histories, long system instructions, retrieval-augmented generation (RAG) context snippets, or dense JSON payloads before evaluating a routing decision.
• Framework Requirements: Running the official weights requires PyTorch and Hugging Face transformers>=5.17 with safetensors, utilizing the custom model.system_one() method with remote code execution enabled. Community GGUF quantizations allow execution via compatible C/C++ runtimes.

Decision-2.0-Lux-9B handles three explicit question primitives: categorical choices, binary (yes/no) validations, and ordinal score ratings. Because it handles these questions concurrently over the same state, it functions as a high-speed telemetry and routing brain in local deployments.

• Agentic Routing and Tool Selection: Instead of relying on fragile prompt engineering to select tools, developers can pass available agent functions as categorical choices. The model returns calibrated probabilities for each branch, enabling deterministic routing thresholds.
• Guardrails and Policy Enforcement: Run input and output filtering in parallel. A single forward pass can check for PII leakage, user jailbreak attempts, and topical alignment before handing execution to a larger model.
• RAG Document Relevance Scoring: Score retrieved chunks on an ordinal scale (such as Routine, Soon, Today, or 1 to 5) to dynamically prune low-confidence context chunks before context assembly.
• Structured Triage Engines: In support ticketing or inbound messaging pipelines, the model can categorize the department, flag urgency as an ordinal score, and verify whether account credentials were provided, all within a 20 ms execution budget.

Deploying this model locally requires understanding its non-generative nature. When people evaluate Decision-2.0-Lux-9B tokens per second, they are typically measuring prompt-ingestion throughput rather than decode generation speeds. Because execution finishes in a single forward pass, total request latency is dominated by prompt prefill and linear algebra execution.

To run Decision-2.0-Lux-9B locally, your GPU memory planning depends primarily on your chosen precision and batch size:

• FP16 / BF16 (Unquantized): Requires ~16 GB to 18 GB of VRAM. This fits comfortably on a single 24 GB card such as an NVIDIA GeForce RTX 3090 or RTX 4090, leaving ample memory for operating system overhead and deep 16k context caches.
• 8-bit Quantization (Q8_0 / INT8): Requires ~9.5 GB to 11 GB of VRAM. An ideal choice for 12 GB and 16 GB GPUs like the RTX 4070 or RTX 4080.
• 4-bit Quantization (Q4_K_M): Requires ~5.5 GB to 6.5 GB of VRAM. This enables running on mainstream 8 GB consumer GPUs, such as the RTX 3060, RTX 4060, or Apple Silicon unified memory hardware.

The best GPU for Decision-2.0-Lux-9B depends on your latency SLA. On an RTX 4090, single-question latency hovers below 20 ms. For local setups running Apple Silicon (such as an M3 or M4 Max), running via unified memory provides more than enough bandwidth to execute routing calls in the 30 ms to 50 ms range.

The best quantization for Decision-2.0-Lux-9B depends on whether your workload is sensitive to subtle probability shifts:

• Q8_0 / FP16: Highly recommended if you rely on raw probability calibrations for fine-grained threshold gating (for example, triggering fallback human review if confidence falls below 0.72).
• Q4_K_M: Recommended for constrained hardware environments where you only care about the argmax winner among multiple choice routes. Q4_K_M cuts memory bandwidth requirements significantly, enabling high request throughput on budget hardware.

To run the model locally using Python and Transformers:

1pip install "transformers>=5.17" torch safetensors

1import json
2from transformers import AutoModel
3
4model = AutoModel.from_pretrained("vllm-sr/Decision-2.0-Lux-9B", trust_remote_code=True).cuda()
5
6result = model.system_one(
7    state="The customer received a broken monitor and wants a refund immediately.",
8    questions={
9        "department": {
10            "type": "choice",
11            "instructions": "Route to the correct department.",
12            "criteria": {
13                "support": "Hardware faults and damages",
14                "billing": "Invoices and subscription queries",
15                "sales": "Pre-purchase questions"
16            }
17        },
18        "has_urgency": {
19            "type": "noul",
20            "instructions": "Does the request require immediate intervention?"
21        }
22    }
23)
24
25print(json.dumps(result["answers"], indent=2))

For teams containerizing local pipelines, community GGUF weights can also be integrated into local model engines or lightweight HTTP microservices that sit in front of local inference endpoints.

Evaluating Decision-2.0-Lux-9B performance requires comparing it against both specialized routing architectures and general-purpose LLMs tasked with classification.

Compared to its predecessor, Decision-2.0-Lux-9B marks a clean architectural step forward:

• JevArena Score: 68.1 vs 65.8 (+2.3 improvement).
• Jev Decision Index: 46.3 vs 43.5 (+2.8 improvement).
• Calibration: Decision 2.0 demonstrates sharper probability separation on edge cases and ambiguous boundary prompts.

Nimble v2 is another notable entry in the sub-10B decision space. On JevArena, Decision-2.0-Lux-9B outscores Nimble v2 by 6.0 points (68.1 vs 62.1) and leads on human-labelled transfer benchmarks (56.2 macro-F1 vs 53.6). Nimble v2 remains competitive for simple binary classifications, but Lux-9B provides higher reliability when handling complex nested JSON inputs across multi-target questions.

A common approach among local developers is prompting a general 8B generative model to return JSON. The tradeoffs are distinct:

• Speed: A general 8B model requires 200 ms to 800 ms to ingest prompts and autoregressively output JSON tokens. Decision-2.0-Lux-9B completes the task in 18 ms to 30 ms.
• Reliability: Generative LLMs risk outputting truncated strings, invalid syntax, or markdown code blocks that break production parsers. Decision-2.0-Lux-9B returns raw tensor probabilities mapped directly to answer keys.
• Flexibility: A generative model can write explanations, generate code, and summarize text. Decision-2.0-Lux-9B cannot generate conversational text: it is strictly built to decide, score, and classify.