Decision-2.0-Kai-0.6B

Decision-2.0-Kai-0.6B is a compact, specialized decision model developed by the vLLM Semantic Router team. Built as a full fine-tune of Qwen3-0.6B-Base, the model occupies an important niche in local inference: high-throughput, structured classification and routing without the compute overhead or latency of autoregressive text generation. Instead of generating text token by token, Decision-2.0-Kai-0.6B ingests input data (either raw text or JSON) alongside structured queries and returns calibrated probability distributions for every possible answer in a single forward pass.

With an 8,192-token context window and an Apache-2.0 open-weights license, this 0.6B parameter model serves as a "System One" decision engine. It is engineered specifically for fast-path agent routing, condition validation, and scoring tasks where traditional large language models introduce unnecessary VRAM footprint, high latency, and unpredictable output parsing failures. On a single GPU, the model achieves a reported median latency of just 4.9 ms per single-question request.

For engineers seeking to run Decision-2.0-Kai-0.6B locally, the architecture delivers deterministic structured decision-making at a fraction of the hardware cost required by 7B or 8B parameter alternatives. It eliminates the need to prompt general-purpose models for JSON output, removing both JSON decode failures and token generation bottlenecks from your inference pipelines.

Decision-2.0-Kai-0.6B is a dense transformer model containing 0.60 billion parameters. Because it is a dense architecture rather than a Mixture of Experts (MoE), every parameter is active during inference, providing consistent execution characteristics and predictable memory bandwidth consumption.

The foundational design diverts fundamentally from standard generative decoders:

• Single Forward Pass Execution: Standard generative models predict token by token, spending hundreds of milliseconds sampling text. Decision-2.0-Kai-0.6B evaluates its decision heads directly during the initial prompt evaluation phase. Multiple questions posed against the same context window are evaluated concurrently in one forward pass.
• Typing and Classification Heads: The model supports three discrete question types natively:
• Choice: Categorical routing between arbitrary named candidates with associated descriptive criteria.
• Yes / No: Binary verification to confirm whether state conditions, facts, or constraints are met.
• Score: Ordinal ranking along user-defined scale points (such as low, medium, or high urgency).
• Context Window Utilization: The 8,192-token context length allows developers to feed extensive conversation histories, retrieved document snippets, or detailed JSON payloads directly into the state parameter without truncation.

Because computation is bounded entirely by the prefill/forward phase, latency scales with input length and question batch size, avoiding the quadratic variance common in open-ended generative loops.

Decision-2.0-Kai-0.6B replaces generative models in workloads where developers need categorical answers, probabilities, or policy checks rather than prose.

In multi-agent systems, routing incoming queries to the proper specialist agent often wastes compute when powered by an 8B model. Decision-2.0-Kai-0.6B evaluates semantic routing criteria across multiple target endpoints in under 5 ms. Because it returns raw probabilities for every choice, systems can set custom confidence thresholds, redirecting low-confidence inputs to fallback handlers.

In document processing pipelines, the model can simultaneously extract sentiment, tag support tickets, and check for prerequisite data. For example, a single forward pass can classify a customer inquiry into a department bucket, verify whether an order ID is present, and rate the urgency of the ticket simultaneously.

Safety filters and policy adherence checks run efficiently at this scale. By configuring binary queries (such as checking for toxicity, confidential data leakage, or off-topic prompt injections), you can evaluate incoming prompts before routing them to larger downstream inference clusters.

Because Decision-2.0-Kai-0.6B does not generate text autoregressively, standard benchmarks like "Decision-2.0-Kai-0.6B tokens per second" are less applicable than raw inference latency and batch throughput. Instead of measuring tokens generated per second, Decision-2.0-Kai-0.6B performance is measured in forward-pass latency, clocking in around 4.9 ms per request on desktop GPUs.

The Decision-2.0-Kai-0.6B hardware requirements are remarkably modest, making it one of the most accessible utility models available for local edge deployment.

• FP16 / BF16 (Unquantized): Requires approximately 1.2 GB of VRAM. This preserves absolute precision across all probability heads.
• INT8 / Q8 Quantization: Requires approximately 750 MB of VRAM. Negligible impact on classification accuracy.
• INT4 / Q4_K_M Quantization: Requires approximately 450 MB of VRAM. Useful for running alongside other models in memory-constrained environments.

Because the unquantized weights consume only ~1.2 GB of memory, the best quantization for Decision-2.0-Kai-0.6B for most practitioners is actually unquantized FP16 or BF16. Unlike 70B parameter models where quantization is necessary to fit consumer hardware, running this 0.6B model in full precision prevents any calibration drift in output probabilities while still fitting easily on entry-level hardware.

You do not need an enterprise server to run this model effectively.

• NVIDIA Desktop GPUs: An entry-level RTX 3060 (12 GB) or RTX 4060 (8 GB) will run Decision-2.0-Kai-0.6B in FP16 with sub-5 ms latency, leaving over 80% of your VRAM available for a concurrent generative LLM. For massive concurrent batching, high-end cards like the RTX 4090 or RTX 5080 can process thousands of state evaluations per second.
• Apple Silicon: Any unified memory Mac (M1 through M4, M4 Pro, or M4 Max) handles Decision-2.0-Kai-0.6B trivially using PyTorch Metal Performance Shaders (MPS) or MLX, consuming negligible system memory.
• CPU and Edge Systems: The model runs well on x86 CPUs via ONNX Runtime or native PyTorch, and fits comfortably on single-board devices like the Raspberry Pi 5 or NVIDIA Jetson Orin Nano for edge automation.

To run Decision-2.0-Kai-0.6B locally using Hugging Face Transformers, install the latest dependencies:

1pip install "transformers>=5.17" torch safetensors

Execute a structured decision query using the native system_one execution path:

1import json
2from transformers import AutoModel
3
4model = AutoModel.from_pretrained(
5    "vllm-sr/Decision-2.0-Kai-0.6B", 
6    trust_remote_code=True,
7    device_map="auto"
8)
9
10result = model.system_one(
11    state="The order arrived damaged yesterday. The customer has a receipt and asks for a replacement today.",
12    questions={
13        "route": {
14            "type": "choice",
15            "instructions": "Which team should handle this request?",
16            "criteria": {
17                "returns": "Refunds, replacements, and damaged deliveries",
18                "billing": "Payments, invoices, and charges",
19                "technical": "Product setup and hardware faults"
20            }
21        },
22        "has_receipt": {
23            "type": "yes_no",
24            "instructions": "Does the customer mention having a receipt?"
25        },
26        "urgency": {
27            "type": "score",
28            "instructions": "How urgent is this request?",
29            "criteria": ["Routine", "Soon", "Immediate"]
30        }
31    }
32)
33
34print(json.dumps(result["answers"], indent=2))

While many users look to tools like Ollama to manage local models, models using specialized classification heads like Decision-2.0-Kai-0.6B are best integrated directly via Hugging Face Transformers, vLLM Semantic Router runtime, or exported ONNX runtimes to retain access to the structured probability output schema.

Evaluating Decision-2.0-Kai-0.6B vs competitors highlights how specialized training improves structured routing over generic small language models.

• Decision-2.0-Kai-0.6B vs GLiNER2.5-Decide: GLiNER2.5-Decide is a strong zero-shot classifier in the compact model space. On the JevArena benchmark, Decision-2.0-Kai-0.6B scores 48.6, outperforming GLiNER2.5-Decide's 42.5. Decision-2.0 also yields higher macro-F1 transfer on human-labelled benchmarks (45.9 vs 44.1) while offering wider native support for multi-criteria scoring.
• Decision-2.0-Kai-0.6B vs Bosun v3.1 0.6B: Bosun v3.1 matches the 0.6B scale, but scores 38.5 on JevArena and 34.2 on transfer tasks. Decision-2.0-Kai-0.6B maintains a 10-point lead on general decision accuracy, primarily due to its Qwen3 base architecture and expanded fine-tuning datasets.
• Decision-2.0-Kai-0.6B vs Decision 1.0 Kai: Compared to its direct predecessor, version 2.0 demonstrates significant progress: +12.7 points on JevArena (up from 35.9) and a massive improvement on the Jev Decision Index (16.3 vs 6.5).

If your local architecture requires deterministic, low-latency routing without allocating significant GPU memory, Decision-2.0-Kai-0.6B delivers best-in-class accuracy for sub-billion parameter decision models.