Bespoke Nimble 9B v2

Bespoke Nimble 9B v2 is a LoRA adapter for Qwen3.5-9B that turns text into typed decisions. It does not generate prose. Given a context and a schema of questions — each a boolean, a categorical choice, or an ordinal score — it returns the selected answer plus a probability distribution over all allowed answers. Bespoke Labs built it for one job: fast, calibrated, local decision-making.

The model is a 9.65B dense adapter, text-only, with an 8,192-token context window and support for up to 255 choices per field. It is Apache-2.0 licensed, and the training recipe, data curation method, and serving code are open. Bespoke Labs explicitly notes the model is inspired by TypeSafe's Jev (System One approach) but was not distilled from it.

This matters because most local models are generalists. You prompt them, they generate text, and you parse the output. That is slow and error-prone. Nimble takes the opposite approach: it reads the prompt once, scores one answer token per field, and stops. On 324 held-out examples, Bespoke-Nimble-9B matches 90.1% of reference labels, compared with 66.4% for its Qwen3.5-9B base model and 93.2% for Jev 1.13.0. For practitioners who need local, deterministic, probability-aware classification without cloud APIs, this is a purpose-built tool.

Nimble is a PEFT LoRA adapter — approximately 173 MB — that requires the separately downloaded Qwen/Qwen3.5-9B base checkpoint (revision c202236235762e1c871ad0ccb60c8ee5ba337b9a). It is not a standalone model. The base is a dense 9.65B-parameter transformer, so all parameters are active during inference. There is no MoE routing, no sparse activation. Every forward pass uses the full weight set.

The context contract is 8,192 tokens. Oversized inputs are rejected, not truncated. This is a deliberate design choice: silent truncation would corrupt the decision. Serving supports up to 255 choices per field, but the largest choice set seen during training was 24. The 255-choice limit describes runtime capability, not measured accuracy at that width. Long schemas can hit the context limit before reaching 255 options.

The scoring mechanism is the key architectural detail. The model does not generate reasoning or free-form answers. It computes logits for the allowed answer tokens and applies a softmax to produce probabilities. The v2 adapter ships with a calibrated temperature of T=2.179078721266035. If you load only the LoRA weights and apply a plain softmax, you get T=1.0 and silently miscalibrated probabilities. The included ParallelScorer helper applies the correct temperature automatically; the helper rejects T=1.0 unless you explicitly pass allow_uncalibrated=True. For custom runtimes, compute softmax(candidate_logits / 2.179078721266035) exactly once.

The adapter retains the original A–Z answer codes for schemas with 26 or fewer choices. Larger schemas use additional uppercase vocabulary tokens, each verified to be exactly one token at the assistant answer boundary. No vocabulary expansion, new embeddings, or retraining is required. The package includes the tokenizer, pinned prompt contract, reference scoring code, and checksums.

Nimble handles three decision types: boolean yes/no judgments, categorical choices from a provided list, and ordinal scores (integer-valued enums such as ["0", "1", "2"]). For ordinal fields, you can request a probability-weighted expected score by passing the field name in score_fields. Every field returns the chosen answer and the probability of each allowed answer.

Concrete use cases where this model excels:

• Content moderation and safety flags — "Does this message violate policy X?" with a calibrated probability you can threshold.
• RAG relevance scoring — "Is this retrieved passage relevant to the query?" as a boolean or a 0–3 ordinal score.
• Intent classification — route user input to one of N predefined categories without generating text.
• Form field extraction with confidence — "Is the date in the past?" or "Which department does this ticket belong to?"
• Survey and rubric coding — assign ordinal scores to open-ended responses using a rubric you define.
• A/B preference and pairwise judgment — "Which response is better?" with probabilities over the two options.
• Structured data validation — check whether a record satisfies a set of boolean conditions.

What it is not for: open-ended generation, summarization, translation, code generation, or any task that requires producing free-form text. The model is text-only and primarily English (language: en in the model card). If you need a chat model, look elsewhere. If you need a fast, local, probability-aware decision layer, this is the right shape.

Running Bespoke Nimble 9B v2 locally means running the Qwen3.5-9B base plus the LoRA adapter. The adapter is small (~173 MB), but the base model dominates VRAM. Here are realistic VRAM requirements for the combined model at different quantization levels:

• BF16 (unquantized): ~19.3 GB for weights, plus 1–2 GB for KV cache at 8k context. Budget 21–22 GB.
• Q8_0: ~10–11 GB weights, ~12 GB total.
• Q6_K: ~8–9 GB weights, ~10 GB total.
• Q4_K_M: ~5.5–6 GB weights, ~7 GB total.
• Q3_K_M: ~4.5 GB weights, ~5.5 GB total.
• Q2_K: ~3.5 GB weights, ~4.5 GB total.

Best GPU for Bespoke Nimble 9B v2: An RTX 4090 (24 GB) runs BF16 comfortably and Q8 with headroom. An RTX 4080 (16 GB) runs Q8. An RTX 3060 12 GB runs Q4_K_M or Q5_K_M. On Apple Silicon, an M4 Max with 64 GB or 128 GB unified memory runs BF16; an M4 Pro with 24 GB runs Q8; an M2 or M3 with 16 GB runs Q4_K_M.

Recommended quantization: For most users, Q4_K_M is the sweet spot. It fits on 8–12 GB GPUs and preserves decision quality well. If you rely on the probability outputs for thresholding or calibration, use Q8_0 or BF16. Quantization can shift the softmax distribution, and the calibration temperature was fitted for the unquantized adapter. Test your thresholds after quantizing.

Expected performance: Because Nimble scores one token per field, tokens-per-second is the wrong metric. Think in decisions per second. On an RTX 4090 at BF16, a 2,000-token prompt prefills in roughly 150–300 ms, and the answer token is scored in a single step. That yields sub-second latency per field, and batching multiple fields or requests pushes throughput into the tens of decisions per second. On a 12 GB GPU at Q4_K_M, expect 2–5× slower prefill, but still interactive latency for typical prompts. The repository describes the model as "blazing fast" for this reason: no autoregressive decoding loop.

Ollama and llama.cpp: The quickest way to get started is the reference inference.py with a CUDA GPU or Apple Silicon. If you prefer Ollama, you will need to convert the base model and LoRA adapter to GGUF and use a Modelfile with the ADAPTER directive. llama.cpp can apply the adapter at runtime with --lora. Note that the reference scorer applies the calibration temperature automatically; custom runtimes must apply it manually. The code repository does not carry a license, though the model weights are Apache-2.0.

Setup requirements: Python 3.11, PyTorch 2.8.0 with CUDA 12.8, and a BF16-capable GPU. The helper processes fields individually and preserves the training prompt and probability calculation. No trust_remote_code=True is required.

Bespoke Nimble 9B v2 vs Qwen3.5-9B base: The base model is a general-purpose text generator. It can answer questions, but it generates tokens, requires prompt engineering or constrained decoding, and does not natively return calibrated probabilities over a fixed answer set. Nimble matches 90.1% of reference labels on held-out decisions; the base matches 66.4%. Nimble is also faster because it does not decode a sequence. Choose Nimble when you need structured decisions; choose the base when you need free-form text.

Bespoke Nimble 9B v2 vs Jev 1.13.0: Jev is the closed-source System One model that inspired Nimble. Jev scores 93.2% on the same held-out set, about 3 points higher. But Jev is not open, not local, and not Apache-2.0. Nimble runs on your hardware, ships with the training recipe, and lets you inspect the probabilities. If you need the absolute highest accuracy on this decision task and can use a hosted service, Jev wins. If you need local execution, open weights, and reproducibility, Nimble is the practical choice.

Bespoke Nimble 9B v2 vs Llama 3.1 8B Instruct with Outlines or JSON schema: Llama 3.1 8B is a strong general instruct model, but using it for typed decisions means generating text and parsing it, or constraining decoding with a grammar. That is slower, and the resulting probabilities are not calibrated for the decision. Nimble is purpose-built for this task, returns native probability distributions, supports ordinal scoring with expected values, and runs a single forward pass per field. For high-volume local decision pipelines, Nimble is the more efficient architecture.