AutoJev-27B

AutoJev-27B is a full-weight fine-tune of Qwen3.8-27B that does one thing: given a question and a set of supplied options, it returns a probability distribution over those options in a single forward pass. It does not write essays, does not produce chain-of-thought, and does not stream tokens. It outputs a calibrated decision.

That narrowness is the point. General-purpose chat models are bad at this task — they hedge, they're verbose, and their confidence scores are uncalibrated. AutoJev-27B is trained specifically to fix that, and the numbers back it up. Against its own base model on text decision benchmarks, it goes from 69.83% to 84.60% overall accuracy, while cutting expected calibration error (ECE) from 0.06483 to 0.04282 and Brier score from 0.40834 to 0.22027. Those are large, meaningful gaps in a domain where a few points of calibration matter more than raw accuracy.

The model was produced by denis-pplx, and the process is unusual enough to note: research, data generation, training, evaluation, and deployment were executed by autonomous agents. A human set the goals and refined scope. The result is a 26.09B dense model with a TypeSafe-compatible REST API and a browser playground, released under Apache-2.0 weights with MIT-licensed code.

Where does it sit? It's a decision layer, not a general assistant. The closest architectural sibling is autotrust/JEV-27B, which pairs System 1 decision heads with an untouched System 2 generator in one weight set. AutoJev-27B takes the opposite approach: one behavior, one forward pass, no generation at all. If your agent already has a main model and you need a fast, calibrated arbiter for routing, gating, and classification, this is the category you're shopping in.

AutoJev-27B is dense, not Mixture-of-Experts. All 26.09B parameters are active on every forward pass. That has direct consequences for how you deploy it:

• No active-parameter discount. Unlike a 30B MoE with 3B active, you don't get small-model speed at large-model quality. You pay full memory bandwidth on every token.
• Predictable memory. Weights are weights. No expert routing means no worst-case VRAM surprises from uneven expert activation.
• Simple serving. No expert-parallel sharding logic, no router tuning. Standard tensor parallelism works.

The inference pattern is unusual and worth internalizing before you size hardware. Because the model answers in one forward pass, prefill dominates and decode is nearly irrelevant. A question with 2,000 tokens of context and 20 candidate options is one long prefill followed by a handful of output positions. Throughput measured in tokens/second is the wrong metric for planning; latency per decision is the right one. A setup that looks mediocre for chat can be perfectly adequate here.

Training used full-weight supervised fine-tuning on one H200: 73,000 unique examples, 286 updates, cross-entropy loss, with a scalar temperature fit separately for calibration. The released checkpoint is update 200. The curated training corpus is not bundled, so you cannot reproduce the fine-tune from the repo alone.

Context length is not specified on the model card or repository. Base Qwen3.8-27B support is the practical upper bound, but verify empirically before you build a pipeline around a specific window. Training cutoff is likewise unpublished.

The model's surface area is the POST /v1/systemone endpoint, which accepts three question types:

• `choice` — pick among labeled options, return probabilities per option.
• `noul` — a binary/threshold decision without a fixed option list.
• `score` — return a numeric score for a candidate.

Optional base64 images can be attached to a request. Be careful here: published benchmarks measure text decisions only, and the model card explicitly states image support is not a natural-image accuracy claim. Treat vision as an experimental input path, not a validated capability.

Concrete workloads this fits:

• Agent tool-call gating. Return allow, confirm, review, or deny before an action executes. The calibration matters more than the label here — you want a 0.7 confidence to actually mean 70%.
• Model routing. Given task stakes, cost, latency, and context requirements, pick from an allowed model list.
• Research triage. Accept a claim, request more verification, or reject it.
• Completion review. Decide whether work is done, needs verification, or is incomplete.
• Batch labeling and data generation. 73,000 training examples were generated with this class of pipeline; the same pattern works for building your own labeled sets.

What it is not for: open-ended generation, long-form reasoning, code synthesis, or anything requiring a response longer than a decision. It's a component, not an assistant.

Approximate weight footprints for 26.09B parameters:

Add 2–4 GB for KV cache and runtime overhead depending on context length.

• RTX 4090 / 3090 (24 GB): Q4_K_M or Q5_K_M are the sweet spot. Q6_K fits but leaves little headroom for long contexts.
• RTX 5090 (32 GB): Q8_0 or FP8 with room to spare. This is the best single-GPU consumer option.
• Mac M4 Max (128 GB unified): BF16 weights fit, but memory bandwidth (~546 GB/s) caps prefill speed. Q4_K_M is the pragmatic choice.
• Mac M3 Ultra (192 GB): BF16 fits comfortably; bandwidth (~819 GB/s) makes it usable for batch work.
• 16 GB cards (RTX 4080, 4060 Ti 16GB): Q4_K_M with partial CPU offload. Expect a real latency penalty on long prompts.

There is no official GGUF yet, and the repo is tagged custom-code — the decision head needs the shipped loader, so a stock llama.cpp or Ollama build won't load these weights without a conversion effort. The practical path is the included server:

1git clone https://github.com/denis-pplx/autojev.git
2cd autojev
3uv sync --frozen --python 3.12
4uv run hf download denis-pplx/autojev-27b --local-dir checkpoints/selected
5AUTOJEV_CHECKPOINT=checkpoints/selected uv run autojev-serve

Requires Python 3.12+ and uv. The playground is at http://localhost:8000, API docs at /docs. Set AUTOJEV_API_KEY to enable authentication. If the weights are gated, run uv run hf auth login first.

Because decode is negligible, plan around prefill throughput. Rough single-stream expectations for Q4_K_M:

• RTX 4090: ~25–35 tok/s equivalent on decode, but decisions in the low hundreds of milliseconds for typical prompts
• RTX 5090: roughly 1.7× the 4090
• M4 Max: ~15–20 tok/s
• H100 BF16: ~80–100 tok/s

For batch scoring, vLLM with tensor parallelism across two 24 GB cards is the efficient configuration.

vs. base Qwen3.8-27B. Same weights, different objective. The base model is a general assistant that can be prompted to pick an option; AutoJev-27B is trained to do only that. It wins by 14.8 accuracy points with materially better calibration (ECE 0.04282 vs 0.06483). If you need generation and decisions, keep the base model. If you need decisions only, this is strictly better and faster.

vs. autotrust/JEV-27B. Also built on Qwen3.8-27B, but with a dual System 1 / System 2 design served through vLLM — one weight set handling both fast decisions and full generation. JEV-27B scores a 84.07 six-group mean across JevBench, Kev, OpenJev, Nimble, VitaminC, and MASSIVE-en; AutoJev reports 84.60 overall on its own suite. These are different benchmark suites and not directly comparable. The real tradeoff is architectural: JEV-27B gives you one engine for both jobs at the cost of a more complex serving stack. AutoJev-27B gives you a smaller, simpler, single-purpose component that drops into an agent that already has a generator.

vs. a general 24–32B dense model (Qwen-class or Mistral Small-class). Those are more flexible and better documented, with mature GGUF and Ollama support. AutoJev-27B beats them decisively on calibrated probability output — which is the only reason to run it. Choose it when your pipeline consumes probabilities, not text.