Benchmarks · 2026

JevBench: JevBench: System One Model Composite

Name: JevBench: JevBench: System One Model Composite
Creator: Benchmark Heaven
Published: 2026
Keywords: JevBench, AI benchmark, decision model evaluation, Benchmark Heaven

The main community board for System One models. One composite of intelligence, calibration, speed, and cost.

Open Dataset

Models Tested

Top Score

64.1

Published

2026

Source

Benchmark Heaven

Tools for AI Builders—All the hardware and tools you need to run private AI at your office or at home.Shop Now

How It Works

JevBench scores decision models on four axes: how often they are right (intelligence), how honest their probabilities are (calibration), how fast they answer, and what each decision costs. The composite rewards models that are good enough and cheap enough to run inside a hot path, which is where System One models are used.

Every model answers the same public and sealed items. Each axis is scaled to 0-100 and combined into one composite. Self-hosted latency is adjusted so local and API models compare fairly. The sealed set limits overfitting to published items.

Dataset size

Hundreds of typed Choice, Score, and Noul decisions, including 308 sealed items that are never published.

Mean score

36.2

Median score

39.4

Open / Closed

30 / 2

Top Scorers

#	Model	Lab	Source	Score
01	decider-4b v2	Mapika	Open	64.1
02	Jev 1.13	TypeSafe AI	Closed	63.3
03	Cygnet	blockbrain	Open	61.8
04	Hopper	HopitAI	Open	59.4
05	Winnow-12B	EldanRing	Open	55.6
06	reflex 4B	kshetrajna12	Open	54.0
07	djev	Maisa	Open	52.2
08	Jev-Omni	akhilaaa3	Open	51.3
AdTools for AI BuildersAll the hardware and tools you need to run private AI at your office or at home.Shop Now
09	Metask-Jev-4B	Wayfind (Metask AI)	Open	47.8
10	SemIf	TheoLeeCJ	Open	47.7
11	Jobe	MantisShrimpdev	Open	46.9
12	system-one-open	mithalouni	Open	45.1
13	spark-s1-4b v6	Abhishek Rai (Nokast)	Open	44.6
14	Malkuth-4B	newfull5	Open	44.5
15	decider-35b-a3b	Mapika	Open	41.2

Score Distribution

Open vs Closed Source

Gap on JevBench:0.8pts open leads

Top Open-Source Models

1decider-4b v264.1
2Cygnet61.8
3Hopper59.4

Top Closed-Source Models

1Jev 1.1363.3
2decision-machine-139.9

Score vs Parameter Count

3 model(s) with undisclosed parameter counts not shown. Most closed-source labs do not publish model size.

Average Score by Lab

Mapika
52.7n = 2
newfull5
41.7n = 2
FlyMy.AI
34.1n = 2
Zefan Cai
10.6n = 2

Most Correlated Benchmarks

Decision Index
+0.86n = 14
Pearson r: −1 to +1. Positive means the two benchmarks rank models in similar order; negative means the opposite.

What It Captures Well

Largest board for the category, with close to 90 ranked entries.
Sealed hold-out items make gaming the leaderboard harder.
Prices speed and cost, which matter more for decision models than for chat models.

Where It Falls Short

The composite punishes LLMs on cost and speed, so a very accurate LLM can rank low.
Many cost figures are estimates from the base model list price.
Some rows measure research previews rather than current checkpoints.

Frequently Asked Questions

Which model leads JevBench?

As of 24 September 2026, decider-4b v2 is first at 64.13 and Jev 1.13 is second at 63.29. Jev still leads on the intelligence axis.

Why do LLMs score low on JevBench?

The composite includes speed and cost. A frontier LLM can top the intelligence axis and still land far down the composite because each decision is slower and more expensive.