Benchmarks · 2026

Decision Index: Decision Index: Chance-Corrected Decision Score

Name: Decision Index: Decision Index: Chance-Corrected Decision Score
Creator: apolinario (Hugging Face)
Published: 2026
Keywords: Decision Index, AI benchmark, decision model evaluation, apolinario (Hugging Face)

Forty public benchmarks recast as typed decisions, scored so that random guessing equals zero.

Open Dataset

Models Tested

Top Score

57.9

Published

2026

Source

apolinario (Hugging Face)

Tools for AI Builders—All the hardware and tools you need to run private AI at your office or at home.Shop Now

How It Works

The Decision Index asks how much better than chance a model decides across intent routing, tool selection, retrieval checks, reasoning, and safety. Chance correction means a task with two options and a task with 150 options count on the same scale.

Each source benchmark is converted into typed decisions with a closed option set. Scores are chance-corrected per task so 0 is random guessing and 100 is perfect, then averaged across areas. Calibration (ECE) is reported alongside but is not part of the headline number.

Dataset size

40 public benchmarks in five areas, including BANKING77, CLINC150, BFCL, RouterBench, MMLU-Pro, BBH, When2Call, RAGTruth, HoVer, HellaSwag, WinoGrande, ANLI, and HLE.

Mean score

26.9

Median score

28.6

Open / Closed

47 / 1

Top Scorers

#	Model	Lab	Source	Score
01	Jev 1.13	TypeSafe AI	Closed	57.9
02	Rune 26B-A4B v3	Invergent	Open	57.4
03	AutoJev-27B	denis-pplx	Open	56.4
04	Winnow-12B	EldanRing	Open	50.0
05	Jevfire	kikoncuo	Open	49.4
06	decider-35b-a3b	Mapika	Open	47.1
07	Decision 1.0 Lux 9B	vLLM Semantic Router	Open	43.5
08	decider-4b v2	Mapika	Open	40.7
AdTools for AI BuildersAll the hardware and tools you need to run private AI at your office or at home.Shop Now
09	Jev-Omni	akhilaaa3	Open	40.5
10	djev	Maisa	Open	40.3
11	Winnow-E4B	EldanRing	Open	39.9
12	Hopper	HopitAI	Open	39.7
13	Bespoke Nimble 9B v2	Bespoke Labs	Open	39.6
14	lev	Interfaze	Open	38.5
15	Kev 9B	Jared Palmer	Open	38.5

Score Distribution

Open vs Closed Source

Gap on Decision Index:+0.5pts closed leads

Top Open-Source Models

1Rune 26B-A4B v357.4
2AutoJev-27B56.4
3Winnow-12B50.0

Top Closed-Source Models

1Jev 1.1357.9

Score vs Parameter Count

Average Score by Lab

EldanRing
45.0n = 2
Mapika
38.9n = 3
Jared Palmer
29.2n = 3
InternLM
23.0n = 3

Most Correlated Benchmarks

JevBench
+0.86n = 14
Pearson r: −1 to +1. Positive means the two benchmarks rank models in similar order; negative means the opposite.

What It Captures Well

Widest task coverage of any decision board.
Chance correction makes tasks with very different option counts comparable.
Reproduction kit is public on GitHub.

Where It Falls Short

Recast public benchmarks may have leaked into training data.
Editions re-weight tasks, so scores move between versions (Jev went from 51.67 on 0.2 to 57.89 on 0.2.1).