Benchmarks · 2026

Decision Index: Decision Index: Chance-Corrected Decision Score

Name: Decision Index: Decision Index: Chance-Corrected Decision Score
Creator: apolinario (Hugging Face)
Published: 2026
Keywords: Decision Index, AI benchmark, decision model evaluation, apolinario (Hugging Face)

Forty public benchmarks recast as typed decisions, scored so that random guessing equals zero.

Open Dataset

Models Tested

Top Score

73.4

Published

2026

Source

apolinario (Hugging Face)

Tools for AI Builders—All the hardware and tools you need to run private AI at your office or at home.Shop Now

How It Works

The Decision Index asks how much better than chance a model decides across intent routing, tool selection, retrieval checks, reasoning, and safety. Chance correction means a task with two options and a task with 150 options count on the same scale.

Each source benchmark is converted into typed decisions with a closed option set. Scores are chance-corrected per task so 0 is random guessing and 100 is perfect, then averaged across areas. Calibration (ECE) is reported alongside but is not part of the headline number.

Dataset size

40 public benchmarks in five areas, including BANKING77, CLINC150, BFCL, RouterBench, MMLU-Pro, BBH, When2Call, RAGTruth, HoVer, HellaSwag, WinoGrande, ANLI, and HLE.

Mean score

41.1

Median score

41.4

Open / Closed

87 / 1

Benchmark data sourced from: Hugging Face, Arena.ai, Artificial Analysis.

Top Scorers

#	Model	Lab	Source	Score
01	StartLux-Decision v1.1 27B AutoThink	startlux-models	Open	73.4
02	StartLux-Decision v1.1 27B Instant	StartLux	Open	72.9
03	Nimble-v4-max 27B	Bespoke Labs	Open	72.7
04	GLiDE 27B	Fastino	Open	72.5
05	d3-turbo 27B	vLLM Semantic Router	Open	71.6
06	vLLM-SR d3-ultra 27B	vLLM Semantic Router	Open	71.6
07	KnowLine-31B-Gen1	PelaAI	Open	70.1
08	H2O-Lightning-31B	H2O.ai	Open	70.0
AdTools for AI BuildersAll the hardware and tools you need to run private AI at your office or at home.Shop Now
09	StartLux-Decision v1.1 9B Instant	StartLux	Open	70.0
10	KnowLine-35B-A3B-Gen1	PelaAI	Open	69.5
11	MISA AI 2.0 Decision Plus · oak 1.1	MISA JSC	Open	68.2
12	StartLux-Decision v1.1 4B Instant	StartLux	Open	67.8
13	Perplexity Decider v1.2 27B	Perplexity	Open	67.7
14	KnowLine-26B-A4B-Gen2	PelaAI	Open	66.9
15	Turion Jev 31B	Turion.AI	Open	66.2

Score Distribution

Open vs Closed Source

Gap on Decision Index:12.7pts open leads

Top Open-Source Models

1StartLux-Decision v1.1 27B AutoThink73.4
2StartLux-Decision v1.1 27B Instant72.9
3Nimble-v4-max 27B72.7

Top Closed-Source Models

1Jev 1.1360.7

Score vs Parameter Count

Average Score by Lab

PelaAI
68.9n = 3
H2O.ai
65.0n = 3
StartLux
64.7n = 5
Perplexity
63.4n = 3

Most Correlated Benchmarks

JevBench
+0.86n = 14
Pearson r: −1 to +1. Positive means the two benchmarks rank models in similar order; negative means the opposite.

What It Captures Well

Widest task coverage of any decision board.
Chance correction makes tasks with very different option counts comparable.
Reproduction kit is public on GitHub.

Where It Falls Short

Recast public benchmarks may have leaked into training data.
Editions re-weight tasks, so scores move between versions (Jev went from 51.67 on 0.2 to 57.89 on 0.2.1).