Benchmarks · 2021

MATH-500: MATH-500 Competition Math Subset

Name: MATH-500: MATH-500 Competition Math Subset
Creator: OpenAI (subset of the MATH dataset by Hendrycks et al.)
Published: 2021
Keywords: MATH-500, AI benchmark, text model evaluation, OpenAI (subset of the MATH dataset by Hendrycks et al.)

Five hundred curated competition math problems used as a fast, repeatable test of mathematical reasoning.

Open Dataset Read Paper

Models Tested

Top Score

98.3

Published

2021

Source

OpenAI (subset of the MATH dataset by Hendrycks et al.)

How It Works

MATH-500 is the curated 500-problem subset most labs report on when measuring step-by-step mathematical reasoning. The problems come from US high-school and undergraduate competitions, covering algebra, geometry, number theory, and combinatorics. Solutions require multi-step reasoning, not memorization.

Each problem has a single short-form answer. The model produces a final expression or number, scored by exact match against the reference. Most leaderboards report pass@1 with chain-of-thought, sometimes with majority voting across multiple samples.

Dataset size

500 problems sampled from the MATH benchmark, balanced across topic and difficulty.

Mean score

77.7

Median score

88.0

Open / Closed

8 / 8

Top Scorers

#	Model	Lab	Source	Score
01	DeepSeek-R1	DeepSeek	Open	98.3
02	Gemini 2.5 Pro	Google	Closed	96.7
03	Gemini 2.5 Flash	Google	Closed	93.2
04	Gemini 2.0 Flash	Google	Closed	93.0
05	GPT-4.1 Mini	OpenAI	Closed	92.5
06	GPT-4.1	OpenAI	Closed	91.3
07	Llama 4 Maverick	Meta	Open	88.9

Score Distribution

Open vs Closed Source

Gap on MATH-500:1.5pts open leads

Top Open-Source Models

1DeepSeek-R198.3
2Llama 4 Maverick88.9
3DeepSeek-V388.7

Top Closed-Source Models

1Gemini 2.5 Pro96.7
2Gemini 2.5 Flash93.2
3Gemini 2.0 Flash

Score vs Parameter Count

8 model(s) with undisclosed parameter counts not shown. Most closed-source labs do not publish model size.

Average Score by Lab

DeepSeek
93.5n = 2
Google
92.6n = 4
Meta
86.6n = 2
OpenAI
86.6n = 3
Mistral AI
42.5n = 4

Most Correlated Benchmarks

MMLU-PRO
+0.99n = 15
GPQA
+0.93n = 16
Arena Score
+0.92n = 16
SciCode
+0.89n = 16
IFBench

What It Captures Well

Fast and cheap to run, with stable scoring.
Covers a broad range of competition math topics.
Long enough to give a stable signal across model generations.

Where It Falls Short

The full MATH set is partially memorized by some models, which inflates scores.
Short-answer scoring penalizes correct reasoning that arrives at the wrong final form.
Older than AIME 2025; less contamination-resistant.

Frequently Asked Questions

Is MATH-500 the same as MATH?

No. MATH is the full 12,500-problem benchmark. MATH-500 is a curated subset of 500 problems chosen to be representative while staying fast to evaluate.