AIME24 Dataset Explained: What This Reasoning Benchmark Measures in Practice

A model scores 80% on AIME24 and a slide deck turns that into “it can do math.” That single number is doing far more work than it can bear. AIME24 is 30 problems from one 2024 competition, each with a single integer answer, and the way those 30 problems get scored varies so wildly between reports that a reported five-point lead can be one lucky sample.

That is the whole problem with reading AIME24 as a leaderboard rank. The benchmark is real, it is useful, and it measures something specific and narrow. What it does not do is certify that a model “can reason,” and treating it that way is how teams end up over-provisioning an expensive reasoning model for a job a cheaper one handles just as well.

What exactly is in AIME24?

AIME24 is the set of problems from the 2024 American Invitational Mathematics Examination — a qualifying competition for elite US high-school mathematics students. The AIME is run as two papers (AIME I and AIME II) of 15 questions each, so the full-year set most people call “AIME24” is 30 problems. Every answer is an integer from 0 to 999. There is no partial credit, no proof to grade, no free-text reasoning to score — the answer is either the correct integer or it is not.

That exact-match integer format is the source of both AIME24’s appeal and its instability. It makes automated scoring trivial: extract the final integer, compare, done. No LLM judge, no rubric, no human grader. But it also means the entire benchmark rests on 30 binary outcomes. Each problem is worth roughly 3.3 percentage points of the total score. A model that solves 24 of 30 reports 80%; solving one more problem moves it to 83.3%. There is no smooth gradient here — the resolution of the metric is coarse, and small sample noise gets amplified into headline-looking gaps.

The problems themselves demand chained arithmetic and symbolic manipulation: number theory, combinatorics, algebra, and geometry, usually requiring several correct intermediate steps to reach the final integer. This is what makes AIME24 a reasoning benchmark rather than a knowledge one. A model cannot pattern-match its way to the answer the way it might on a factual multiple-choice item; it has to carry a multi-step derivation without dropping a sign or misreading a constraint. That is genuinely informative. It is also a narrow slice of what “reasoning” means in a production system.

How are AIME24 scores computed, and why do they move so much?

Here is where the reported numbers start to diverge. A single AIME24 accuracy figure hides at least three methodology choices, and different labs make different ones.

Sampling temperature. Reasoning models often perform differently at temperature 0 (greedy, deterministic) versus a nonzero temperature that lets the model explore alternative derivations. Some reports run greedy decoding; others sample at temperature 0.6 or higher. The same model on the same 30 problems can land several points apart depending on this one setting.

Sample count and pass@k. Because outputs are stochastic at nonzero temperature, many reports draw multiple samples per problem and report an aggregate. pass@1 typically means the average accuracy over k independent samples (so it estimates the probability a single attempt succeeds), while pass@k or cons@k (majority-vote / self-consistency over k samples) reports whether any — or the most common — of k attempts is correct. A model reported at “pass@1 averaged over 64 samples” and the same model at “cons@64” are not comparable numbers, even though both are labelled with the model’s name on the same benchmark.

Prompt scaffolding. Chain-of-thought instructions, answer-extraction regexes, few-shot examples, and the maximum generation length all move the result. A reasoning model cut off before it finishes a long derivation scores lower for reasons that have nothing to do with its reasoning ability.

Put those together with a 30-problem set and the variance is structural, not accidental. A reported “5-point lead” between two models — roughly 1.5 problems — can easily be one or two samples falling the right way (this is an [observed pattern] in how these numbers are cited, not a claim from a single controlled benchmark). This is the same reproducibility trap that shows up when reading DeepSeek-R1 benchmark scores: the headline number is meaningless without the sampling protocol attached to it.

Quick-answer: what to check before you trust an AIME24 number

How many samples per problem? One sample on 30 problems has enormous variance.
Greedy or sampled, and at what temperature? These are different measurements.
pass@1, pass@k, or cons@k? Confirm the exact aggregation. They are not interchangeable.
Max generation length? Truncated reasoning depresses scores artificially.
AIME I only, AIME II only, or both? Some reports use 15 problems, not 30.

If a comparison does not disclose these, treat the ranking as unverified.

Why do reported AIME24 numbers differ so much between labs?

Everything above compounds. When one lab reports greedy pass@1 on AIME I only (15 problems) and another reports cons@64 on the full 30, the two numbers describe different experiments that happen to share a name. There is no shared harness the way MLPerf Client provides a controlled runner for inference throughput — AIME24 is a dataset, not a standardized evaluation protocol, and each report brings its own scaffolding.

Add to that the small-N problem: with 30 items, the confidence interval around any single accuracy figure is wide. Two models that are genuinely indistinguishable can report a several-point gap purely from sampling luck. Treating that gap as a capability difference is a statistical error dressed up as a benchmark result.

How does AIME24 compare with broader benchmarks?

AIME24 is deliberately narrow, and that is easiest to see against benchmarks built for different purposes. The point is not that one is better — it is that they answer different questions.

Benchmark	What it measures	Size / format	Best used for
AIME24	Chained arithmetic-symbolic reasoning	30 problems, exact-integer answer	Probing multi-step math reasoning depth
MMLU	Broad factual + academic knowledge	~14,000 multiple-choice, 57 subjects	General knowledge breadth
HELM	Multi-metric holistic evaluation	Many scenarios, multiple axes	Comparing models across many dimensions at once
SWE-bench	Real GitHub issue resolution	Thousands of software tasks	Agentic coding / repo-level capability

MMLU rewards breadth of recall; a high MMLU score says little about whether a model can hold a ten-step derivation together. HELM is explicitly multi-metric, designed so no single number stands in for capability. SWE-bench measures whether a model can actually fix a real codebase issue — a task that stresses tool use and long-context comprehension more than symbolic arithmetic. A model can top AIME24 and struggle on SWE-bench, or vice versa, because they load different abilities. The mistake is collapsing all of them into “how smart is this model.”

There is a useful analogy here to the Turing Test. The Turing Test is a user-experience evaluation pattern — it measures whether a system can pass as human in conversation, which is narrower than “intelligence” even though the popular framing conflates the two. AIME24 has the same shape: it measures one narrow, well-defined slice of reasoning, and the headline temptation is to read it as a proxy for general capability it was never designed to certify. The same discipline applies to reading any leaderboard — see how we treat Elo scores for LLMs as a relative-preference signal rather than an absolute capability measure.

What does a high AIME24 score not tell you?

A strong AIME24 result tells you a model can carry multi-step symbolic reasoning on competition-style problems with a single verifiable integer answer. It does not tell you the model will:

reason correctly over your domain’s messy, underspecified inputs (competition problems are clean and self-contained);
resist hallucinating a confident wrong answer when the problem is out of distribution;
perform under a latency and cost budget — a model that self-consistency-votes over 64 samples to hit its headline score is expensive to run that way in production;
generalize to open-ended tasks with no exact-match answer to grade against.

This is the gap between what a benchmark certifies and what a system can be asked to do — the same distinction at the heart of a GenAI feasibility audit. Knowing what a benchmark does and does not certify before it drives spend is what keeps a model-selection decision defensible.

How should a team actually use AIME24 for model selection?

Use it as one signal, read at the right resolution, with the protocol attached.

First, normalize the comparison. If you are comparing two models, run them yourself under identical sampling settings, or only compare published numbers that disclose the same protocol. A pass@1 greedy number against a cons@64 number is not a comparison.

Second, respect the coarse resolution. With each problem worth ~3.3 points, treat gaps smaller than a few problems as noise unless you have run many samples and looked at the variance. When two candidate models are statistically indistinguishable on AIME24, the reasoning benchmark has told you what it can — pick on cost, latency, and integration fit instead. Over-provisioning an expensive reasoning model where a cheaper one is indistinguishable is a spend decision made on benchmark theater.

Third, match the benchmark to the job. If your production task involves open-ended domain reasoning, retrieval, or tool use, AIME24 is at best a weak proxy. Pair it with an evaluation that actually resembles your workload — and where feasibility is the real question, that is exactly where a structured feasibility scoping earns its place, because it starts from what the system must do rather than which model tops a leaderboard.

FAQ

What’s worth understanding about the AIME24 dataset first?

AIME24 is the set of 30 problems from the 2024 American Invitational Mathematics Examination, each with a single integer answer from 0 to 999. In practice it works as an exact-match reasoning benchmark: a model produces a derivation, the final integer is extracted and compared to the key. It measures a narrow slice of chained arithmetic-symbolic reasoning, not general capability.

What exactly is in AIME24 — how many problems, what format, and where does the data come from?

It contains 30 problems drawn from the two 2024 AIME papers (AIME I and AIME II, 15 questions each). Every problem has a single integer answer, so scoring is exact-match with no partial credit. The problems span number theory, combinatorics, algebra, and geometry, each requiring several correct intermediate steps.

How are AIME24 scores computed, and why do pass@k, temperature, and sample count change the result so much?

Scores are the fraction of the 30 problems answered correctly, but the aggregation varies: pass@1 (average over k samples), pass@k, or cons@k (majority vote). Because the set is only 30 problems, each is worth ~3.3 points, and because outputs are stochastic at nonzero temperature, sample count and temperature move the number by several points. Two reports with different protocols are simply different experiments sharing a name.

How does AIME24 compare with broader benchmarks like MMLU, HELM, and SWE-bench for judging a model?

AIME24 probes multi-step math reasoning; MMLU measures broad factual knowledge across 57 subjects; HELM is a deliberately multi-metric holistic evaluation; SWE-bench measures real software-issue resolution. They load different abilities, so a model can lead one and trail another. Collapsing them into a single “how smart is this model” is the core error.

What are the limits of AIME24 — what does a high score not tell you about a model’s real-world capability?

A high score shows a model can carry clean, self-contained multi-step derivations to a verifiable integer. It does not show the model handles messy underspecified inputs, resists confident hallucination out of distribution, performs within a latency and cost budget, or generalizes to open-ended tasks with no exact answer to grade.

How should a team actually use AIME24 when selecting or evaluating a reasoning model?

Use it as one signal read at coarse resolution. Normalize the sampling protocol before comparing, treat gaps smaller than a few problems as noise, and when two models are statistically indistinguishable, decide on cost, latency, and integration fit instead. Match the benchmark to your actual workload rather than letting a headline number drive spend.

Why do reported AIME24 numbers differ so much between labs and papers?

Because AIME24 is a dataset, not a standardized harness, each report brings its own temperature, sample count, aggregation (pass@1 vs cons@k), scaffolding, and even problem subset (AIME I only vs the full 30). With only 30 problems the confidence interval is wide, so genuine ties can look like multi-point gaps. Without the disclosed protocol, a ranking is unverified.

Read this way, AIME24 stops being a scoreboard and becomes what it always was: a bounded probe of one kind of reasoning, useful only when you know the conditions it was run under. The open question for any team is not “which model wins AIME24” but “does the slice AIME24 measures resemble the reasoning my system will actually be asked to do” — and if the answer is no, the leaderboard was never the right input to the decision.