Line up three vendors’ published numbers and pick the highest one. It looks like a comparison. It isn’t. Each of those numbers came from a different dataset, a different prompt template, different hardware, and a different definition of what “accuracy” even means. When you rank models by numbers measured under conditions you don’t control, you aren’t comparing the models — you’re comparing whose marketing team found the most favourable test.

A genuine AI performance comparison holds the evaluation fixed and lets the model be the only variable. Same task inputs, same harness, same scoring, same hardware envelope. Change one model, change nothing else, read the delta. That is the only construction a procurement committee can defend, and it is surprisingly rare in practice because it takes real engineering to set up.

Why can’t you compare models by lining up published vendor numbers?

Consider what happens when Vendor A reports 89% on a reasoning benchmark and Vendor B reports 91%. The instinct is that B is 2 points better. But B may have used a 5-shot prompt where A used zero-shot. B may have scored with an exact-match rule that A rejected as too brittle. B may have run at a temperature setting that suppressed variance, or on a curated subset of the dataset, or with a system prompt tuned specifically for that eval. None of these choices are dishonest on their own. Together they make the two numbers non-comparable — the 2-point gap is noise from methodology, not signal about the model.

This is the apples-to-oranges failure, and it has a direction: it rewards whoever tested under the most favourable conditions, not whoever performs best on your workflow. A model that leads a public leaderboard can rank lower once you run it on your own task-representative inputs, because the leaderboard’s inputs are not your inputs. We see this pattern regularly in procurement engagements — the committee’s shortlist, built from published numbers, reshuffles the moment every candidate runs through one shared harness.

The deeper issue is that a benchmark number is a tuple, not a scalar. It encodes (model, dataset, prompt, decoding config, scoring rule, hardware). When you compare two published numbers, you are silently assuming five of those six fields are identical across vendors. They almost never are. Public leaderboards like Chatbot Arena and MLPerf each fix some of that tuple — which is exactly why they are useful and exactly why they still can’t stand in for a comparison on your task; we cover that boundary in what Chatbot Arena can and can’t replace in a spec-driven eval.

What has to be held fixed to make a comparison valid

The rule is simple to state and demanding to execute: everything except the model must be identical across candidates. Four things carry most of the weight.

The dataset. Every model sees the same inputs, drawn from — or representative of — your actual workflow, not a public academic set. If your production traffic is customer-support tickets, the eval set is customer-support tickets, not MMLU.

The prompt template. The same system prompt, the same few-shot examples (or zero-shot for all), the same formatting. A model that only shines with a bespoke prompt is telling you something, but you find that out in a separate prompt-sensitivity pass, not by silently advantaging it in the main run.

The scoring definition. One scoring rule applied identically to every candidate’s output. If you use an LLM-as-judge, it is the same judge model with the same rubric for all candidates, and you account for its own bias. If you use exact match, it is the same normalisation for all.

The run conditions. Same hardware class, same decoding parameters where they’re comparable, same concurrency and load profile. Latency and cost-per-request are meaningless unless the serving conditions are pinned — a point we develop in which machine-learning model metrics actually decide a serving config.

Hold those four fixed and the model becomes the only free variable. That is the level field.

How do you build a shared harness that runs every candidate the same way?

A shared harness is a single evaluation program that accepts a model behind a uniform interface, feeds it the fixed input set, captures raw outputs, and scores them with fixed logic. The engineering discipline is that no candidate gets a code path the others don’t.

In practice the interface is usually an OpenAI-compatible endpoint or a thin adapter per model, so a hosted API, a self-hosted vLLM deployment, and an SGLang server all present the same call signature to the harness. The harness logs the full request-response pair for every item, not just the score, so a disputed ranking can be re-examined against raw evidence rather than re-run from scratch. Decoding config is declared once and applied to all — you do not let one model default to greedy decoding while another samples at temperature 0.8.

Frameworks like lm-evaluation-harness, HELM, or a custom runner built on your own task data all work; the framework matters less than the invariant it enforces. When we build these for clients, the harness captures per-item quality, per-request latency (time-to-first-token and total), and token counts for cost, all in one pass, so the three axes come from the same run rather than three separate exercises that drift apart.

One structural warning. If a candidate needs a fundamentally different prompt or a different post-processing step to produce valid output, that is real information — but it does not belong hidden inside the harness as a per-model special case. Record it as a separate finding about that model’s integration cost. Otherwise the “level field” quietly tilts.

Combining quality, latency, and cost into one table

A committee cannot act on three separate rankings that disagree with each other. The job is to produce a single table where each row is a candidate and each column is an axis measured on the same run, so trade-offs are visible in one place.

Here is the shape of a defensible comparison table. The numbers are illustrative — the point is the structure, not the values.

Candidate	Task quality (fixed scorer)	p95 latency	Cost / 1k requests	Integration notes	Evidence class
Model A	0.84	1.9 s	$2.10	Standard adapter	benchmark (this harness)
Model B	0.86	3.4 s	$4.80	Standard adapter	benchmark (this harness)
Model C	0.85	2.1 s	$1.70	Needs output reformatter	benchmark (this harness)

Read this table and the decision is no longer “who scored highest.” Model B leads on quality by 2 points but costs more than twice as much per request and is nearly twice as slow at p95. Model C nearly matches B on quality, is cheapest, and is fast — but carries an integration cost the others don’t. That is a real trade-off a committee can debate, precisely because every number in the table came from one harness on one input set. The “Evidence class” column matters: these are benchmark-class numbers from a named, reproducible harness, not vendor claims, and labelling them that way is what lets the ranking survive scrutiny. For the quality-axis metric choices themselves, our note on the AI performance benchmark and what to measure goes deeper.

Do not collapse the three axes into a single weighted score too early. A single blended number hides the trade-off the committee is there to make and invites arguments about the weights instead of the models. Present the axes side by side; let the decision-makers own the weighting.

When does a model genuinely win, and when is the difference noise?

The most common mistake after building a fair harness is over-reading small gaps. A 1-point quality difference on a 300-item eval set is very likely within noise. Before you declare a winner, you need a sense of the eval’s variance.

Two cheap practices handle most of this. First, run each candidate two or three times if there’s any sampling in the decoding path; if a model’s own score moves by more than the gap to its neighbour across reruns, the gap is not real. Second, bootstrap a confidence interval over the item scores — if the intervals for two models overlap heavily, treat them as tied on quality and let latency and cost break the tie. This is an observed-pattern guideline from evaluation work, not a fixed threshold; the right interval width depends on your set size and how costly a wrong pick is.

The discipline here is restraint. A fair harness can produce a ranking with more precision than the data supports, and a confident-looking table is exactly the thing a committee will over-trust. Say plainly when two models are statistically indistinguishable on quality. That honesty is what makes the rest of the table credible.

Presenting a ranking that holds up under procurement scrutiny

A comparison ranking faces two kinds of challenge in review: a vendor disputing the method, and an internal stakeholder questioning the pick. Both are answered by the same artefact — the harness itself, plus the logged evidence.

The ranking survives scrutiny when you can show, on demand: the exact input set (or its provenance), the prompt template applied identically to all, the scoring code, the raw per-item outputs for every candidate, and the run conditions. A challenge to “did you test them the same way?” is then answered by pointing at code and logs rather than by re-running the entire evaluation — which is the round of re-eval that a controlled comparison exists to eliminate. A defensible ranking also needs to meet the evidence bar of your governance process; connecting the harness output to those requirements is where the governance-grade evidence layer closes the loop.

For a worked treatment of the whole candidate-comparison decision — shortlist construction, tie-breaking, sign-off — the companion piece on comparing AI model candidates for a procurement decision walks the full flow, and the broader MLPerf-and-procurement view covers where standardised public benchmarks fit alongside your own harness. The shared harness that produces all of this is the core of our [production AI monitoring and validation harness](Production AI Monitoring Harness), and the same discipline underpins how we think about AI infrastructure decisions across the board on the main practice page.

FAQ

What does working with ai performance comparison involve in practice?

It means running every candidate model through one shared evaluation harness on the same task-representative inputs, with the same prompt template, scoring rule, and run conditions. In practice that makes the model the only variable, so the resulting numbers reflect the models rather than differences in how each was tested. The output is a single ranking table a committee can act on.

Why can’t you compare models by lining up their published vendor benchmark numbers side by side?

Because each vendor number encodes a different dataset, prompt, decoding config, scoring rule, and hardware — not just a different model. Comparing two such numbers silently assumes five of those six fields match across vendors, which they almost never do. The gap you read is largely methodology noise, and it rewards whoever tested under the most favourable conditions rather than whoever performs best on your workflow.

What has to be held fixed — dataset, prompts, hardware, metric definitions — to make a comparison valid?

All four, applied identically to every candidate: the same input set drawn from your workflow, the same prompt template, one scoring definition, and the same hardware class and decoding conditions. Only the model changes. If a candidate needs a different prompt or post-processing step, that is recorded as a separate integration finding, not buried inside the harness as a per-model exception.

How do you build a shared harness that runs every candidate model on the same task-representative inputs?

You put every model behind a uniform interface — usually an OpenAI-compatible endpoint or a thin per-model adapter — and feed all of them the fixed input set through one program that scores with fixed logic and logs the full request-response pair for each item. No candidate gets a code path the others don’t. Frameworks like lm-evaluation-harness or HELM work; the invariant they enforce matters more than the framework.

How do you combine quality, latency, and cost-per-request into a single comparison table a committee can act on?

Measure all three axes on the same harness run and lay them out as columns against each candidate row, so trade-offs are visible in one place. Keep them as separate columns rather than collapsing them into one weighted score, which would hide the trade-off and invite arguments about weights instead of models. Let the committee own the weighting.

When does one model genuinely ‘win’, and when is the difference within noise or just favourable test conditions?

A model genuinely wins when its lead exceeds the eval’s variance — check by rerunning candidates with any sampling in the decoding path and by bootstrapping a confidence interval over item scores. If reruns move a score by more than the gap to a neighbour, or if the intervals overlap heavily, treat the models as tied on quality and break the tie on latency and cost. Say plainly when two models are statistically indistinguishable.

How do you present a comparison ranking so it holds up under procurement scrutiny?

Ship the harness and its evidence alongside the ranking: the input set and its provenance, the identical prompt template, the scoring code, the raw per-item outputs for every candidate, and the run conditions. A challenge to “did you test them the same way?” is then answered by pointing at code and logs rather than by re-running the whole evaluation. Label the numbers as reproducible harness results so they meet your governance evidence bar.

The honest version of this work ends with a ranking you can defend and a boundary you can name: which gaps are real, which are within noise, and which candidates the harness could not separate. Whoever owns the sign-off should be able to point at the harness and say the model was the only thing that changed — if they can’t, the comparison isn’t finished.