A benchmark result is trustworthy in proportion to how precisely it says what it left out. That is the uncomfortable part: the integrity of a number does not come from the measurement being exhaustive, it comes from the measurement being bounded on purpose and saying so.
Most disputes about benchmark credibility are not really disputes about the measurement. They are disputes about an unstated boundary. Someone reads a throughput figure produced on a fixed model catalogue, mentally substitutes their own fine-tuned model and their own batch policy, finds the deployed system slower, and concludes the benchmark was wrong. The benchmark was not wrong. It was silent — and silence about scope is what invites the over-read.
So this article does the opposite. It states, plainly, what a LynxBenchAI run does not cover.
What a run deliberately leaves outside the boundary
A LynxBenchAI measurement is scoped to four things it does not attempt, and each exclusion is a design decision rather than an omission.
It does not measure your application. A run exercises the fixed model catalogue of one named release. Those models were chosen so that two devices can be compared on identical work; they were not chosen because they resemble your production graph. The score speaks about that catalogue.
It does not measure post-settling behaviour. A run discards a warm-up phase, then counts completed iterations inside a continuous timed measurement window. It does not wait for a device to reach thermal equilibrium, and it does not include a soak or endurance period. Whatever a card does in hour three of a sustained job is outside the boundary.
It does not produce a distribution. Each test is one timed measurement window. Not a median of five trials, not a trimmed mean, not a percentile band. One window, reported as one window.
It does not infer what it did not run. If a boundary is not declared, it cannot be recovered from the score. A number carries no information about the conditions it was not produced under.
The measurement window, and what “declared” buys you
The mechanics matter here because they are the part readers most often reconstruct incorrectly.
A run first executes a warm-up phase whose iterations are discarded. This exists because the first iterations of a workload are dominated by things that have nothing to do with steady-state execution: kernel autotuning, cuDNN algorithm selection, memory pool growth, JIT or graph compilation under torch.compile, TensorRT engine warm-up paths. Counting those iterations would measure compilation, not computation. Discarding them is not generosity toward the hardware; it is the removal of a known confound.
Then the timed window opens. Inside it, completed iterations are counted at a workload size that has already been scaled up until throughput stops improving — the saturation point, which is where a device’s sustained practical peak actually lives rather than at whatever batch size happened to be convenient. The window is continuous. It is not restarted, not stitched from fragments, not extended because a result looked disappointing.
The window length is declared. That is the whole integrity claim, and it is deliberately modest: a measurement window earns trust by being stated, not by being long. A thirty-second declared window and a thirty-minute declared window are both honest. An undeclared window of any length is not, because the reader cannot tell whether it covers steady state, a thermal transient, or a lucky burst.
This is also why extending the run until it “feels exhaustive” is the wrong instinct. Every extension pushes the measurement toward a different question — endurance, thermal sustainability, driver stability over hours — without ever arriving at an answer, and it makes the boundary fuzzier rather than wider. The honest move is to draw the line where the measurement is actually valid and publish the line alongside the number.
Why one timed window is reported as one timed window
There is an obvious objection: single-shot measurements have variance, so why not repeat and report a median?
Because a median across repeated trials is a different claim, and presenting one thing as the other is the substitution that erodes trust. A median implies a distribution was characterised, that outliers were identified and handled by a stated rule, and that the reported figure is a central tendency the reader can reason about statistically. A single timed window implies none of that. It says: this device, this software stack, this workload, this window, this many completed iterations.
If we reported a median, the honest version would require declaring the trial count, the dispersion, and the outlier policy — a larger and slower measurement whose extra machinery buys a different kind of confidence than the one the result is used for. What a reader comparing two executors needs is that both were measured the same way, with the boundary visible. Reproducibility here comes from the run being repeatable by the reader, not from the number being pre-averaged on their behalf.
Take the variance seriously, though. A single window is more sensitive to a noisy host, a background process, or an unlucky clock state than an averaged figure would be. That is a real limitation of the method, stated as one, and it is precisely why the run is designed to be re-executed rather than trusted on faith.
What happens when a workload fails
The most consequential boundary decision is not about time at all. It is about what a benchmark does with things that do not work.
A precision a device cannot execute is scored as zero, not quietly skipped. A workload that fails or times out is recorded as zero, not dropped from the set. This looks harsh, and it is intentional.
Recording the zero is what makes the failure visible. A skipped test disappears from the result; a zero-scored test appears in it, in the same column as every device that completed the work. If a card cannot run a low-precision path — a missing FP4 or FP8 execution path, an unsupported kernel, a driver that refuses the cast — the reader sees the gap instead of seeing a suspiciously clean subset of results. This connects directly to why numerical precision is a first-class trade-off rather than a footnote: precision support is a capability, and a capability a device lacks belongs in its score.
Making the failure visible is only half of it. The other half is monotonicity: the zero stays in the calculation, and raising any input can never lower the aggregate. That property is what makes the scheme un-gameable. Because the aggregate is monotone, running a workload slowly always beats not completing it. There is no configuration in which a vendor or a tuner improves a published score by making a test fail, timing it out, or declining a precision. The incentive points one way — complete more work — and it points that way structurally rather than by policy.
Those two properties do different jobs and should not be collapsed into one. Visibility is an interpretation property: the reader can see what did not run. Monotonicity is an incentive property: nobody profits from arranging for something not to run.
Quick reference: what the number covers
| Question a reader asks | Inside the boundary | Outside the boundary |
|---|---|---|
| Which models were run? | The fixed model catalogue of one named release | Your fine-tuned model, your graph, your serving stack |
| What was timed? | Completed iterations inside one continuous declared window | Any settling, soak, or endurance period |
| Was warm-up counted? | No — the warm-up phase is discarded before timing starts | Compilation and autotune cost as a performance figure |
| At what workload size? | Scaled until throughput stops improving | An arbitrary or vendor-chosen batch size |
| Is the figure a median? | No — one timed window, reported as one | A distribution, percentile band, or outlier-handled mean |
| What if a precision is unsupported? | Scored zero, visible in the result | Silently skipped, invisible in the result |
| What if a workload times out? | Recorded zero, retained in the aggregate | Dropped so the remaining tests look clean |
| Can an undeclared condition be inferred? | No | Any conclusion drawn from the score alone about untested conditions |
The table is the boundary. Everything in the right column is a question a LynxBenchAI result does not answer, and pretending otherwise is the failure mode this method exists to prevent.
Reading a bounded result against your own workload
The practical question is what a bounded number is good for, given all of that.
It is good for comparison. Two devices measured under the same declared boundary can be ranked on that work, which is the whole point of treating hardware and software together as an AI Executor rather than a piece of silicon. It is good for falsification: if a vendor’s headline figure is several times a device’s measured sustained figure on catalogue work, the headline is describing a transient. It is good as a floor — a device that cannot complete a precision path in a controlled catalogue run will not acquire that capability in your production stack.
It is not good as a forecast. We see this misread regularly: a team takes a catalogue throughput number, divides by their token budget, and produces a capacity plan. The arithmetic is fine and the input is wrong, because the catalogue model is not their model. Turning a bounded result into a procurement or capacity decision is a separate discipline with its own evidence requirements, and the LynxBenchAI methodology treats it as one rather than folding it into the measurement.
On the applied side, the same discipline shows up whenever a client team reads a public benchmark — an HPC suite, a general-purpose CPU score, an MLPerf submission — and imports assumptions the measurement never made. That is the same over-read, one layer out, and it is why we spend engagement time on what a published number covers before anyone builds a plan on it: production AI reliability.
FAQ
What does a LynxBenchAI run deliberately leave outside its measurement boundary?
Four things: your own application, any post-settling behaviour, statistical distribution over repeated trials, and any condition that was not declared. A run exercises the fixed model catalogue of one named release inside one continuous timed window, with no soak or endurance phase. Anything beyond that is outside the boundary and cannot be recovered from the score.
Why does a fixed model catalogue mean the score speaks about that catalogue rather than about my application?
The catalogue models are chosen so two devices can be compared on identical work, not because they resemble any particular production graph. Your model, batching policy, and serving stack differ from the catalogue in ways the measurement never observed. The result is reliable as a comparison between executors and unreliable as a forecast for your workload.
What is the timed measurement window, and why discard warm-up first?
The window is a continuous period during which completed iterations are counted, at a workload size already scaled until throughput stops improving. Warm-up iterations are discarded first because they are dominated by autotuning, memory pool growth, and graph or engine compilation — counting them would measure compilation rather than computation. The window’s length is declared, and its integrity comes from being stated rather than from being long.
Why report a single timed window instead of a median across repeated trials?
Because a median is a different claim. It implies a characterised distribution, a stated trial count, and an outlier policy — none of which a single window provides. Reporting one window as one window keeps the claim honest; reproducibility comes from the reader being able to re-run it, not from the figure being pre-averaged.
Why is declaring the boundary the integrity move rather than extending the run until it feels exhaustive?
Every extension drifts the measurement toward a different question — thermal sustainability, endurance, long-run driver stability — without ever fully answering it, and it blurs the boundary instead of widening it. A stated boundary lets a reader see exactly where the result stops being valid. An undeclared one cannot be inferred from the score at all.
What happens to a precision a device cannot execute, or a workload that fails?
Both are recorded as zero rather than skipped or dropped. Recording the zero is what makes the failure visible in the result instead of vanishing from it. Monotonicity is what makes it un-gameable: the zero stays in the calculation and raising any input can never lower the aggregate, so running a workload slowly always beats not completing it.
Does a run measure a device only once it has thermally settled?
No. A run discards a warm-up phase and then times a continuous window; it does not wait for thermal equilibrium and includes no soak period. The declared window covers steady-state execution of catalogue work at saturation, not sustained-load behaviour over hours — that is a separate measurement with a separate boundary.
Does this number cover the thing you actually care about?
If you disagree with a LynxBenchAI result, the productive argument is almost never about the number. It is about the boundary: whether the declared window covers the behaviour you care about, whether the catalogue work resembles yours closely enough to be informative, whether a zero you find harsh reflects a capability gap you would actually hit in production.
Those are answerable arguments, because the boundary is written down. The version we cannot have is the one where a reader silently assumes a wider scope than was measured and then treats the mismatch as evidence of bad faith. Ask the sharper question instead: what would have to be inside the window for this number to decide your workload — and is that a measurement, or a deployment?
