Two LynxBenchAI numbers are comparable when they were produced under the same release name. That is the whole scope rule, and it is narrower than most readers expect the first time they meet it.
The rule is not a hedge. It is not a disclaimer bolted onto the results page to protect against complaints. It follows from what a release name actually denotes: a frozen definition of the measured thing — model catalogue, precisions, correctness thresholds, scoring formula. Hold that definition constant and the gap between two scores is a statement about the two devices. Change it and the gap is a statement about two different questions, which is not a statement about devices at all.
Most of the trouble we see comes from readers who treat a benchmark score the way they treat a physical measurement. A memory bandwidth figure in GB/s means the same thing this year as last year, because the unit is defined outside the benchmark. A composite AI benchmark score has no such external anchor. Its unit is defined by the release that produced it.
What a release name actually freezes
A release name is a scope declaration. Four things are pinned inside it — the model catalogue, the precisions, the correctness thresholds, and the scoring formula — and what each one determines, and what drifts when it is not fixed is set out in full on the release-identity page. This article assumes that decomposition and asks the next question: given two numbers, are they on the same axis?
Everything else about a run is a variable to be reported, not a constant to be assumed. The executor — the specific hardware plus the specific software stack that ran it — varies by definition; that variation is the thing being measured. Batch size varies too, and deliberately so, which is the one exception that trips people up and which we come back to below.
Release names are versioned, and the version carries meaning. Two release names sit either side of a major version bump, and the scoring methodology itself is inside the scope of what a major version may redefine. That is the substantive content of the scope rule. A major bump is permitted to change the catalogue, retire a workload that no longer represents anything, tighten a correctness threshold, or reweight the aggregate. Once any of those move, the composite number is measuring a different construct.
So the reason a cross-release comparison fails is worth stating precisely, because the usual instinct is wrong. The failure is not that one of the numbers is unreliable — it is that the measured thing changed underneath them. Both numbers can be perfectly reproducible, perfectly audited, perfectly honest, and still not belong on the same axis.
Why can’t I compare scores across benchmark releases?
Because subtraction requires a shared unit, and the unit moved.
Think about what it would take for a cross-release comparison to be valid. You would need every element of the definition to be either unchanged or changed in a way whose effect on the score is known and invertible. In practice, catalogue changes are not invertible: if a release retires a workload that a given device happened to be strong at, the effect on that device’s composite is specific to that device’s performance profile. There is no scalar you can apply to correct for it.
This is why we do not publish a conversion factor between release names, and why one should not be inferred from overlap. If two release names happen to share several workloads and a reader computes a ratio from a device that appears in both, that ratio describes that device under those two definitions. Applying it to a second device assumes the two devices respond identically to the definition change, which is the assumption the benchmark exists to test.
What can legitimately cross a release boundary is qualitative and structural, not numeric. A per-workload observation — that a given executor was memory-bandwidth-bound on long-context attention, say — often survives a release change, because it is a statement about mechanism rather than about the composite. A conclusion about the software stack (“the TensorRT path outperformed the eager PyTorch path on this device”) can survive if the workload it was drawn from is still present and defined the same way. The composite score itself never crosses.
The comparability checklist
Before reading a gap between two LynxBenchAI numbers as meaningful, work through this. It is deliberately short; each item is a hard gate, not a consideration.
| # | Check | If it fails |
|---|---|---|
| 1 | Do both results state a release name? | Not yet a comparison. Stop. |
| 2 | Is it the same release name — not merely the same major version family, but the same name? | Not comparable. No conversion exists. |
| 3 | Did both runs clear the release’s correctness thresholds on every workload in the composite? | A partial run has no composite. Compare per-workload only, if at all. |
| 4 | Is the executor fully stated for both — hardware and the software stack version that ran it? | The gap is unattributable; it may be a driver, not a device. |
| 5 | Did both runs reach saturation under the release’s saturation rule? | An unsaturated run understates its device. Gap is not readable. |
| 6 | Is the run a sustained measurement rather than a transient peak? | Peak-versus-sustained gaps are artefacts of run length, not device capability. |
| 7 | Are you reading the composite, or a per-workload number? | Fine either way — but do not mix. A composite gap and a per-workload gap answer different questions. |
Checks 1 and 2 are the release-scope gates. Checks 3 through 6 are run-quality gates: two numbers can sit inside the same release name and still not be readable against each other, because one of them was produced by a run that did not do what the release asked of it. A comparison whose release scope is unstated is not yet a comparison; a comparison whose runs were not saturated is a comparison of run conditions.
The batch-size exception, and why it is not a hole in the rule
The obvious objection: if comparability requires holding the setup constant, why is batch size allowed to differ between two runs in the same release?
Batch size is adapted per device to reach saturation. The catalogue and precisions are not. The distinction is not arbitrary, and it is the clearest illustration of what the scope rule is actually protecting.
The catalogue defines the question. Change which models run and you have asked a different question, so the answers are no longer commensurable. Batch size is part of how the device is driven to answer it. A device with more HBM and more SM occupancy headroom needs a larger batch before its pipelines are full; pinning every device to one batch size would not produce a fair comparison, it would produce a comparison of who happens to saturate at that particular batch. Fixing batch size fixes the wrong variable — it holds constant something the reader does not care about while letting the thing they do care about (peak sustained capability) go unmeasured on half the fleet.
So the rule generalises: hold constant what defines the question; adapt what is needed to let each executor answer it at its own best. That is the same principle as bounded optimization applied to the input side. Both runs are being given the conditions under which they perform best, within limits the release declares. This connects directly to how the AI Executor treats hardware and software as a single measurable unit — the executor is what varies, and the release is what does not.
Anyone reproducing a published number should expect their batch size to differ from the published one and should not treat that difference as a discrepancy. What they should check is whether their run reached saturation under the same rule, which is a different and more useful question than whether the numbers match at batch 32.
What a reader may and may not do with two numbers
It helps to state the permitted operations explicitly, because the boundary is sharper than the intuition.
Permitted, under one release name:
- Read the gap between two composites as a difference between the two executors, subject to the checklist above.
- Read a per-workload gap as a difference in how the two executors handle that workload class.
- Reproduce a published number on your own hardware and compare against the published executor of the same device.
- Draw a mechanism-level conclusion — bandwidth-bound, kernel-launch-bound, precision-limited — from the shape of the per-workload profile.
Not permitted, in any circumstance:
- Carry a number out of its release name into any other comparison, including into a spreadsheet that also contains numbers from another release.
- Infer a conversion factor from devices that appear under both release names.
- Conclude that the newer release name produces “better” numbers, or that a score moving up between releases means the device got faster. It may mean the catalogue got easier, or the aggregation changed weight.
- Treat a within-release gap as settling whether a device is suitable for a specific application. A benchmark composite is not an application; that inference is a separate step with separate evidence requirements.
That last one deserves a sentence of its own. Reading a gap correctly and acting on it correctly are different disciplines. The composite tells you that executor A sustained more of the release’s workload mix than executor B under equal definitional conditions. Whether that difference survives contact with your model, your context lengths, your latency budget, and your utilisation pattern is a question the composite does not answer. Turning benchmark evidence into a purchase belongs to the decision-grade evidence discipline that governs how published results are used in procurement — not to this scope rule.
Worked example: two numbers, one legitimate reading
Suppose a reader has two figures in front of them. Device A scored 100 under release name R; device B scored 118 under release name R. Both runs report full executor detail, both cleared correctness on every workload, both reached saturation, both were sustained runs of the release’s declared duration. (Numbers illustrative — the shape of the reasoning is what matters, not the values.)
The legitimate reading: under release R’s definition of the workload mix, executor B sustained roughly 18% more of it than executor A. That is a claim about two executors — hardware plus stack — measured against one fixed question.
Now change one thing. Suppose device B’s figure came from release name R+1. The 18% gap is now uninterpretable. Not smaller, not less certain — uninterpretable. It could be entirely definitional. It could be understating a real gap. There is no residual signal in it to salvage, because the two numbers were never answering the same question. The correct action is to find or produce a device B result under R, or to make no comparison.
And one more variant. Both under R, both fully reported, but device A’s run shows a batch size well below the point at which its throughput curve flattens. That run did not reach saturation, so its 100 is not device A’s number under release R — it is a lower bound. The gap is not readable until the run is redone. This is exactly the failure mode that the sustained practical peak framing exists to name: a run that stopped short is not a smaller measurement of the same thing, it is a measurement of something else.
How does an auditor check a comparison someone else made?
The audit question is narrower than the reading question, and it is mostly about what is stated rather than what is true. An auditor is checking whether the comparison declared enough of itself to be checkable.
Three failure signatures show up repeatedly in comparisons we are asked to review. The first is an unstated release name — a table of scores with device names and numbers and nothing else. That table cannot be repaired by asking which is fastest; it has to be repaired by recovering the provenance of each row, and often the provenance is gone. The second is a mixed table, where most rows share a release name and one or two do not, usually because a device was only ever measured under an older name and someone wanted the table to be complete. Completeness is not worth the corruption. The third is an executor stated only as hardware — “RTX 5090” with no stack version — which leaves any gap ambiguous between the device and the CUDA, driver, and runtime versions that drove it.
The remedy in all three cases is the same and it is unglamorous: drop the rows that cannot be scoped, and say why the table is shorter than the reader expected. A short honest table beats a complete unreadable one. The reproducibility discipline behind this — what must be recorded for a result to be re-runnable at all — is covered separately in how LynxBenchAI results are made reproducible and auditable.
FAQ
Why is comparability scoped to a single release name rather than to the benchmark as a whole?
Because the release name is what freezes the definition of the measured thing. Model catalogue, precisions, correctness thresholds, and scoring formula are constant inside one release name and may change between them. The benchmark as a whole is a family of definitions, not one definition, so it cannot supply a shared axis.
What exactly changes between release names, and why does that break comparison across them?
Two release names sit either side of a major version bump, and the scoring methodology itself is inside the scope of what a major version may redefine. Workloads can be retired or added, thresholds tightened, and the aggregation reweighted. Once any of those move, the composite is measuring a different construct — so the failure is that the measured thing changed, not that either number is unreliable.
How can a reader check that two LynxBenchAI numbers sit on the same axis before drawing any conclusion from the gap?
Run the seven-item checklist in this article. The first two items are hard scope gates: both results must state a release name, and it must be the same name. The remaining five are run-quality gates covering correctness, executor detail, saturation, sustained duration, and not mixing composite with per-workload figures.
Is there a conversion factor that adjusts a score from one release name onto another, and what can legitimately be carried across release names at all?
There is no conversion factor and none can be inferred from devices appearing under both names, because catalogue changes affect each device according to its own performance profile. What can cross a release boundary is qualitative: mechanism-level observations such as a workload being memory-bandwidth-bound on a given executor. The composite score never crosses.
Why does adapting batch size per device preserve comparability while changing the model catalogue would destroy it?
The catalogue defines the question; batch size is part of how each device is driven to answer it. Pinning one batch size across all devices would measure who saturates at that batch rather than each device’s sustained capability. Batch size is adapted per device to reach saturation; the catalogue and precisions are not.
What is the right reading of a result whose release name is missing or unstated?
There isn’t one. A comparison whose release scope is unstated is not yet a comparison. The correct handling is to recover the provenance of each figure or to drop the unscoped rows entirely, rather than to compare and caveat.
When two numbers do sit inside the same release name, what still has to be checked about the run itself before the gap between them can be read as meaningful?
That both runs cleared the release’s correctness thresholds on every workload, that both reported the full executor including software stack version, that both reached saturation, and that both are sustained rather than transient measurements. A run that stopped short of saturation is a lower bound, not a smaller measurement of the same thing.
Before you subtract, check that both numbers answered the same question
Every LynxBenchAI figure travels with a release name attached, and the attachment is not decorative. Strip it and the figure becomes a decimal with no unit — still precise, still reproducible, and no longer capable of supporting a comparison. The discipline this asks of a reader is small: before you subtract, check that both numbers answered the same question, and check that both runs were given the chance to answer it fully.
The harder question sits one step past the scope rule. Two devices measured cleanly under one release name will still diverge differently under your own workload than they did under the catalogue, because the catalogue is a proxy and yours is the real thing. Knowing that the gap is real is where this article ends; knowing what the gap will be worth to you is where the selection of the workload you should actually be measuring begins.
