Four different jobs sit behind every performance number you are handed: someone defined the methodology, someone executed it, someone hosted the result, and someone read it. When those four collapse into one organisation, the number does not become false — it becomes harder to read. That is the whole of the governance problem, and most of the argument about vendor benchmarks is really an argument about this collapse without naming it.

The naive version of the question is “can I trust this figure?” It is the wrong question because trust is not a property of a number. What you can actually establish is the shape of the evidence: which role was occupied by whom, and which of those roles you are now occupying yourself. Once that is written down, a disagreement about performance stops being a matter of opinion and becomes something you can re-run.

The four roles, named separately

Benchmark governance sounds like a compliance word. In practice it is bookkeeping about authorship.

• Methodology definition — who decided what counts as the workload, what the warm-up is, which precision is used, when the run is considered saturated, and what may be optimised.
• Execution — who actually ran it, on which hardware, with which driver, container, and framework build.
• Hosting — who publishes the result, controls the presentation, and decides what appears alongside it.
• Interpretation — who reads the result and converts it into a claim about a system they care about.

These are separable in the strict sense: each can be occupied by a different party without the others changing. A standards body can define, a vendor can execute, a trade publication can host, and an engineer in a procurement meeting can interpret. That configuration is unusual. Far more common is a single organisation defining, executing, and hosting — with only interpretation left to the reader.

Nothing about that configuration implies dishonesty. A vendor typically has the best access to the hardware, the deepest knowledge of its software stack, and a legitimate interest in showing what the part can do. The problem is narrower and more mechanical: when definition, execution, and hosting are held by one party, the reader has no independent handle on any of the three, so every question about the result routes back to the same source. That is a structural property of the evidence, not a moral judgement about the publisher.

Why “who ran this” is a governance question

A run is not a fact about a chip. It is a fact about an executor — the hardware and the software stack together, at a specific version, under a specific load pattern. Change the CUDA version, swap TensorRT for ONNX Runtime, alter the batch shape, and the number moves without a single transistor changing. We see this regularly: two teams measuring “the same GPU” disagree by a wide margin and both are correct about what they measured.

That is why execution authorship carries governance weight. The person who ran it chose:

• the software stack (driver, CUDA, cuDNN, framework build, inference runtime)
• the workload shape and whether it reaches sustained load rather than a transient burst
• the precision, and whether accuracy was checked at that precision
• how far optimisation was permitted before the comparison stopped being like-for-like

None of those choices are visible in a headline throughput figure. They are visible only if reported. So “who ran this” is not trivia — it tells you whose choices are baked into the number and, therefore, which questions you still need answered. Our framing of the executor as the real unit of performance is the reason this matters at all: if the unit were the chip, authorship would be a curiosity.

What changes when the reader can run it themselves

Here is the structural shift that makes this more than a philosophy discussion. LynxBenchAI’s Personal Edition is installable, and anyone can run it on their own machine, free and non-commercially. That moves the execution role to the hardware owner.

Consequences, in order of how much they change your day:

1. A supplied figure becomes one data point among many. It does not stop being useful. It stops being the only measurement in the room. Once you hold a comparable self-produced number for your own executor, the published one is context rather than verdict.
2. Disagreement becomes testable rather than arguable. “That number looks high” is an opinion. “I ran the same defined workload at the same precision on my hardware and got materially less, and here is my stack” is a test result. The conversation changes register.
3. The methodology-definition role stays where it was. Self-execution does not let you invent your own definition and still call the comparison like-for-like. If you change the workload, you have produced a different measurement, not a rebuttal.
4. You now occupy two roles at once — executor and interpreter — which means the discipline you would have demanded of a vendor now applies to you. Report your stack. State your load pattern. Say whether you reached sustained load.

That last point is the one people skip. Self-measurement is not automatically more accurate than a lab result. A careful lab run with a documented stack and a saturation criterion beats a casual desktop run with background processes and a thirty-second window, every time. What self-measurement gives you is proximity — the measured executor is the one you will actually deploy — and symmetry — you can be questioned the same way you question others. Whether the window you used ever reached sustained practical peak rather than a transient burst is the first thing a careful reader of your own number will ask.

How do you read a performance figure whose author you do not control?

This is the working rubric. It is deliberately about role occupancy, not about vendor reputation.

Question	What a good answer looks like	If unanswerable
Who defined the workload?	A named, published methodology with declared warm-up, saturation criterion, and optimisation bounds	Treat the figure as a demonstration, not a comparison
Who executed the run?	Named party plus full stack: driver, runtime, framework build, container	You cannot reproduce it; do not compare it to your own numbers
Was the load sustained?	An explicit sustained-load window and a stated saturation point	Assume transient peak; discount accordingly
At what precision, and was accuracy checked?	Precision named and an accuracy delta reported at that precision	The number and the model quality are unlinked
How much optimisation was allowed?	A stated bound applied equally to every entry	Comparison is not like-for-like
Who hosts the result, and who could contradict it?	A surface that reports within one named release and shows the stack	The result answers only to itself
Can I re-run it?	A published, installable procedure	Disagreement will stay rhetorical

The rubric is extractable on its own. Four or more unanswerable rows and you are not holding a comparable measurement — you are holding a marketing artifact that may well be true.

Role separation is not an independence claim

There is a tempting shortcut here, and it is worth refusing explicitly. Whoever publishes a benchmark can always assert independence. Assertions of independence are cheap, unfalsifiable, and — importantly — they are made by the same party whose authorship is in question. Role separation is a different kind of statement: it describes an observable configuration. Either the methodology is published separately from the runs or it is not. Either the reader can execute it or they cannot.

We build LynxBenchAI, so we are not a neutral party to its methodology, and pretending otherwise would be its own governance failure. The claim we make is narrow: because the benchmark can be run by the hardware owner, the execution role is genuinely transferable, and the results surface reports within one named release rather than adjudicating between them. A leaderboard is a reporting surface. It does not settle disputes; it gives disputes somewhere concrete to happen. The distinction between bounded optimisation as a fairness condition and an editorial verdict is exactly this distinction, one level down.

Nor does role separation excuse you from reading measurement boundaries. A perfectly separated set of roles can still produce a figure that does not transfer to your workload — because the workload was different, because the scale was different, or because the precision trade-off was acceptable to the publisher and not to you. Knowing who ran it tells you whose assumptions you inherited. It does not tell you whether those assumptions match your deployment. That question is handled by reproducibility and audit discipline.

What a testable disagreement looks like

A worked example, with the assumptions stated because the point is the form, not the figures.

Suppose a published result reports throughput for a transformer inference workload on a given accelerator, and your own run on nominally identical hardware lands materially lower. Illustratively: if the published figure were 1.0× and your measurement came in around 0.7×, the useful response is not to dispute the 1.0×. It is to close the gap between the two runs, one variable at a time:

• Report your executor fully. Driver and CUDA version, inference runtime (TensorRT versus a plain PyTorch eager path is often the entire gap), container image, batch and sequence shape.
• Check the load window. A short run may never leave warm-up; a long one may thermally throttle. State the window and where the throughput curve flattened.
• Check topology. PCIe versus NVLink placement, NUMA affinity, and host-to-device transfer overhead account for a surprising share of “my numbers are lower” cases in our experience.
• Check precision parity. If the published run used a lower precision than yours, you are comparing two different trade-offs, and the accuracy figure has to travel with the throughput figure.
• Re-run under the same declared bounds. If the published run used kernel fusion or graph capture and yours did not, the difference is a stack difference, not a hardware disagreement.

If after all that the gap persists and both stacks are documented, you have produced something genuinely useful: two comparable measurements that disagree, with the disagreement localised. That is a result, not a complaint. Publishing yours alongside the original is how the evidence base actually improves, and it is the same instinct that drives decision-grade evidence for procurement — although turning any of this into a purchase decision is a separate discipline from establishing whether the number is readable in the first place.

Teams that engineer this separation inside their own organisation end up building something close to it deliberately: a named author for each measurement and a named signer for the conclusion drawn from it. That pattern shows up in AI governance and trust engineering work, where the question “who is accountable for this gate” is answered on paper before anything ships. On the benchmark side, the separation is written into the LynxBenchAI methodology rather than left as an editorial promise.

FAQ

Which distinct roles exist around a benchmark, and why does naming them separately matter?

Four: defining the methodology, executing it, hosting the results, and interpreting them. Each can be occupied by a different party without changing the others. Naming them matters because when definition, execution, and hosting sit with one organisation, every question about the number routes back to a single source — which makes the result harder to read, independent of whether it is accurate.

What changes about a performance figure when the person who owns the hardware can produce it themselves?

The execution role moves to the hardware owner. A supplied figure becomes one data point among many rather than the only measurement available, and a disagreement about it becomes something you can re-run instead of something you argue about. The methodology-definition role does not move: change the workload and you have produced a different measurement, not a rebuttal.

Why is “who ran this” a governance question rather than trivia about a result?

Because the unit of performance is the executor — hardware plus software stack — not the chip. Whoever ran the benchmark chose the driver, runtime, framework build, load pattern, precision, and optimisation bound, and none of those choices are visible in a headline number. Knowing the executor tells you whose assumptions are baked into the figure and which questions remain open.

How does role separation differ from an independence claim made by the publisher?

An independence claim is an assertion made by the party whose authorship is in question, and it cannot be falsified from outside. Role separation describes an observable configuration: either the methodology is published apart from the runs, or it is not; either the reader can execute it, or they cannot. We build LynxBenchAI and are therefore not neutral about its methodology — the transferable claim is that the execution role can be occupied by the hardware owner.

Why is a statement about the shape of the available evidence different from an accusation about a vendor’s honesty?

Because it is a claim about structure, not conduct. A vendor usually has the best hardware access and the deepest stack knowledge, and its numbers are frequently correct. The point is that when one party defines, executes, and hosts, the reader has no independent handle on any of the three — a property of the configuration, not evidence of bad faith.

Which role does a reader occupy when interpreting someone else’s result, and what responsibility comes with it?

The interpretation role, and often the execution role too once they run their own comparison. Occupying both means the discipline you would demand of a publisher now applies to you: report the full stack, state the load window and whether sustained load was reached, name the precision, and say what optimisation you allowed. Self-measurement is closer to your deployment but is not automatically more accurate than a documented lab run.

When a result is disputed, what does a testable disagreement look like in practice?

You re-run the same defined workload on your own executor and close the gap one variable at a time: runtime and driver versions, batch and sequence shape, load window and saturation point, PCIe or NVLink topology and NUMA affinity, and precision parity with the accuracy delta reported alongside throughput. If the gap survives with both stacks documented, you have two comparable measurements that disagree with the disagreement localised — which is publishable evidence rather than a complaint.

Four roles, and who filled each one for the number in front of you

The uncomfortable part of role separation is that it does not end with a verdict on someone else’s benchmark. It ends with a question about your own workload: which executor do you actually need to characterise, on which sustained load, at which precision — and have you run it yet? A published number you cannot place a measurement of your own beside is still only a reading aid. Once you can — and result provenance governs what that second number has to carry with it — the framework above becomes an argument you can win or lose on evidence, which is the only kind worth having.