A GT of 4,120 does not mean a device is 12% faster than one scoring 3,680. It does not mean the device reached 41.2% of anything. GT is an ordinal aggregate — a rank-bearing summary formed from three category scores, which are themselves aggregates — and the arithmetic you are allowed to do on it is narrower than the arithmetic a spreadsheet will happily let you do.
That gap between what a number permits and what a reader does with it is where most benchmark misreadings live. Not in the measurement. In the interpretation of a figure that was already correct.
A LynxBenchAI run emits four scores: Training, Inference, Compute, and GT. This article is about what each of those four means, how the aggregation is built, and — the part that matters most in practice — what a GT number does and does not license you to say out loud in a review meeting.
Four scores, not one score and three ingredients
The most common structural misreading is treating Training, Inference, and Compute as intermediate quantities that exist only to be folded into GT. They are not. Each of the three category scores is independently meaningful and readable on its own.
That distinction has teeth. A device can be strong at one class of work and mediocre at another, and the aggregate will smooth exactly that shape away. Consider two executors that land within a percent of each other on GT: one earned it on Training with a weak Inference figure, the other on Inference with training throughput that would make a fine-tuning loop painful. The GT numbers agree. The devices do not agree about anything a buyer cares about.
So the correct reading order is inverted from the intuitive one. Read the three category scores first. Read GT last, and read it as a rank hint — a way of ordering a long list before you look at it properly, not a verdict that replaces looking.
This is the same principle that makes the AI Executor the real unit of performance rather than the chip: collapsing a multidimensional object into one figure is convenient, and the convenience is paid for in what you can no longer see.
What “ordinal” rules out
GT is an ordinal aggregate, not a physical quantity. It has no unit. It is unbounded, and it carries no reference-device normalisation — there is no “score of 100 machine” against which everything else is a fraction. Any reader who assumes a 0–100 scale has imported an assumption the score never made.
The practical consequence is a short list of operations that are meaningful and a short list that quietly invent precision.
Which arithmetic on a GT score is meaningful?
| Operation | Meaningful? | Why |
|---|---|---|
A > B — ordering two devices within one release |
Yes | Ordering is exactly what an ordinal aggregate is for |
| Sorting a whole list of devices by GT | Yes | Same property, applied at scale |
A - B — treating the gap as a size |
No | The scale is not interval; a 400-point gap at the top of the range is not the same object as a 400-point gap at the bottom |
A / B — reading the ratio as a speed multiplier |
No | A ratio of two ordinal aggregates is not a multiplier of any physical rate |
A / B as “1.12× the performance” in a slide |
No | This is the previous row with a unit bolted on, which is worse |
| Comparing GT across different release names | No | Weights and test composition are fixed per release; cross-release GT comparison compares two different functions |
| Using GT to shortlist, then category scores to decide | Yes | This is the intended workflow |
The row that gets violated most often is the ratio. It is seductive because the division executes without error and the output looks like a percentage improvement. If you need a multiplier — “this configuration serves 1.4× the tokens per second” — take it from the underlying throughput measurement in the category you care about, not from GT. The category scores sit closer to the physics; GT sits furthest from it.
How the aggregate is actually formed
Both aggregation levels — the three category scores, and GT itself — use the same construction: a shifted geometric mean.
The mechanism, stated plainly: every input is offset by a constant equal to a tenth of the mean of all the inputs. The offset values are combined geometrically. Then the offset is removed. The constant is derived from every value in the set, which means it is not a magic number tuned per device; it falls out of the data being aggregated.
That shift solves a specific, well-known failure. A plain geometric mean collapses to zero the moment any single input is zero — one unrun or unsupported workload and the entire aggregate is annihilated, which makes the score fragile rather than informative. With the shift in place, a zero participates on the same footing as any other number. It pulls the aggregate down, as it should. It does not delete it.
If a geometric mean dies on a zero, why use one at all?
Because the alternative behaves worse in the direction that matters. An arithmetic mean lets a single enormous throughput figure dominate the summary, so an executor that is spectacular at one test and unremarkable at nine can look broadly excellent. The geometric mean resists that: it rewards consistency across the test set and penalises a hollow profile. The shift is what makes the geometric mean survivable in a real test matrix where some cases legitimately return nothing.
The second structural property is monotonicity. Raising any single input can never lower the aggregate. That sounds obvious; it is not automatic in weighted composite scores, and non-monotone aggregates are genuinely pathological — they let a device improve at a real workload and lose score for it. This property has been checked across roughly 200,000 randomised trials with no violations observed (benchmark-class: property test over the released aggregation implementation, not a performance measurement).
Monotonicity is also the answer to the gaming question, and it is worth being blunt about it.
Can a device score higher by failing a test?
No. Monotonicity forbids it directly. Skipping or failing a workload contributes a lower input, and a lower input cannot raise the aggregate. There is no threshold effect to exploit, no floor that a failure lands above, no configuration where opting out of a hard test is scored as neutral. If a device does not run something, its score reflects that it did not run something.
The second half of the gaming question is weights, and the answer there is procedural rather than mathematical: every weight in the aggregate is fixed by the release and applies identically to every device and every vendor. Nobody tunes a weight in their own favour because nobody tunes a weight. This is the aggregation-side counterpart to why fair benchmarking requires bounded optimization — the optimization envelope constrains what a submitter may do to the run, and fixed release weights constrain what anyone may do to the summary. Both constraints are published in the LynxBenchAI methodology rather than left to the reader to infer from results.
The weights, and why they are not equal
GT combines the three category scores with deliberate, published weights:
| Category | Weight | Rationale |
|---|---|---|
| Training | 0.4 | A full model workload, end to end |
| Inference | 0.4 | A full model workload, end to end |
| Compute | 0.2 | Raw matrix operations rather than a model workload |
Compute carries less weight because it measures something further from what anyone deploys. Raw matrix throughput is a real and useful signal — it is often the first place a software-stack regression shows up — but it is not a model. Giving it equal footing with Training and Inference would let a device with excellent GEMM numbers and awkward end-to-end behaviour present better than it deploys.
Inside Compute there is a second scaling that surprises people on first reading. As each test is scored, the dense matrix test is scaled by 0.1 and the sparse test by 10.0.
That is not a judgement that sparse work is ten times more valuable, or that dense work is a tenth as interesting. Dense and sparse operations sit orders of magnitude apart in natural throughput, because of how each one reaches memory — dense GEMM streams predictably and saturates arithmetic units; sparse operations chase indices and spend their life waiting on memory. Left unscaled, the dense figure would swamp the sparse figure inside the category and the sparse test would effectively stop contributing. The scalars bring two quantities of very different magnitude onto comparable footing so that both actually participate.
We treat this as the ordinary hazard of any composite: unequal natural magnitudes are a numerical problem before they are a philosophical one, and pretending otherwise produces a score that silently ignores half its own inputs.
Memory-weighted throughput: why a low-precision speed-up has to earn it
One more property changes how the aggregate reads, and it is the one most relevant to anyone evaluating quantized deployments.
Aggregation weights throughput by the memory a run actually moves. A run that halves its precision typically halves what it moves per element, and it therefore does not get to bank the full nominal speed-up as pure improvement — a lower-precision run has to earn its speed-up rather than collect it by definition.
This is deliberate and it is the semantic hinge of the whole score. Without it, “go to a smaller dtype” would be the dominant strategy for improving a benchmark number, regardless of whether the resulting model was still usable. With it, a precision reduction shows up as what it actually is: a trade, priced in the aggregate. The accuracy half of that trade is a separate question, treated in numerical precision as a first-class benchmark trade-off, and the two halves have to be read together. A score that improved on a precision change tells you the throughput side of the trade went well. It says nothing whatsoever about whether the outputs are still acceptable.
A worked reading of four scores
Assume a published result set with these figures. The numbers are illustrative — chosen to show the reading discipline, not taken from any specific device.
| Score | Device A | Device B |
|---|---|---|
| Training | 3,900 | 2,600 |
| Inference | 2,700 | 4,150 |
| Compute | 5,100 | 4,400 |
| GT | 3,600 | 3,570 |
What you may say:
- The two devices sit essentially adjacent in GT order within this release.
- Device A is the stronger training executor in this test set; Device B is the stronger inference executor.
- Device A’s Compute lead is the least decision-relevant of the three gaps, given Compute’s 0.2 weight and its distance from a real model workload.
What you may not say:
- “Device A is 0.8% faster.” GT differences are not percentages of anything.
- “Device A is 1.5× better at training.” That ratio is over category scores, which are also ordinal aggregates; take multipliers from throughput, not from scores.
- “They are equivalent.” Their GT figures are close. Their profiles are opposite, and a fine-tuning team and a serving team should reach different conclusions from this same table.
The last line is the whole point. Two near-identical GT values concealing opposite shapes is not an aggregation defect — it is aggregation working correctly, doing the only thing an aggregate can do. The defect appears when a reader stops at the aggregate.
FAQ
What is GT an aggregate of, and in what sense is it ordinal rather than a physical quantity?
GT aggregates the three category scores — Training, Inference, and Compute — each of which is itself an aggregate over its test cases. It is ordinal: it supports ordering devices within one release and nothing stronger. It has no unit, it is unbounded, and it carries no reference-device normalisation, so there is no implied 0–100 scale and no “fraction of an ideal machine” reading available.
Why can Training, Inference, and Compute each be read on their own, and what does collapsing them hide?
Each category score is a complete, independently meaningful summary of one class of work, not a scratch value on the way to GT. Collapsing them hides profile shape: a device strong at training and weak at inference can land at the same GT as its mirror image. Two matching GT figures can therefore describe devices that a fine-tuning team and a serving team should rank in opposite orders.
Which arithmetic on an ordinal aggregate is meaningful, and which invents precision?
Comparison and sorting within a single release are meaningful. Differences and ratios are not — a 400-point gap is not a size, and dividing one GT by another does not produce a speed multiplier, however plausible the decimal looks. If you need a multiplier, take it from the throughput measurement inside the relevant category, and never compare GT across different release names, since the weights and test composition are fixed per release.
How does the shift constant let a zero score participate without wiping out the aggregate?
A plain geometric mean collapses to zero if any single input is zero, which would let one unsupported workload annihilate an entire result. LynxBenchAI offsets every input by a constant equal to a tenth of the mean of all the inputs, combines the offset values geometrically, then removes the offset. Because the constant is derived from every value in the set, a zero participates on the same footing as any other number: it drags the aggregate down without deleting it.
Can a device reach a higher score by failing a workload or by tuning the weights?
No, on both counts. The aggregate is monotone by construction — raising any single input can never lower the result, and a failed or skipped workload contributes a lower input, so it cannot raise the score. Monotonicity has been checked across roughly 200,000 randomised trials with no violations. Weights are fixed by the release and applied identically to every device and vendor, so there is no per-submission weight to tune.
Why does Compute carry less weight, and why are the dense and sparse tests scaled so differently?
Compute is weighted 0.2 against 0.4 each for Training and Inference because it measures raw matrix operations rather than a model workload, and raw GEMM throughput is further from what anyone actually deploys. Inside Compute, the dense test is scaled by 0.1 and the sparse test by 10.0 because dense and sparse operations sit orders of magnitude apart in natural throughput through how each reaches memory. The scalars equalise magnitude so both tests contribute; they are not a statement that one kind of work is better.
How does weighting throughput by memory moved change what a low-precision speed-up gets credited with?
Aggregation weights throughput by the memory a run actually moves, so halving precision roughly halves what is moved per element and the nominal speed-up is not banked as pure improvement. A lower-precision run has to earn its speed-up rather than receive it by definition. That keeps “use a smaller dtype” from becoming the dominant score-improvement strategy, and it leaves the accuracy side of the trade as a separate question the score does not answer.
What GT will and will not settle
If a discussion about two devices stays on GT for more than a minute, the discussion has gone wrong. GT is a sorting key. It orders a directory so you can find the four or five executors worth examining, and then its job is finished.
The number worth arguing about is the category score for the class of work you actually run — and behind that, the sustained figure under load rather than a transient one. Before either reading is safe, both numbers have to sit under one release name, which is the scope rule set out in comparability within a release; and what the aggregate deliberately leaves outside its scope is listed in declared measurement boundaries. Turning any of it into a purchase decision is a further step again, with its own evidence requirements.
So the question to carry into the next result you read: which of the three category scores corresponds to the workload you will actually put on this hardware — and can you defend the answer without once dividing one GT by another?
