A driver update changes kernel scheduling. Throughput drops 12%. Nobody touched the model.
This is the kind of event that makes sense only after you abandon one mental model and replace it with another. Under the old model — “hardware has performance, software just uses it” — a driver update shouldn’t change throughput by double digits. Under the replacement model, it’s not only possible but predictable, because performance was never a property of the hardware alone.
AI performance is an emergent property of the hardware, the software stack that drives it, and the workload that shapes what both of them do. You don’t get a performance outcome by summing up layer contributions. You get it from interactions — and those interactions can be surprisingly sensitive to changes in any single layer. The stack, not the device, is the correct unit of performance reasoning for AI systems.
Emergence, not aggregation
When we say performance “emerges” from the hardware × software stack, we mean something specific: the final throughput, latency, or efficiency number is not decomposable into independent contributions from hardware and software. You can’t say “the GPU contributed 70% of the performance and PyTorch contributed 30%.” That’s not how the system works.
Consider what happens during a single forward pass of a transformer model. The framework decides how to lower the computation graph — which operators to fuse, which memory layout to use, which kernels to dispatch. The CUDA runtime schedules those kernels onto the device. cuDNN or a custom attention kernel (say, FlashAttention) determines the actual execution path on the hardware. The memory subsystem serves data according to access patterns that the software stack created. Thread scheduling, synchronization points, and stream management all shape how the hardware’s resources are actually consumed.
Change any one of those decisions — swap a kernel, alter a graph transformation pass, modify the runtime’s allocation strategy — and you’re not just “tuning.” You’re changing what the hardware actually does, which changes where the bottleneck lands, which changes the measured outcome. The coupling between layers is where the performance story lives, not in any single layer’s properties.
Why does hardware-only reasoning keep producing surprises?
Hardware matters. Nobody serious argues otherwise, and framing the argument as “hardware vs. software” misses the point entirely. The problem is more specific: treating hardware as the explanatory unit for performance outcomes.
That looks like “we upgraded from GPU A to GPU B — we should see a proportional speedup” and then being puzzled when the gain is smaller than expected, varies by model, or disappears under sustained load. It looks like comparing two systems by their theoretical FLOPS ratio and finding the actual throughput ratio is nothing like it. It looks like a team purchasing hardware based on a single spec-sheet advantage and then discovering that the workload spends most of its time somewhere the spec sheet doesn’t describe.
We’ve found that when a team is surprised by an AI performance result, the explanation almost always involves a software or system-level interaction that the hardware-only model can’t account for (observed across our engagements; not a benchmarked failure rate). The hardware didn’t fail to perform — the hardware was never the whole story.
The stack is the performance definition
A pragmatic shift is to stop asking “what GPU is this?” and start asking “what execution stack is this?” — because the stack determines what actually runs.
The drivers and runtime decide how work is scheduled, synchronized, and allocated. The framework decides which operators execute and how graphs are partitioned. It’s a reminder that GPUs are part of a larger system, not isolated performance islands. Libraries and kernels decide what instructions actually hit the device. The system topology — PCIe layout, NUMA configuration, NVLink connectivity — decides how data moves between components. And the workload itself decides what gets stressed, for how long, and in what pattern.
Execution stack layers and their performance roles
| Stack layer | What it controls | How it affects measured performance |
|---|---|---|
| Hardware | Compute units, memory, interconnect, power/thermal envelope | Defines the theoretical ceiling for arithmetic, bandwidth, and sustained operation |
| Drivers & runtime | Kernel scheduling, memory allocation, synchronization | Determines how efficiently hardware resources are claimed and released |
| Frameworks | Graph construction, operator fusion, kernel dispatch | Decides which execution path the workload takes through the hardware |
| Libraries & kernels | Actual device instructions (cuDNN, FlashAttention, NCCL) | Sets the practical throughput ceiling for individual operations |
| Workload | Model shape, batch size, sequence length, precision mode | Determines which hardware subsystem is stressed and where the bottleneck lands |
None of that is optional detail. Those are the mechanisms that produce the number. When someone presents a performance result without stack context, the number isn’t wrong — the claim is just incomplete, in the same way that a benchmark result with hidden methodology is a datapoint without interpretation. As we explored in thinking about why identical GPUs can produce different results, the execution context is frequently the dominant source of variance, not the hardware identity.
Naming the unit: the AI Executor
“The stack matters” is easy to say and hard to hold anyone to. It becomes checkable only when the unit of measurement is named precisely enough that two people can agree whether they measured the same thing.
The unit we use is the AI Executor: the device together with the backend it ran through — cuda for NVIDIA, cuda via ROCm for AMD, xpu for Intel, or CPU — plus the driver, framework, and runtime present on that machine. A LynxBenchAI result is bound to that whole tuple. It is not attributed to silicon alone, which is precisely what turns the stack argument from rhetoric into something a reader can audit or contradict.
The consequence is a rule about what you may not say. No result belongs to a device. “This GPU delivers 500 tokens per second” is not a hardware statement; it is a statement about one executor, and stripping the backend, driver, framework, and runtime out of it does not make it more general — it makes it unfalsifiable. Comparability is also bounded by release name: two numbers from different release names are not answering the same executor question, so like-for-like reading holds within a release name only (26Q3 to 26Q3). And the run records what it measures. Thermal state and settled-clock conditions are not among them, so neither is claimed.
What a number needs alongside it
- The device, named exactly, and the silicon type it belongs to.
- The backend the work went through (
cuda,cudavia ROCm,xpu, or CPU). - Driver, framework, and runtime versions present on the machine at run time.
- The workload configuration: model, precision mode, batch shape, sequence length.
- The release name, because comparability does not travel across release names.
Strip any one of those and the number stops being reusable by someone else later. It may still be true; it just stops being a claim anyone can check.
Keeping optimization effort out of the comparison
There is a second hidden variable that quietly wrecks cross-vendor performance discussions, and it is not a layer of the stack at all. It is human effort. If one vendor’s hardware got three engineer-weeks of kernel tuning and another got an afternoon, the resulting numbers describe the tuning budget as much as the executor.
The discipline we hold is to bound that effort and make it identical. Models are prepared once, before any target device is known, and the same prepared artefacts ship to every vendor’s hardware. Where two vendors’ kernels disagree about what is permissible, the stricter constraint is applied to all of them rather than tailored per vendor. That deliberately leaves performance on the table for whichever vendor could have gone further — and accepting that cost is the point, because the alternative is a comparison whose leading variable is who we chose to optimise for.
This is also where the stack model gets its practical payoff. Teams that adopt it find that performance discussions get calmer and more accurate, because performance ownership spans hardware and software teams. Discrepancies stop looking like mysteries and start looking like the natural consequence of running different executors. Vendors’ claims become interpretable rather than confusing. Capacity planning shifts from “buy the GPU with the best number” to “validate performance under our actual execution context” — harder, and much more likely to hold.
For a deeper look at how the software layer specifically creates performance ceilings and pathways, see our discussion of the software stack as a first-class performance component.
Not reductionism in reverse
One misreading of this argument is “so you’re saying hardware doesn’t matter and it’s all software.” That’s wrong in the opposite direction.
Hardware determines the envelope of what’s possible. A GPU with more memory can serve larger models; a device with higher bandwidth can move data faster when the access patterns are favorable; architectural features like hardware support for specific precision formats create capabilities the software stack can exploit. None of that is diminished by recognizing that the software stack mediates how those capabilities are realized. The same GPU, driven by a newer driver, a different CUDA runtime, an upgraded framework, or a swapped kernel library, will exhibit a different effective ceiling — not because the silicon changed, but because the drivers, runtime, compiler, and framework collectively decide which fraction of that envelope the workload actually reaches. A fixed device is not a fixed performance number; it is a performance range whose realized value is set by the stack that drives it.
Which is also the point at which the hardware-versus-software framing stops earning its keep. Specs on one side, code on the other is a serviceable way to talk about procurement and a poor way to reason about a forward pass, because the interesting behaviour lives at the seams: what the graph compiler decided, what the allocator did under memory pressure, which kernel variant the dispatcher picked. Neither column owns those. The outcome belongs to the interaction, and reasoning about the layers in isolation produces incomplete and often misleading conclusions. If your performance model doesn’t include the executor, it’s not a performance model — it’s a hardware description with an implicit hope that everything else will be fine. In our experience, that hope is not a reliable engineering strategy.
The stack model applies in concentrated form to integrated silicon too: system-on-a-chip designs for AI tighten the physical coupling between components without removing the software stack from the performance equation.
LynxBenchAI holds this unit constant in practice rather than proposing it. A released 26Q3 run records the device, the backend, and the software present on the machine alongside the number, and pip install lynxbench-ai reproduces that record on ordinary consumer hardware — the free, non-commercial Personal Edition. So the honest question to carry out of here is not which device wins. It is whether the executor you are about to measure has been written down completely enough that someone else could disagree with your number for the right reasons.
Frequently Asked Questions
Why does AI performance emerge from the hardware × software stack rather than from hardware alone?
Because the throughput, latency, and efficiency numbers a team actually observes are produced by interactions between layers — kernels dispatched by the framework, scheduling decisions made by the runtime, memory access patterns shaped by the workload — not by hardware properties in isolation. The hardware sets the envelope of what is possible, but it does not choose which execution path the workload takes through that envelope. That choice belongs to the software stack, and the outcome belongs to the interaction.
Why does reasoning about performance one layer at a time tend to break down on AI workloads?
Because the layers are not independent contributors that can be summed. The framework’s graph lowering decisions change what the runtime schedules; the runtime’s scheduling changes what the kernels can sustain; the workload’s shape changes which subsystem gets stressed. Reasoning layer-by-layer assumes a decomposition that does not hold, which is why hardware-only or software-only models keep producing surprises in practice.
What does a hardware × software stack actually look like in layers, and where do the interactions between those layers most often determine real-world AI performance?
The stack runs from hardware (compute units, memory, interconnect, thermal envelope), through drivers and runtime (scheduling, allocation, synchronization), through frameworks (graph construction, operator fusion, kernel dispatch), through libraries and kernels such as cuDNN, FlashAttention and NCCL, to the workload itself (model shape, batch size, sequence length, precision mode). The execution stack layers table above maps each layer to what it controls and how it moves the measured number. The interactions that dominate real-world performance sit at the seams: kernel selection, memory access patterns, runtime scheduling, and system topology across PCIe, NUMA, and NVLink.
If two machines have the same GPU but different drivers, frameworks, and runtimes, what exactly is being compared when their benchmark numbers are placed side by side?
Two different executors, not one device measured twice. The device is only one element of the tuple; the backend, driver, framework, and runtime are the rest of it, and each of them changes which instructions actually reach the silicon. Placing the numbers side by side is legitimate only if you are asking a question about stacks — and even then, comparability holds within a release name, since results from different release names are not answering the same executor question.
What has to be recorded alongside a benchmark number for that number to remain meaningful to someone else later?
The device and its silicon type, the backend the work ran through (cuda, cuda via ROCm, xpu, or CPU), the driver, framework, and runtime versions present on the machine, the workload configuration including precision mode, and the release name. Anything the run did not measure — thermal state, settled-clock conditions — must not be implied. A number without that record may still be true, but it is no longer a claim anyone else can check or contradict.
How do you keep optimization effort from becoming the hidden variable when the same model is run across hardware from different vendors?
By bounding the effort and making it identical rather than generous. The models are prepared once, before any target device is known, and the same prepared artefacts ship to every vendor’s hardware. Where two vendors’ kernels disagree about what is permissible, the stricter constraint is applied to all of them instead of being tuned per vendor — which leaves some performance unclaimed on both sides, and is preferable to a comparison whose leading variable is which vendor received more attention.
Where does a hardware-versus-software framing (specs on one side, code on the other) stop being useful for reasoning about AI workload performance?
It stops being useful the moment the question becomes “why did this run behave this way,” because the behaviour lives at the seams neither column owns: graph lowering choices, allocator behaviour under memory pressure, kernel-variant dispatch, synchronization points. The framing survives for procurement conversations, where spec envelopes genuinely constrain what is buyable. For explaining a measured result, the executor tuple is the working unit and the two-column split simply has nowhere to put the interaction.
