“Can we just switch to FP8?”
The request sounds simple. FP8 offers roughly 2× the throughput of BF16 on supported hardware, halves the memory footprint, and lets larger models fit on fewer GPUs. An ML engineer or infrastructure planner hears “faster and cheaper” and reasonably asks why we aren’t already using it.
The answer, more often than not, is hardware. The precision formats a GPU can accelerate natively are determined by its tensor core architecture, and that architecture varies across generations. A format the silicon doesn’t support natively doesn’t just run slower — it may offer no throughput benefit at all, or the deployment framework may not support it on that target at all.
Precision decisions are hardware-conditional. Understanding the constraint is the prerequisite to making the decision well.
Tensor core generations and their numerical affordances
NVIDIA’s tensor core architecture has evolved across GPU generations, each one adding support for new numerical formats while keeping backward compatibility with the old ones.
Volta (V100): First-generation tensor cores. Native FP16 matrix multiply with FP32 accumulation. No BF16, no INT8 tensor core path, no FP8. Foundational for mixed-precision training, but a narrow format menu by today’s standards.
Ampere (A100): Third-generation tensor cores. Added native BF16 and TF32 (an internal 19-bit format applied transparently to FP32 operations), plus INT8 and INT4 tensor core support for inference quantization. No native FP8. A100s are still widely deployed and handle BF16 training and inference well, but cannot accelerate FP8.
Hopper (H100, H200): Fourth-generation tensor cores. Added native FP8 in both E4M3 and E5M2 variants, along with the Transformer Engine that manages dynamic per-tensor scaling for FP8. BF16 throughput also rose substantially over Ampere.
Blackwell (B100, B200): Further FP8 optimisation and FP4 support, with an extended Transformer Engine.
Each generation defines a different menu of viable precision choices. A deployment targeting V100s is limited to FP16 for tensor core acceleration. A deployment on A100s can use BF16, INT8, or INT4, but not FP8. A deployment on H100s can use any of those plus FP8. This is not a software limitation waiting for a patch: the silicon either has execution units for a format or it does not.
Tensor core format support by GPU generation
| GPU generation | Native tensor core formats | Notable limitation |
|---|---|---|
| Volta (V100) | FP16 | No BF16, no INT8, no FP8 |
| Ampere (A100) | FP16, BF16, TF32, INT8, INT4 | No FP8 — FP8 work falls back to roughly BF16 speed |
| Hopper (H100/H200) | FP16, BF16, TF32, INT8, INT4, FP8 (E4M3, E5M2) | Full menu with Transformer Engine scaling |
| Blackwell (B100/B200) | All above + FP4 | Newest software stack required to reach it |
Vendor documentation is the source for each row; the table is a reading of published architecture specifications, not a measurement.
What happens when you run a format the chip does not accelerate
Running an unsupported precision format rarely produces an error. The framework falls back to a supported format or emulates in software. The performance consequence is where the damage lands.
FP8 operations on A100 hardware execute on BF16 or FP16 tensor cores with conversion overhead, producing throughput roughly comparable to BF16 — no FP8 compute advantage despite the narrower format. The memory savings from an FP8 model representation still apply, since the weights really are smaller in HBM, but the throughput doubling FP8 promises on Hopper simply does not materialise on Ampere. INT8 inference on V100 behaves the same way: without a dedicated tensor core path it drops to CUDA cores, well below the INT8 throughput available on A100.
There is a subtler version of the same trap. A format can be nominally supported and still not be efficient, because the format is only half the story — the scaling machinery it depends on must also exist in silicon. FP8 is the clearest case. Sustained FP8 accuracy relies on per-tensor or block scaling that rescales values to use a narrow exponent range well, and on Hopper the Transformer Engine performs that dynamic scaling on dedicated units. Finer-grained blockwise quantization, scaling per small block rather than per tensor, is more aggressive still. Where that scaling has no hardware acceleration it runs on slower paths and erodes — sometimes erases — the throughput the format was chosen to deliver. “FP8 supported” on a specification sheet does not tell an architect whether the block-scaling path that makes FP8 usable is one the hardware does efficiently.
So when tensor core support is absent or partial, the work does not stop; it falls back. The matrix multiply drops to a wider-format tensor core or to shader/CUDA-core execution, and any scaling the format needs runs in software. A reported throughput number measured on a fallback path describes the fallback, not the format. Before trusting a per-precision figure, ask which units actually ran the workload. Benchmark results measured on H100 at FP8 say nothing about FP8 on A100, because at the hardware acceleration level A100 FP8 performance does not exist.
Why the spec sheet and the machine disagree
Hardware support is necessary but not sufficient. The deployment framework must also support the format on that hardware, with optimised kernels and correct numerical handling.
TensorRT gained FP8 support alongside Hopper. Earlier TensorRT versions targeting A100 handle INT8 and FP16 but not FP8. PyTorch’s native FP8 support arrived in specific releases and expects Hopper-class hardware with a compatible CUDA toolkit. That produces a three-layer compatibility requirement: the silicon must support the format, the framework must support it on that silicon, and the driver and CUDA stack must be at a version that exposes the feature. A mismatch at any layer blocks the strategy — a current GPU with an old framework fails as reliably as a current framework on previous-generation silicon.
This is why precision support has to be treated as a property of the whole executor rather than of the chip. In LynxBenchAI every score binds to the AI Executor: the device together with the backend it ran through — cuda for NVIDIA, cuda via ROCm for AMD, xpu for Intel, or CPU — and the driver, framework, and runtime present on the machine. Never to silicon on its own. A card whose datasheet lists FP8 and whose run records nothing for FP8 is telling you something about the software layers, not about the transistors.
The released 26Q3 run handles this by declaring support through execution rather than through a capability query. Each precision in the catalogue is attempted on the device, the result binds to the executor that produced it, and a format the machine cannot run is recorded as a zero rather than quietly skipped. The unsupported format stays visible on the page instead of vanishing from it. That zero is not a verdict on the device — it records the absence of that regime under one release’s conditions, and it does not travel across releases: a per-precision figure carrying one release name should not be compared with one carrying another. pip install lynxbench-ai turns a specification-sheet claim about precision support into something the owner of the card can check.
The same discipline is what makes cross-vendor precision comparison meaningful. One catalogue and one rule set run across NVIDIA, AMD, Intel, and CPU, so a precision-support gap between vendors is measured by a single instrument rather than assembled from each vendor’s preferred test. Results accumulate on a public leaderboard, where a device’s absence is itself readable information about what has been measured rather than a silence to fill in with assumptions.
As we set out in how FP8, FP16, and BF16 represent different operating regimes, the format choice encodes assumptions about numerical behaviour. The hardware choice determines which of those assumptions can actually be realised at speed.
Making precision a first-class line in hardware evaluation
Precision support belongs in the evaluation criteria, not in a footnote after the purchase decision.
If the inference workload benefits substantially from FP8 — large language models, high-throughput serving, memory-bandwidth-bound operations — then FP8 tensor core support is a first-class requirement, because the gain from moving down a format often exceeds the raw compute improvement between generations. If the workload is precision-sensitive and will stay at BF16 or FP32, the absence of FP8 on the target is irrelevant and the evaluation should spend its attention elsewhere.
The failure we see most often is evaluating at one precision and deploying at another without accounting for the change, or assuming a format available on the newest generation is available on the fleet already racked. An honest hardware evaluation states plainly which precision format is intended for deployment, that the hardware accelerates it natively, and that the benchmark numbers were measured at that format on that stack. Anything less opens a gap between evaluation and deployment that the economics of precision choice eventually exposes.
Teams running this decision across a mixed fleet rather than a single card face the engineering bill directly — what cross-platform GPU performance portability actually requires is the applied counterpart to the constraint described here.
The hardware does not bend to the precision strategy. So which layer of the stack — silicon, driver, framework, or scaling path — will you check first the next time a per-precision score reads lower than the datasheet implies?
Frequently Asked Questions
How does GPU and accelerator architecture constrain which precision formats are viable on a given system?
The silicon either has execution units for a format or it does not. V100 tensor cores accelerate FP16 only; A100 adds BF16, TF32, INT8, and INT4; Hopper adds FP8; Blackwell extends to FP4. A format outside the chip’s native menu may still execute, but via emulation or fallback to a wider format, which surrenders the throughput the format was chosen for.
Why does tensor core support shape which precisions are actually efficient, not just which are technically supported?
Almost any format can be made to work through software conversion; efficiency comes from dedicated execution units. FP8 multiplied on BF16 tensor cores with conversion overhead delivers roughly BF16 throughput, not the 2× advantage seen on Hopper. The economic case for low precision collapses without the hardware path, so the question worth asking is about native acceleration, not nominal support.
How do different hardware generations differ in their numerical affordances for AI workloads?
Each generation defines a different menu. Volta gives FP16 mixed precision. Ampere adds BF16, TF32, INT8, and INT4 for inference quantization. Hopper introduces native FP8 in E4M3 and E5M2 variants plus the Transformer Engine. Blackwell pushes to FP4. Backward compatibility holds, but new formats do not retrofit onto older silicon.
How do block scaling and blockwise quantization interact with hardware support, and why can a precision format be nominally supported but require scaling machinery that not all hardware generations provide efficiently?
FP8 in practice depends on per-tensor or block scaling to keep values inside its narrow exponent range, and finer blockwise quantization scales per small block rather than per tensor. On Hopper the Transformer Engine accelerates that dynamic scaling on dedicated units. Where the scaling path has no hardware acceleration it runs in software and eats the throughput the format promised — so a format can be listed as supported while the machinery that makes it usable is not efficiently provided.
If a card’s specification sheet lists FP8 support but a benchmark run records a zero for that precision, what does the gap between the two actually tell an architect about the driver, framework, and runtime on the machine?
It tells you the format was attempted and did not execute on this executor, which points at the software layers rather than the transistors — a framework build without FP8 kernels for that backend, a CUDA or driver version that does not expose the feature, or a runtime path the release does not exercise. Because every score binds to the device plus its backend, driver, framework, and runtime, the zero is a statement about that whole stack under one release’s conditions. It is not a judgement on the device, and it does not carry over to a differently configured machine.
Why does a per-precision comparison across NVIDIA, AMD, Intel, and CPU only hold when the same catalogue and rule set are applied to all four, rather than assembled from each vendor’s own precision test?
Because each vendor’s own test is free to choose the workloads, shapes, and scaling strategies that its hardware handles best, so a precision-support gap measured that way records test design as much as capability. Running one catalogue and one rule set across all four backends means a difference in a per-precision figure is attributable to the executor rather than to the instrument. It also makes an absence legible: a missing or zero result means the same thing everywhere.
What is the practical difference between tensor-core execution and general shader/CUDA-core execution for a given precision, and why does that difference matter when interpreting a reported per-precision score?
Tensor cores are fixed-function matrix-multiply units with a specific list of supported input formats and accumulation behaviour; shader or CUDA cores are general-purpose lanes that can compute the same result at far lower matrix throughput. When the format is outside the tensor core’s menu, the multiply drops to those general lanes and any scaling runs in software. A per-precision score measured on that path describes the fallback, so an FP8 or INT8 figure from hardware without native support is not evidence of what the format can do where it is accelerated.
