The cost line nobody expected
An inference team deploys a model in BF16, measures the per-request cost, and builds a unit economics model. Six months later, request volume has tripled. The GPU fleet is growing proportionally. Someone asks the obvious question: what happens to cost if we shift the model to FP8?
The arithmetic is revealing. FP8 halves the memory footprint, so the model fits on fewer GPUs — or serves larger batches per GPU. FP8 tensor cores deliver roughly 2× the throughput of BF16 on Hopper-generation silicon. Combined, the effect isn’t just “inference is faster”; it’s “inference costs substantially less per request, at scale, over time.” The precision format change didn’t improve the model. It didn’t add features. It changed the economics of running the model in production.
That is what makes precision an economic lever rather than a numerical detail. And it is why the decision belongs in the infrastructure conversation, not in a late-stage optimisation ticket.
The three-axis impact
A precision reduction in an inference system moves at least three cost-relevant dimensions at once.
Throughput. Lower precision means more operations per tensor core cycle on hardware that supports the format natively. FP8 on H100 tensor cores runs at roughly 2× the FLOPS of BF16. For compute-bound workloads that translates directly into more requests per second per GPU, and fewer GPUs for the same request volume.
Memory. A model in FP8 uses half the HBM of the same model in BF16, and a quarter of FP32. That either fits a larger model on a single GPU — avoiding multi-GPU serving overhead entirely — or allows more concurrent requests in larger batches. Both reduce cost per request.
Power. Lower-precision operations generally consume less energy per operation. At data-centre scale, power is a meaningful fraction of total infrastructure cost, and a fleet drawing less power per request stretches the effective capacity of the power and cooling envelope you already paid for.
These effects compound rather than add. Throughput improvement reduces the GPU count. The memory improvement enables better batching, which raises utilisation on the GPUs that remain. The power reduction lowers operational cost on every one of them. The total cost impact of a precision format change can exceed the impact of a hardware generation upgrade — without purchasing any new hardware.
Precision’s economic impact on inference systems
| Impact axis | Mechanism | Approximate magnitude (FP8 vs. BF16) | Evidence class |
|---|---|---|---|
| Throughput | More tensor core operations per cycle | ~2× on Hopper tensor cores | Vendor-published tensor-core spec |
| Memory | Smaller model footprint in HBM | Half the memory; may enable single-GPU serving | Arithmetic from format width |
| Power | Less energy per operation | Lower per-request draw across the fleet | Observed pattern; not a benchmarked rate |
| Compound effect | All three axes reinforce each other | Total reduction typically exceeds any single axis | Observed pattern across engagements |
Higher precision can be economically wasteful
This is the less comfortable half of the argument. If a model’s output quality at BF16 and at FP8 is equivalent within the application’s requirements — and accuracy loss from lower precision is task-dependent rather than uniform — then running at BF16 is paying for precision the application never consumes.
It is the equivalent of shipping every parcel by priority overnight courier when standard post arrives on time. The premium buys nothing except the reassurance of having paid for it.
In a system serving millions of requests, the cost of unnecessary precision is real and cumulative. Each unneeded bit is extra HBM occupied, extra memory bandwidth consumed, extra power drawn, extra GPU-seconds billed — with no change in the user-facing output. None of which means lower precision is automatically correct. It means precision should be chosen against what the task requires, validated against quality metrics, and then deployed at the lowest format that clears those requirements. Defaulting to the most precision available is not conservative engineering; it is an unexamined cost assumption.
Cost-optimal precision depends on workload and SLA
The right precision is not universal. It is a function of workload characteristics, quality requirements, and infrastructure constraints, and those three rarely line up the same way twice.
A large language model generating text for a customer-facing chatbot may need BF16 to preserve the subtle reasoning quality users perceive. The same model powering internal document summarisation — where a human reviews every summary before it is used — may produce equivalent utility at INT8. An image classification model in a real-time video pipeline may need the latency headroom FP8 provides to hold a frame-rate SLA, while a batch classifier running overnight has time to spare and can weight accuracy above throughput. A system with fixed GPU capacity absorbing growing traffic faces a different calculus from one on autoscaled cloud instances where GPU-hours are billed directly.
Each scenario lands on a different optimum, which is why treating precision as a design parameter rather than a binary quality gate matters. The design question is narrow and answerable: what precision does this workload need, at this SLA, at this scale, on this hardware?
How does precision choice affect infrastructure decisions?
Precision feeds back into infrastructure in ways that go well beyond per-GPU performance.
Fleet composition. If the target format is FP8, the fleet needs Hopper-generation hardware or newer. If BF16 is acceptable, Ampere remains viable. Precision choice can pull a hardware refresh forward or push it back, with capex consequences attached.
Deployment topology. A 70B-parameter model at BF16 requires multi-GPU serving — 140 GB exceeds single-GPU memory. At FP8 it fits on one H100 with room to spare. That single change removes inter-GPU communication overhead, simplifies the serving architecture, and deletes a class of failure modes. In our experience the topology effect frequently outweighs the direct throughput gain in the cost model.
Capacity planning. As set out in the wider argument about how FP8, FP16, and BF16 represent different operating regimes, each format implies a different throughput-per-GPU, therefore a different GPU count, rack density, and power budget for the same request volume.
Total cost of ownership for an inference system is shaped by precision at every level of the stack. Teams that defer the question until after the fleet is provisioned have already spent the money that the decision was supposed to influence.
What a released benchmark can and cannot settle
A precision cost argument needs two inputs: a throughput figure that survives sustained load, and an honest statement of what the figure leaves out. A released 26Q3 LynxBenchAI run supplies both. A warm-up phase is discarded, completed iterations are counted inside a continuous timed window, and the workload is scaled until throughput stops improving — so the number reflects a system under load, not a burst. Per-precision behaviour is reported across the catalogue, and throughput is weighted by the memory a run actually moves, which means a cheap-looking low-precision result has to earn its speed-up rather than be credited simply for being cheap.
Results arrive as Training, Inference, and Compute scores plus GT, an ordinal aggregate over those three category scores. Keeping the categories separate keeps a precision decision attached to the class of work being paid for, instead of collapsing everything into one undifferentiated rating. GT is not a physical quantity and not a 0–100 rating; treating it as either turns an ordering into a measurement it never claimed to be.
The exclusions matter as much as the inclusions. A score is not a price, not a cost-per-request figure, and not a stand-in for your own bill — the run covers a fixed catalogue under one named release. Conditions the run never records, thermal state among them, cannot be charged against it afterwards. Figures do not travel across release names either: 26Q3 is current, 27Q1 is next, and that is a naming scheme rather than a schedule. Converting a throughput figure into a currency figure is the reader’s step, done against the reader’s own utilisation, billing model, and quality floor.
What has changed is access to the input. The measurement comes off the machine already on the desk — pip install lynxbench-ai, 15 to 30 minutes, Personal Edition free for non-commercial use — so building a precision cost argument no longer depends on privileged hardware access. The Press, Pro, and Enterprise editions are contact-gated and unpriced; only the Personal Edition has shipped.
Cost per token, hardware choice, and the CPU-vs-GPU question
Two questions surface repeatedly once a system is in production: where is cost per token heading over time, and is inference better on CPU or GPU? Both turn on the same precision logic.
Cost per token drifts as precision and hardware choices change. A workload that launched on BF16 Ampere hardware sees its cost per token fall on FP8 Hopper, because throughput-per-GPU rises and per-request power draw falls. For capacity planning, treat today’s cost per token as a ceiling rather than a constant: the same request volume can be served on a shrinking fleet as precision support and hardware mature. Budget against the trend, not the launch-day figure.
The CPU-versus-GPU choice is also a precision question rather than a raw-speed one. For low-volume, latency-tolerant, or sparse workloads, CPU inference at INT8 can be the cost-optimal configuration once you account for the GPU’s idle-capacity tax — a half-utilised GPU still bills for the whole card. GPUs win decisively when the workload is compute-bound and the precision format maps onto native tensor-core acceleration, because that is where the throughput multiplier and the batching gains compound. The decision is not “CPU or GPU” in the abstract; it is which device runs this workload’s precision format at the lowest cost per token under the SLA.
Precision is one input into a larger calculation. Unit economics for production AI is where the same per-request arithmetic gets worked out against a real deployment and a real invoice, and where the buyer’s go/no-go on lower precision has to hold up against a quality bar afterwards.
The operational conversation
Precision deserves a seat in infrastructure planning alongside hardware selection, capacity modelling, and SLA definition — before procurement, not after deployment. What precision can the target workload tolerate, and has that been validated empirically rather than assumed? What hardware accelerates that format natively? What is the cost differential between the options at the anticipated request volume, across the hardware’s projected life?
Those questions produce better decisions than “buy the fastest GPUs and run at default precision.” The fastest GPU at default precision is rarely the most cost-effective configuration. The cost-effective one is usually where precision, hardware, and workload requirements were aligned explicitly, and the alignment was measured before the purchase order went out. Which raises the question worth carrying into the next planning cycle: if a run is what turns a precision assumption into a defensible number, why is the run still happening after the fleet has been bought rather than before?
Frequently Asked Questions
How does numerical precision act as an economic lever on inference throughput, latency, and cost simultaneously?
A precision format change moves three cost-relevant axes at once: throughput (FP8 on Hopper tensor cores runs at roughly 2× the FLOPS of BF16), memory footprint (FP8 halves HBM use versus BF16), and power per operation. Because the axes compound — fewer GPUs needed, better batching on the GPUs you do run, lower power-per-request across the fleet — a single precision decision can shift unit economics more than a hardware generation upgrade, without buying new hardware.
Why can higher precision become economically wasteful for an inference workload?
When BF16 and FP8 produce equivalent output quality within the application’s requirements, running BF16 is paying for precision the application doesn’t consume. Each unnecessary bit is extra HBM, extra memory bandwidth, extra power, and extra GPU-seconds billed — multiplied across millions of requests — with no change in what the user sees. Defaulting to the highest available precision isn’t a conservative engineering choice; it’s an unexamined cost assumption.
How does cost-optimal precision depend on the workload and the service-level constraints around it?
The optimum is a function of workload characteristics, quality requirements, and infrastructure constraints. A customer-facing chatbot may need BF16 to preserve subtle reasoning quality, while the same model doing internal document summarisation may deliver equivalent utility at INT8. A real-time video pipeline may need FP8’s latency headroom to meet frame-rate SLAs; an overnight batch job has no such pressure. Fixed-capacity fleets and autoscaled cloud billing also produce different calculus on the same model.
How should quality constraints be kept in view when treating precision as a cost lever?
Precision should be selected by what the task requires, validated empirically against quality metrics, and then deployed at the lowest precision that meets those requirements. The order matters: define the quality envelope first, measure model behaviour across precision formats, and only then lock in the format that maximises throughput and minimises cost inside that envelope. Cost optimisation without a measured quality floor is not optimisation — it’s gambling on user-facing regressions.
How do CPU vs GPU inference choices interact with precision selection when optimizing for cost rather than raw speed?
The device choice is a precision question, not a speed contest. For low-volume, latency-tolerant, or sparse workloads, CPU inference at INT8 can be cost-optimal once the GPU’s idle-capacity tax is counted — a half-utilised GPU still bills for the whole card. GPUs win when the workload is compute-bound and its precision format maps onto native tensor-core acceleration, such as FP8 on Hopper, because that is where the throughput multiplier and batching gains compound. The real question is which device runs this workload’s precision format at the lowest cost per token under the SLA.
How should a per-precision throughput figure be weighted by the memory a run actually moves, so a cheap-looking low-precision result has to earn its speed-up?
A 26Q3 LynxBenchAI run weights throughput by the memory the run actually moves, rather than reporting raw operations per second. That weighting stops a low-precision format from being credited for a speed-up it obtained purely by moving fewer bytes: the format has to deliver work, not just a smaller footprint. Without it, a precision comparison rewards the cheapest representation regardless of what the system accomplished.
What does a released 26Q3 per-precision result explicitly exclude, and why do those exclusions matter before anyone turns a throughput figure into a cost argument?
It excludes anything the run does not record — thermal state among them — and it covers a fixed catalogue under one named release, so the figure is not a price, a cost-per-request number, or a stand-in for your bill. Figures also do not travel across release names; 26Q3 is current and 27Q1 is next, which is a naming scheme rather than a schedule. Those exclusions matter because converting throughput into currency requires your own utilisation, billing model, and quality floor, and a cost argument that hides that step is not auditable.
