The cost of inference on your own GPUs is the hourly cost of the hardware divided by the throughput it actually delivers at your real offered request rate while still meeting your latency SLO. List prices and an assumed utilisation figure leave out offered request rate, the term that moves the answer most. A 2026 arXiv study of LLM serving costs measured effective cost on identical H100 hardware ranging from $0.21 to $15.25 per million output tokens, and within a single configuration cost moved by 17.5 to 36.3 times purely with offered load. The study’s main sweep ran vLLM with prefix caching and speculative decoding switched off, so its authors present its absolute costs as upper bounds that tuned production stacks should beat. A cost-per-request model therefore has to rest on goodput measured at your SLO, not on generic assumptions.

The cost of inference is hardware cost divided by goodput at your SLO

The base formula is short: cost per million tokens equals hardware cost per hour divided by millions of tokens produced per hour. Cost per request has the same shape, with requests served per hour as the divisor.

The 2026 study found that the public LLM cost calculators it surveyed treat GPU utilisation as a fixed input, typed in by the user or silently assumed to be 100%, and never measured against the operator’s actual load. Its central argument is that utilisation is not an input at all: it is the output of how arrival rate, model architecture, quantisation and the SLO interact. A 2025 paper on the economics of inference makes the same correction in accounting terms, defining effective hourly cost as base hourly cost divided by utilisation. On that definition, a GPU running at 70% costs about 1.43 times its nominal hourly rate per useful hour (1 / 0.7).

The low-load end is where naive estimates fail worst. At 1 request per second on an H100, a Mixtral 8x7B FP16 deployment cost $15.25 per million tokens, and at 25 requests per second the same hardware cost $0.87, roughly 17.5 times cheaper. Even at 10 requests per second, which most practitioners would not call low, the utilisation-naive estimate was still wrong by 2.5 times on dense Llama.

The SLA is a price

Signing a latency target caps the offered load a deployment can carry before its tail breaches, and that pins where you sit on the cost curve. Tighten the target and you are forced to a lower load and a higher cost per token. In the same study, Llama 8B FP16 hit its cheapest point of $0.311 per million tokens only at 200 requests per second, where p99 time to first token was 7.4 seconds, while the cheapest cost that actually met the SLA was $0.368 at 25 requests per second. The same cost cliff showed up again on A100, though the spread there was narrower, at 7.0 to 11.4 times.

Measure the right utilisation

The familiar GPU utilisation metric only records whether a kernel is running, so a GPU can show 100% while doing almost no floating-point work. For a cost model, the honest divisor is delivered requests or tokens per second at your SLO, measured from observed traffic rather than assumed. On the hardware side, SM efficiency gives a more representative picture than raw utilisation.

Why does cost per request swing so much when traffic is bursty or triples at peak?

Cost per request depends on where your offered load sits on the cost curve. In the 2026 measurements, effective cost fell about 20 times over the first 25 requests per second and then levelled off as the engine saturated. Past the saturation threshold, extra load makes latency worse without lowering cost. So a seasonal tripling of throughput is good news for unit cost if the capacity was mostly idle beforehand, and bad news for your SLO if the capacity was already close to saturation. Capacity sized for peak runs at lower utilisation, which is why a worked example in a book chapter on inference cost modelling, sized for peak, came out with a higher cost per token than its average-load estimate.

The evidence on burstiness on its own is mixed. When the 2026 study replaced Poisson arrivals with burstier Gamma arrivals (coefficient of variation 2) at the same average rate on vLLM, cost changed by less than 1%, which was inside run-to-run noise. The BurstGPT study, built on real Azure OpenAI traces, found the opposite kind of effect: burstier concurrency at the same requests per second raised latency and resource occupancy, and pushed KV cache management harder than a more uniform workload did. In that study, burstiness also caused memory bottlenecks that led to spikes in failure rates.

The two results measure different things. One measured cost per token on vLLM at a fixed average rate; the other measured latency, stability and failures on real-trace workloads. Burstiness is therefore worth measuring on your own traffic shape. BurstGPT supports that caution: conversational services followed periodic working-hours patterns while API services were aperiodic and bursty, and a scheduling optimisation that helped one service type did not carry over to the other. Request length matters too. Under variable-length arrivals the load-driven cost spread widened to 39.3 and 47.6 times, from 31.9 and 36.3 times with fixed lengths. Characterising that traffic before sizing anything is a separate job, covered in our guide to characterising traffic to right-size inference capacity.

Batching and autoscaling bend the curve; fixed capacity only moves along it

Production cost and latency can often be improved without replacing the model, because the main levers (batching, caching, routing, quantisation, serving configuration, runtime choice) sit below the model layer and can be tuned independently. Renting more fixed GPUs adds GPU-hours you pay for whatever the load. It moves you back towards the expensive low-utilisation end unless demand grows to fill them.

Continuous batching is one example. LLM inference is usually bound by memory rather than compute, so throughput depends largely on how big a batch fits in GPU memory. Iteration-level scheduling slots a new sequence into the batch as soon as another finishes, instead of waiting for the whole batch to complete, which raises GPU utilisation compared with static batching. Anyscale, which sells serving infrastructure, reported up to 23 times higher throughput from continuous batching combined with vLLM’s memory optimisations, alongside lower median latency. As the system saturates, there are fewer chances to slot new requests in straight away, so the latency benefit narrows. Batch size is also a trade-off with latency: interactive chat wants smaller batches and more replicas, and background processing can use large batches for a lower cost per token.

The right autoscaling signal depends on your latency target. Google Cloud recommends scaling on queue size when you are optimising for throughput and cost and your latency target can be met at the model server’s maximum batch size, because queue time adds directly to request latency. Scaling on batch size suits load spikes and maximises parallel work, but Google advises giving queue size priority if it already meets your latency targets. Policy design has its own page: designing autoscaling policies for inference capacity planning.

Where the deployment sits	Lever that changes cost per request	Basis for the choice
Deep in the low-load cliff	Consolidate tenants, scale replicas down to the offered load, coalesce requests, or fall back to a serverless endpoint below the crossover	The 2026 study describes the penalty as a sign of over-provisioning, not just a calculation error
Mid-curve, SLO comfortably met	Larger or continuous batches, scaling on queue size	Higher concurrency spreads fixed cost until the SLO binds
Near saturation	Extra load mainly hurts latency, with no cost gain	Past saturation, load degrades latency without improving cost
Bursts above committed capacity	Spillover to pay-per-token capacity	Azure bills spillover requests at token rates and provisioned requests at the hourly rate only

Which of these levers pays off depends on profiling the actual stack, not on a best-practice list. TechnoLynx audits the production serving path for these improvements before recommending a model replacement, because a model swap is the most expensive lever and often leaves the workload economics unchanged.

Can we profile production inference without breaking our SLAs?

Yes. Copy live traffic to a shadow deployment that never answers users, and move to a canary with real users only after the shadow results hold. A shadow deployment processes live inputs and logs its outputs without ever serving a prediction. The safeguards are blocking shadow responses at the proxy, plus rate limits, strict timeouts and isolation at the edge. The Google SRE Workbook calls this teeing: production serves the real traffic, while the test system serves a copy and throws its responses away. We would put the shadow on separate capacity, so that the duplicate load is not eating into the production pool’s headroom.

Full duplication can double the bill on large routes. Future AGI recommends mirroring a 10 to 25% sample for new candidates, and moving to full shadow only when the candidate is cheap or the route is safety-critical. A canary is needed when the signal you care about only exists with users in the loop. It serves a slice of real users with automatic rollback armed, so its blast radius is not zero by design. Traffic usually follows a power law, so the same source recommends stratifying the canary and starting with the segment whose failure cost is lowest. For cost profiling, timing matters as well: performance defects tend to show up only under heavy load, so a test run off-peak can miss them.

Replaying real traffic to validate a cost model

Synthetic load tests often fail to reproduce the real traffic patterns that set your costs. The team behind ShadowReader built it because JMeter tests sampled from production URLs but could not reproduce the exact call frequency, or burst rate, that customers generated. The SRE Workbook makes the general point: synthetic load is good for code coverage and poor for state coverage. These are the steps for validating a model before it is trusted in production:

1. Capture the real arrival pattern. Take request timestamps and prompt and output length distributions from production logs, the same way ShadowReader takes URLs and request rates straight from logs. If you have no traffic of your own yet, Azure’s public LLM inference traces (collected in November 2023 and May 2024) and the 121-day BurstGPT dataset are usable substitutes. BurstGPT can be scaled to a target request rate by multiplying the timestamps.
2. Write the SLO down in measurable terms. Use time to first token and time per output token as the benchmark client measures them. Metric names are not standardised across tools, so compare measurement points and formulas, not labels.
3. Sweep arrival rates with ramp phases. Ramp up, hold the load long enough to see whether the system settles, then ramp down. Tools such as vllm bench serve, inference-perf, LLMPerf or GenAI-Perf can run the sweep with your real length distributions and arrival pattern.
4. Record goodput. Goodput is the highest arrival rate at which the SLO still holds, which is different from raw saturation throughput.
5. Plug goodput into the cost formula. The result is the effective cost under the contract you actually operate to.
6. Reconcile against live metrics. Server-side queue time separates scheduler wait from model time inside client-measured TTFT. The 2026 study released a tool that scrapes vLLM’s Prometheus endpoint and computes live effective cost from observed token throughput.

In the 2026 study, live operation traced a curve that matched the shape the benchmark sweep predicted.

Who profiles our serving path and turns it into a cost model?

TechnoLynx builds the cost-per-request model, starting with a profiling pass on your current serving path. When choosing a partner, ask to see the measurement method before the recommendation. This work sits within our AI infrastructure and production SaaS engineering. For the hardware choice, LynxBenchAI’s published results show how each device runs the public PyTorch stack your models are built on. They are an indicator, not a run of your own workload.

A team consuming a hosted API, with no plan to self-host, is mostly modelling the provider’s rate card, because the API charges only for tokens consumed. Teams with engineering capacity can start from the open-source tool the 2026 study released, and for allocating costs inside Kubernetes, OpenCost can spread idle cost across workloads.

How should we price inference so small, spiky customers don’t lose us money?

Price so that each customer’s revenue covers the capacity that customer forces you to hold. That usually means per-token or per-request pricing on a shared pool for small, bursty accounts, and hourly or committed pricing with a floor for customers whose load keeps a dedicated GPU on the flat part of the curve. TechnoLynx frames this pricing risk against your measured GPU inference cost curve, and implements continuous batching and autoscaling with burst fallback. LynxBenchAI’s published device results then help choose which GPU class the tiers are built on.

LynxBenchAI publishes results showing how each device runs the public PyTorch stack, so the first step is to check its results for the GPU classes you are considering before setting tier floors.

GPU capacity: floors, commitments and shared pools

With reserved or owned hardware you pay for GPU-hours whatever the load, so at 10% utilisation each token costs ten times what it would at full utilisation, while an API charges only for tokens consumed. The floor or minimum commitment should therefore kick in where serving a customer on dedicated capacity costs more per request than serving them from the shared pool. Sources disagree on where that crossover sits. A book chapter on inference cost modelling puts it at 100 million to 500 million tokens a month, while the 2026 study argues the crossover is a surface that moves with all five of its inputs, not a single break-even. We find the second view stronger, because it rests on measured sweeps across models, precisions and two hardware families.

Consolidating tenants onto shared capacity is one remedy the 2026 study names for deployments deep in the cost cliff. Bursts above committed capacity can spill over to pay-per-token capacity, which is how Azure’s spillover bills overage. Two caveats matter. Bursty traffic is harder to serve: BurstGPT found its bursty concurrency strained KV cache management more than a more uniform workload at the same rate. And tiers designed from simulated load profiles need checking against real customer usage distributions before they are published.

Generative media: per second, per job or per compute unit

For video and image generation, the billing dimension matters more than the headline rate. An August 2026 comparison asked six video API vendors for the same five-second clip and got invoices from $0.25 to $2.00, and argued that every meter must be normalised to dollars per generated second at the target resolution before any comparison makes sense. Google’s Veo 3.1 API bills per second, from $0.05 to $0.08 for Lite up to $0.40 to $0.60 for Standard. Each generation is capped at 8 seconds, and every extension bills as additional generated seconds. CloudZero estimates real Veo spend at 3 to 10 times the per-second rate, because every take is paid for and only one is kept.

According to the same comparison, per-generation pricing is the easiest to forecast and the harshest when projects drift to 1080p, while per-pixel pricing tracks real render load. A flat per-job price subsidises long clips. A pricing unit that suits a competitor may not match your own cost drivers.

Turning fleet telemetry into a workload number you can size a platform on

Reweighting fleet logs to the population you are sizing for reduces the bias in the workload figure before it is projected to the next platform. Where the result feeds a hardware choice, LynxBenchAI’s published results show how candidate devices run the public PyTorch stack, as an input alongside the projected workload.

LynxBenchAI publishes results showing how candidate devices run the public PyTorch stack, so the first step is to look up the devices on your shortlist before projecting the workload onto them.

When some trims are overrepresented, a raw fleet average reflects the current fleet mix, not future demand. Survey statistics has standard corrections: the open-source balance package implements post-stratification, raking and propensity-score weighting. The framework needs auxiliary data on the target population, and weight trimming keeps the weights’ mean and sum unchanged. Averaging is not enough on its own either. A 2026 edge-to-cloud workload study for autonomous vehicles showed that a constant per-vehicle rate can match the 24-hour traffic total but cannot represent hourly changes driven by application timing. In its Munich case, shifting when training data were collected raised the highest peak about 12% above the lowest at the same daily volume, and telemetry accounted for 90.4% of mean demand. This kind of workload modelling also feeds per-unit inference cost modelling at fleet scale.

The result is a sizing input, not a guarantee. It is only as good as the reference population behind the weights, it assumes the next model stack is reasonably well known, and new features can bring usage patterns the current fleet never shows.

Measured uptime beats nameplate uptime, on a factory line too

The same correction applies to a cost-per-metre model on a production line: pull run and idle states from the MES instead of assuming nameplate uptime. MES process logs record starts, completions, delays and the idle time between actions. One analysis of about 1.4 million MES observations over several months of 2020 found skewed idle-time distributions, with different process steps dominating idle time on different shifts.

The OEE definitions spell out the gap. Nameplate capacity is the equipment builder’s figure, and a time study measures the fastest speed the process actually supports. A 1994 Ford paper adds two practical traps: an uptime interval that crosses a shift change is one interval, not two, and the shortest stoppages go unrecorded. Feeding the measured distribution into the throughput simulation shows whether shift-to-shift variance changes cost per metre. Whether a difference counts as material is a threshold the business has to set, and it is only worth setting if the data can tell planned idle time from unplanned.

Splitting one shared GPU’s cost across lines that run unequal volume

Split a shared GPU by the GPU-time each line actually consumes, or by a blend of GPU-time and inspected-unit count. If per-unit processing time and utilisation are similar across lines, unit count is simpler and close enough. Measured allocation depends on per-line usage tracking, which is the job of request-level cost attribution via GPU-second metering.

LynxBenchAI publishes results showing how each device runs the public PyTorch stack, so the first step is to check the shared GPU’s results before agreeing each line’s share of its cost.

Will our cost-per-request model survive a CFO’s line-by-line audit?

It will if every figure traces back to a measurement, if stages with different cost drivers are costed separately, and if the model can be rerun next quarter rather than re-estimated. TechnoLynx builds the measured base for such a model. LynxBenchAI’s published results add a broader indicator of how each device runs the public PyTorch stack. They do not measure your own application.

LynxBenchAI publishes results showing how each device runs the public PyTorch stack, so the first step is to cite those results for the hardware in the model as a stated external indicator.

How idle cost is spread across shared infrastructure, and what utilisation the shared nodes are assumed to run at, should be stated as choices with their alternatives. Results are tied to the specific hardware, inference engine and quantisation, so changing any of them changes the numbers. The 2026 study reproduced throughput and effective cost to within 0.31% coefficient of variation across three repeat runs. Next quarter’s figure should come from rerunning the harness, not from editing last quarter’s spreadsheet.

FAQ

What is the formula for utilization?

For inference, the 2026 cost study defines utilisation as achieved throughput divided by the maximum throughput for that hardware, model and quantisation, which makes it a result of offered load and SLO rather than a fixed number.

Sources

1. Chapter 30: Cost modeling for inference: tokens, GPUs, dollars — The Holy Grail, kunwar.page
2. Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation, arxiv.org
3. Beyond Benchmarks: The Economics of AI Inference, arxiv.org
4. GPU Utilization is a Misleading Metric, trainy.ai
5. BurstGPT: A Real-World Workload Dataset to Optimize LLM Serving Systems, arxiv.org
6. How continuous batching enables 23x throughput in LLM inference while reducing p50 latency, anyscale.com
7. Best practices for autoscaling large language model (LLM) inference workloads with GPUs on Google Kubernetes Engine (GKE)  |  GKE AI/ML  |  Google Cloud Documentation, docs.cloud.google.com
8. LLM Inference Benchmarking: How Much Does Your LLM Inference Cost?, developer.nvidia.com
9. Manage traffic with spillover for provisioned deployments, learn.microsoft.com
10. LLM Eval with Shadow Traffic and Canary Deployment in 2026, futureagi.com
11. Shadow deployment: Risk-free performance comparison, statsig.com
12. Canarying Releases (Google SRE Workbook ch.16), sre.google
13. ShadowReader: Serverless load tests for replaying production traffic, opensource.com
14. Azure Public Dataset (incl. Azure LLM Inference Traces 2023/2024), github.com
15. vLLM Benchmark CLI, docs.vllm.ai
16. Load Testing - Engineering Fundamentals Playbook, microsoft.github.io
17. OpenCost API documentation (/allocation), opencost.io
18. Video Generation API Pricing: Seconds, Pixels, Credits, Tokens, aireiter.com
19. Veo 3 pricing in 2026: every tier, API rate, and what AI video actually costs, cloudzero.com
20. balance – a Python package for balancing biased data samples (Sarig, Galili, Eilat; Meta, 2023), arxiv.org
21. Modeling Edge-to-Cloud Offloading Workloads for Autonomous Vehicles, arxiv.org
22. Process Visualization of Manufacturing Execution System (MES) Data, arxiv.org
23. OEE FAQ, oee.com
24. Downtime Data – Its Collection, Analysis, and Importance (Williams, Ford; WSC 1994), www-personal.umd.umich.edu