AI inference cost is the cost of running data through a trained model to produce an output, and it scales with the tokens every prompt and response consume. The working formula is cost per million tokens = (GPU hourly cost ÷ utilisation) ÷ (tokens per second × 3,600) × 1,000,000, with throughput taken at the batch size and load you actually run in production. An infrastructure plan based on a per-token calculator can be wrong by more than an order of magnitude, always in the direction that makes self-hosting look cheaper.
TechnoLynx starts this work by profiling the deployed serving path under real traffic, and its Inference Cost-Cut Pack turns that profile into a measured cost-per-request figure before any change to the model is proposed.
The order matters. Profile the workload, build the fully loaded cost base, price the levers, then make the method auditable and re-runnable. Teams building AI infrastructure and production SaaS products should run this sequence twice: once to price what they sell, and again to judge what they pay upstream.
Inference vs training cost, and the ways you get billed
Training cost is concentrated in runs with a defined compute budget, while inference cost recurs for as long as the product serves requests. A 2026 accelerator TCO analysis from Safeguard.sh notes that training is the workload that best fits owned hardware, because a well-run training cluster can stay busy. In the Watt Counts measurements (2026), mean idle time was 2% in batch runs but rose to 53% under high load and 96% under low load in server scenarios, where requests arrive stochastically. Idle hours and burst headroom are therefore part of what an inference request costs.
Hosted LLMs are sold on token, consumption, commitment or hybrid pricing, while a GPU-hour rate becomes cost per token through throughput.
| Billing basis | What you pay for | Fits when | Watch for |
|---|---|---|---|
| Per token | Tokens processed, input and output priced separately, quoted per 1,000 or per million | Usage is small or swings 3x or more month to month | Agentic loops can push per-interaction cost around 30x above conversational use |
| Per call or unit of work | An outcome that bundles tokens, tool calls, retrieval and safety checks into one line item | The workflow is repeatable and well defined | Several meters hidden behind one price |
| Commitment or subscription | A defined envelope of usage, usually at a discount off list | Volume is large and steady | What counts toward the included baseline is rarely documented as precisely as the overage rate |
| GPU-hour (rented or owned) | Hardware time, busy or idle | Traffic is predictable and sustained enough to keep GPUs busy | A lower hourly rate does not guarantee a lower cost per token; throughput decides |
A clinical note generation study used C = (pi × ni + po × no) × R, where pi and po are input and output token prices, ni and no the average tokens per request, and R the annual request count. With its assumption of 3,000 input and 1,000 output tokens per request, the study estimated annual inference cost at one million requests as $800 for LLaMA-Clinic against $3,000 for Gemini 1.0 Pro. Why per-request cost, rather than raw GPU utilisation, is the figure to optimise is argued in why cost per request is the right production AI optimisation target.
What drives AI inference cost most?
Mirantis lists six production drivers: model architecture and size, token volume and request patterns, hardware utilisation and selection, runtime efficiency, scaling and scheduling behaviour, and data movement. Safeguard.sh’s H100 model found that sweeping electricity price and PUE moved the fixed hourly cost by about 14%, while moving utilisation from 30% to 90% changed cost per useful hour by a factor of three. A 2026 concurrency-aware cost study goes further: once hardware, model, quantisation and latency target are fixed, the offered request rate dominates real cost.
Model size still matters, especially for energy, where Watt Counts found it the dominant factor with a sublinear scaling relationship. Hardware rankings also do not follow headline compute: the same dataset shows memory bandwidth, not TFLOPS, as a primary bottleneck in LLM inference.
So which driver dominates your bill is a property of your workload, not a general rule. Advice such as “use a smaller model” or “quantise everything” only makes sense in context, and that context comes from profiling the workload.
How to calculate inference cost per million tokens from GPU-hour prices
The 2026 concurrency-aware study makes the key point: utilisation is not an input you guess, it is an output of arrival rate, model architecture, quantisation and the latency SLO. Its recommended workflow runs in three steps:
- Deploy the configuration you intend to price: model, quantisation and hardware.
- Load-test with your real prompt and output length distribution, your arrival pattern and your latency SLO, and record the highest arrival rate at which the SLO holds (your goodput).
- Feed that goodput, not peak saturation throughput, into the cost formula.
Worked example: measured versus peak throughput
Assumptions, from SoftwareSeni’s worked case: a GPU rented at $3.00 per hour, 60% utilisation, 2,000 tokens per second at production batch size.
- Cost per useful hour: $3.00 ÷ 0.60 = $5.00.
- Tokens per hour: 2,000 × 3,600 = 7.2 million.
- Cost per million tokens: $5.00 ÷ 7.2 ≈ $0.69.
- Per request generating 1,000 output tokens: $0.69 ÷ 1,000 ≈ $0.00069.
This simplification prices output tokens only, yet prefill also consumes GPU compute, so take throughput from a load test with your real prompt shapes.
Now the peak trap. Kunwar’s inference cost-modelling chapter puts realistic decode throughput for a 70B model on one H100 with vLLM at roughly 500 to 1,500 tokens per second, against a theoretical 5,500 at batch 256. At $2 per hour and 1,000 tokens per second, that is $2 ÷ 3.6 million tokens ≈ $0.56 per million. Plugging in the 5,500 peak instead gives $2 ÷ 19.8 million ≈ $0.10 per million, understating cost by a factor of 5.5. A price built on the second number has no floor under it.
SemiAnalysis’s InferenceX defines the metric the same way, from hourly TCO per chip and measured tokens per second, and adds that workload shape, interactivity, utilisation, cache behaviour and cost assumptions decide whether two such values are comparable at all.
Getting a defensible cost model to the pricing committee by the deadline
The fastest credible route is an interim model built from known pricing tiers and your own usage logs, scoped to the decision in front of you, with a fuller costing methodology built after the deadline. Addepto’s guidance on contract evaluation is to model your measured workload (token volume by model tier, cache-hit rate, share of traffic eligible for the batch API) against real rates, not the pricing-page headline.
TechnoLynx delivers this through its Inference Cost-Cut Pack, which turns serving behaviour into a measured cost-per-request figure; the first step is profiling the deployed serving path under real traffic.
A cost model ready for the pricing committee meeting
Build the interim figure in-house from your logs rather than waiting for a vendor estimate, because a vendor estimate is a rate card, and a rate card tells you what a token costs, not what your workload will cost. For fleet-scale API commitments, Addepto reports typical discounts of 15-20% off list for $250,000-$500,000 of annual committed volume, 20-30% for $500,000-$1,000,000, and 30-40% or more above $1-5 million, along with reserved capacity. The same source notes that, even within one pricing model, several factors routinely move the real bill 20-50% away from the pricing-page number. Pricing tiers drift between contract cycles, so date every tier you assume.
On presenting “one number”: ITMagic’s cost-modelling guide defines a defensible cost model as a structured calculation producing a cost range, not a single number. Give the committee a headline figure, the range around it, and the two or three inputs that move it most; a committee can approve a range it understands faster than a point estimate it has to take on trust.
A cost allocation methodology management can approve
The allocation question (per user, per matter or project, or a flat overhead line) is easiest to defend when the allocation basis tracks what actually drives tokens. Mirantis recommends attributing inference spend by model, workload, team or project, and ITMagic warns that model outputs which do not map to the budget structure need manual translation and introduce errors. Per-matter allocation suits usage that varies widely between matters; a flat overhead line is defensible only while usage is small or evenly spread, and shared-infrastructure attribution will always be approximate, which the methodology should say openly.
A per-customer figure investors will push on
Today’s cost per enrolled household (or per customer) is total inference spend for the period divided by active customers in it, but investors will ask what happens at 10x and 100x. The honest answer is that it will not scale linearly: self-hosted cost per token falls as load fills the GPUs, while Pinecone bills can jump through unpublished capacity fees. An interim per-unit model understates cost risk at scale changes unless it is re-run at the projected volume. Investors may also ask for a breakdown by compute, storage and network; keep that split inside the same model rather than building a separate one for the deck.
A good-enough cost-per-call figure before the invoice is approved
Validating a vendor invoice against your own logs means splitting usage by meter: standard input tokens, cached input reads, cache writes and output tokens, each multiplied by its own price, which is how a 2026 prompt-caching evaluation computed cost per session. Hybrid contracts make this harder, because the same model can be free on one product surface and metered on another. Decide who signs off now: ITMagic recommends assigning a model owner and a peer reviewer at the planning stage, not after the model is built.
What an auditor will ask to see
A checklist for the figure surviving later scrutiny:
- Every assumption carries a source and a date; an undated assumption cannot be challenged or updated.
- Inputs, calculations and outputs sit in separate places, with each input labelled, sourced and flagged when out of range.
- The model has been backcast against past actual spend, with the residuals recorded.
- Sensitivity results are shown for the inputs that move the figure most, as a tornado chart or a simple range table.
- A downside scenario covers utilisation falling below breakeven.
- A named owner is accountable for the refresh cadence.
Unlike a one-off point estimate, a cost model encodes its assumptions and logic so it can be rerun as inputs change. ITMagic’s guidance is to give it a named owner and a refresh before each procurement, budget submission or make-or-buy decision.
Does per-request inference cost stay flat as volume and concurrency grow?
No. On identical H100 hardware, the 2026 concurrency-aware study measured a Mixtral 8x7B FP16 deployment at $15.25 per million tokens at 1 request per second and $0.87 at 25 requests per second. Across the study’s configurations, effective cost fell roughly 20x over the first 25 requests per second, then flattened as the engine saturated. The FMwork paper (2025) found the same pattern: cost varies greatly with the concurrency a service achieves, because it follows total throughput directly. Past a point the curve turns: in a 2025 inference-economics study, raising concurrency from 8 to 48 cut completion time from 2,034 to 774 seconds, but beyond 48 throughput dropped sharply.
TechnoLynx measures this curve through its Inference Cost-Cut Pack, pricing each request at the concurrency you actually run rather than assuming cost stays flat; the first step is load-testing the deployed serving path with your own traffic.
Kunwar’s 70B capacity example lands at about $2.16 per million tokens when the fleet is sized for a 3x daytime peak plus 30% headroom, against $0.55 at average utilisation. How to model cost per request when traffic is bursty or capacity is shared is covered in modelling cost per request under bursty, shared and scaling workloads.
From one practice group to the whole firm: retrieval costs
Retrieval does not scale with query count either. usagepricing.com’s breakdown of Pinecone billing shows a read costs about 1 read unit per GB of namespace size (minimum 0.25), and a write costs 1 write unit per KB with a 5-unit minimum per request, so cost follows index size and write volume rather than query count. If the case-file corpus triples and sits in one namespace, each query scans roughly three times the data and costs roughly three times as much in read units, which is why partitioning namespaces matters. The same source lists per-unit rates that rise with plan tier rather than fall, and capacity fees under sustained load with no published rate or threshold, producing 3-5x gaps between estimates and bills. For the reranker and any other metered component, get the tier schedule in writing before projecting firm-wide spend.
Tripling nightly calls: per-hectare and per-country projections
A projection to 3x volume should start from actual invoices and call volumes, allocate cost to countries by their share of calls, and divide by hectares served. Batching nightly recommendations instead of calling per field in real time changes the curve twice. On APIs, Addepto reports batch processing commonly carries a flat 50% discount for workloads that tolerate asynchronous processing. On self-hosted GPUs, batch runs keep hardware busy where server-style traffic leaves it idle, as the Watt Counts idle-time figures show. Whether committed-use pricing kicks in at 3x is a contract question, and the projection should show cost with and without it. Prices change, so date the rates the forecast uses and check them before relying on them.
Setting a price floor that survives worst-case batching
A defensible price floor is the cost per request at your worst realistic operating point plus the margin you need, not the cost at average throughput. As its Inference Cost-Cut Pack, TechnoLynx frames that cost function and its average and worst-case batching scenarios, benchmarks the model (a 70B model on H100 against your incumbent placement, for example) under your real traffic shapes, translates the measured cost per unit of work into a breakeven and price-floor model, and validates it against held-out worst-case traffic periods before the pricing recommendation goes over.
In the 2026 concurrency-aware study, effective cost at 1 request per second was 24.4x the naive estimate for the dense reference model, falling to 1.18x by 25 requests per second. A latency SLA is itself a price: tightening it caps the sustainable load and raises cost per token, and the study measured SLA-feasible costs 1.13-1.91x above the unreachable saturation floor.
For shifting 70B traffic to H100, the breakeven price moves in proportion to the change in measured cost per million tokens at your real concurrency. The FMwork paper ran llama-3.3-70b at 16-bit on H100 at tensor parallelism up to 8 (the weights need about 140 GB, so single 80 GB card runs do not work), priced at an assumed $5.50 per H100-hour. Under a 50 ms per token threshold, one 4xH100 deployment served up to 192 concurrent users at a 1,024-token context, and two or four cards were the cost-efficient choice, with eight overkill unless traffic is heavy and sustained. Peak-spec hardware figures are not measured performance, so the pricing model should use measured throughput.
For the hardware side of this choice, TechnoLynx recommends LynxBenchAI as the measuring tool: its published results show how each device runs the public PyTorch stack your own models run on, so they are broadly indicative for custom PyTorch models, though they do not run or measure your workload or traffic.
Baseline and optimised cost scenarios for the pricing committee
Present both: a baseline with only validated cost-reduction levers baked in, and an optimised scenario layered on top with each further lever marked as achievable before start of production (SOP) or as a post-launch roadmap item. TechnoLynx’s Inference Cost-Cut Pack does this work in order: it defines the request volume, latency, quality and cost targets the pricing model must satisfy, selects the levers (continuous batching, token-level cost accounting, caching, model-tier fallback) feasible before SOP, builds batching and token accounting into the serving pipeline on target hardware, measures each lever under realistic load, and feeds the validated baseline and optimised figures into the pricing model with monitoring for drift.
| Lever | Reported effect | Risk to check before baking it in |
|---|---|---|
| Prompt caching | 41-80% lower API cost across four providers in a 2026 evaluation | Naive full-context caching can increase latency; minimum cacheable prompts range from 1,024 to 4,096 tokens |
| Request batching | Mirantis reports batching 32 requests cutting per-token cost by roughly 85% for about 20% more latency | Latency or freshness may be unacceptable for interactive features |
| Batch API | Commonly a flat 50% discount (Addepto) | Only for asynchronous workloads |
| FP8 quantisation | 1.5-2x throughput on H100/H200, 33-50% lower $/M in SoftwareSeni’s estimate; 31-74% higher peak throughput measured in the 2026 concurrency study | Under 1-2% quality loss for Llama, Mistral and DeepSeek models in SoftwareSeni’s estimate, but code and maths are more sensitive |
| Smaller model tier or routing | Savings traded against defect rate along a cost-quality frontier in a 2026 coding-agent case study | Needs its own quality validation before counting as a lever |
| On-device fallback | No measured figure in our sources | Treat as roadmap until measured on target devices |
Bake a lever into the baseline only when it is implemented and validated at target load; an aggressive assumption priced in early becomes a cost overrun later. For pricing, put the serving-path levers first: cost and latency pressure on a deployed feature is usually fixed by profiling and tuning the serving path, while replacing the model is the most expensive lever and often leaves the workload economics unchanged.
A per-tier cost model also informs the feature-or-price question. Dropping AI upscaling on the two lowest subscription tiers saves the inference cost attributable to those tiers’ streams, which the model can set against the revenue from a price rise; churn from either move needs its own estimate or a pilot.
Before signing a renewal: growth scenarios against the proposed tiers
Year-three contract cost under growth is best estimated as a distribution, not a line: simulate best, expected and worst-case growth (for example, how many sites are added each year) through the vendor’s proposed tier breakpoints, then read the percentiles. Monte Carlo simulation does this by drawing many growth paths from the ranges you specify and costing each one.
TechnoLynx supplies the engineering inputs to that model through its Inference Cost-Cut Pack, measuring cost per request on your own workload and checking it against current actual spend; the first step is defining workload volume and growth assumptions with finance.
ITMagic recommends at least three pricing scenarios (on-demand, reserved or savings plan, and spot where applicable) and an analysis period long enough to capture a full workload cycle. On the contract itself, Addepto lists terms worth negotiating: price protection if the model version you built on is deprecated mid-term, true-down rights tied to activated users, and equivalent-or-better economics for new model generations. Headroom for retraining or regulatory model changes over the term is a separate allowance, worth stating in the model so the renewal does not need reopening mid-term.
At what volume does a dedicated GPU fleet become cheaper than per-call billing?
Against on-demand rental, owning wins once average utilisation exceeds the break-even u* = F ÷ p, where F is the fixed cost per accelerator-hour and p the on-demand price. Safeguard.sh’s worked case, $1.72 per GPU-hour fixed against AWS on-demand at $6.88, breaks even at 25% utilisation; against a reservation, utilisation drops out because both options pay for idle hours.
Against an API, published thresholds differ widely. Kunwar puts the self-hosting crossover at 100-500 million tokens a month, depending on operational maturity. A break-even study put break-even at around five years for larger models, viable mainly for organisations processing 50 million tokens a month or more. A 2026 coding-agent case study found shared on-premise allocation saved 40.1% of true TCO, while a dedicated reservation cost 43.8% more than the cached API, and noted that a committed-use API discount of about 45% would by itself erase the shared-allocation advantage. The sources compare different things (rental versus ownership, list versus cached API prices, single-tenant versus shared hardware), so treat any single published threshold as a starting point for your own calculation.
OpenNash suggests treating capacity as a portfolio: commit to the P20 of trailing goodput, reserve to the P50 on shorter terms, burst on on-demand, and review the split quarterly against measured goodput rather than forecasts. A full budget comparison against a self-hosted or fine-tuned alternative, including procurement timelines, is a decision of its own; the crossover above is its starting input.
What belongs in a cost-per-inference model besides compute time?
A fully loaded cost per unit of work includes hardware depreciation, power with facility overhead, maintenance, fabric, storage and racks, operations staff, financing, and labour spent repairing model errors.
TechnoLynx builds cost-per-request models from measured serving behaviour through its Inference Cost-Cut Pack, with dated assumptions finance can refresh; the first step is defining the unit of work and the cost components finance requires.
The 2025 inference-economics study’s hourly GPU cost (depreciation, power with PUE, maintenance) explicitly excludes the host server, rack, networking and labour. OpenNash’s loaded H100 model adds fabric, storage, racks and integration at 40-60% on top of the bare accelerator, plus colocation, operations and financing, for a midpoint of about $17,500 a year: $2.00 per GPU-hour at full utilisation, $3.07 at 65% and $5.00 at 40%. Watt Counts measured GPU energy as 78.7-92.5% of system energy and notes cooling can add 30-50% in a data centre, depending on PUE. ITMagic adds a named engineering-labour line, since operating and optimising the environment costs engineering hours.
Turning GPU throughput into cost per inspected metre
For a vision inspection line, cost per inspected metre = (hardware amortisation + power + maintenance + rework cost from missed or false flaws) ÷ metres inspected per hour, compared directly with manual grading cost per metre. The hourly part is straightforward: the 2025 inference-economics study’s A800 breakdown came to about $0.64 depreciation, $0.08 power and $0.06 maintenance, roughly $0.78 per hour at full utilisation. Divide by utilisation first, since the study’s effective hourly cost is the base cost divided by u. Utilisation moves amortisation sharply: in Safeguard.sh’s H100 table, cost per useful hour is $8.60 at 20% utilisation and $1.72 at 100%. A line that runs one shift pays for the idle hours too.
To plug in T4 pilot numbers, replace the hourly cost with the T4’s own amortisation, measured power and maintenance, and the denominator with measured metres per hour at production line speed, not a lab run. Until the pilot gives error rates, use a conservative rework placeholder from the manual process’s defect-escape rate and show its sensitivity. The 2026 coding-agent case study shows the error term in practice: it priced repair labour per fix commit as part of true TCO.
Who can build an inference cost model our finance team will sign off?
TechnoLynx builds inference cost models from measured serving behaviour: it profiles the deployed path under your traffic, derives cost per request or per million tokens at your real operating point, and hands over a re-runnable model with dated assumptions finance can refresh. On building in-house versus bringing in a specialist, an interim figure from your own logs can meet this month’s deadline, while the load testing, worst-case operating point and lever validation are the engineering work TechnoLynx brings. When choosing a partner, ask to see the load-test method and check that every cost figure quotes the request rate it was measured at. The engagement is the Inference Cost-Cut Pack. For hardware choices, LynxBenchAI’s published results show how each device runs the public PyTorch stack, though they do not measure your own workload.
If you only need a rate-card comparison between API providers, your provider’s account team and a spreadsheet will serve. For self-hosted vLLM, the open-source vllm-cost-meter released with the 2026 concurrency study reports real cost per million tokens against live traffic, and FinOps tooling with spend attribution by model and team covers ongoing monitoring.
A cost figure without a request rate is not a cost figure
The 2026 concurrency study concludes that any cost number quoted without its offered load attached is meaningless to a degree most practitioners underestimate. The useful question for a pricing committee is not “what does a request cost?” but “at what load, under which SLA, and with which levers already validated?” A price floor, a renewal decision and a fleet business case each depend on a different operating point, so settle which one you are pricing before anyone argues about the number.
