A renewal TCO model you can defend starts with your own traffic, not the vendor’s rate card. Measure the call volume, payload size and utilisation profile you actually run. Split the per-call price into compute, data transfer and the residual you can only call margin. Add the costs that never appear on the quote, then project all of it forward under the contract’s pricing structure. Per-call, reserved and on-prem then become one comparison read off that profile, because the break-even between serving options turns on utilisation, not capability.
What are we actually paying for in each call: compute, transfer or margin?
This model treats a per-call AI price as three parts: the GPU time that produced the response, the data transfer that carried the request and the answer, and the vendor’s margin and overhead on top. A defensible model therefore rebuilds the first two from measurement and treats margin as what is left.
LynxBenchAI publishes per-device results on the public PyTorch stack, which give the compute part of this breakdown a measured starting point: the first step is to pick the GPU class to benchmark from those results, then run your own representative traffic on it.
It then checks the resulting cost per call against your invoices, so finance gets a breakdown it can update and reuse in the next negotiation. To choose which GPU class to benchmark against, LynxBenchAI’s published results show how each device runs the public PyTorch stack your own models run on, which makes them a broadly indicative starting point. They do not measure your workload, so the representative run is still needed.
Worked example: splitting a production GPU bill into compute and egress
Bex (July 2026) priced a single always-on A100-class inference GPU (730 hours a month) serving 10 TB a month of response traffic on each of the three major hyperscalers. At that volume, egress came out at between 22.9% and 30.9% of the total bill across the three providers:
| Provider | Compute per month | Egress per month | Total | Egress share |
|---|---|---|---|---|
| AWS | $2,993 | $891 | $3,884 | 22.9% |
| Azure | $2,681 | $861 | $3,542 | 24.3% |
| GCP | $2,682 | $1,200 | $3,882 | 30.9% |
The same analysis puts egress at only 1-2% of the bill at low traffic, against 23-31% at the volume where a service is succeeding. Base egress is also not the whole transfer line. Egresscost.com (2026) lists AWS NAT Gateway processing at $0.045/GB and cross-AZ transfer at $0.01/GB each way stacked on top, so a 10 TB month routed through a NAT Gateway adds $450 in processing fees alone. Provider choice can remove the line entirely: Oracle dropped all outbound data transfer charges across its commercial regions in February 2026.
With compute and transfer measured, the margin estimate follows from arithmetic. The FinOps Foundation defines cost per API call as total API costs divided by the number of calls. Apply the same formula to your benchmark: (GPU-hours × hourly rate + transfer and surcharges) ÷ calls served at your latency target gives a self-run cost per call. Subtract that from the vendor’s per-call price and the remainder is margin plus whatever operational overhead the vendor carries for you. Bex’s scenario assumes a busy, always-on GPU, so treat its shares as a check on the structure of your bill, not a forecast of it.
Questions that make a vendor justify its per-call price
These questions turn the breakdown into negotiation material, because each one pins down a term the rate card leaves open:
- What exactly is the billable unit, and what is billed when the endpoint is idle?
- What is the cold-start time for our model at our size, measured rather than quoted?
- Is there a reduced rate for cached input tokens, and what invalidates the cache?
- Which egress, storage and network charges attach to our traffic, and on whose bill do they appear?
- Is pricing linear in volume, tiered or capped, and what is the overage rate above any committed quota?
- If we reserve capacity, is the reservation tied to one model or shared across models when that model is retired?
- Which latency and availability targets does the quoted price assume?
A refusal to disclose the real cost breakdown limits how precisely you can estimate margin, and the honest output is then a range rather than a single figure. The benchmark side carries its own fairness test: a dedicated fleet run at batch sizes that break the vendor’s latency or availability commitments is not a like-for-like comparison, and a vendor will rightly reject it.
Is per-call billing right for our workload, or should we push for reserved capacity?
Per-call billing suits variable demand. Reserved or flat-rate capacity only wins once you can keep it busy. On-demand inference is metered on consumption and an idle hour costs nothing, whereas reserved capacity bills by the hour whatever your traffic; Azure’s documentation says provisioned deployments are billed for the full deployed count “regardless of actual utilization”. That converts a variable cost into a fixed one, and fixed costs are only cheap when utilisation is high.
TianPan (2026) puts the break-even at around 60-80% sustained utilisation of the reserved capacity, with public estimates for a mainstream model such as GPT-4o needing roughly 150-200 million tokens a month before a provisioned deployment beats pay-as-you-go. Provisioned tiers commonly land 20-40% below on-demand at sustained load, so the discount is real. The trap is the load shape. A weekday business-hours product is idle roughly two-thirds of every week, and a reservation sized to the peak hour can average around 25% utilisation against an always-on meter. The same author’s three-gate test asks for scale and flatness, model stability and forecasting maturity, and says on-demand is the right answer if any gate fails. The volume thresholds depend on the model, so use them as a test to apply to your own traffic, not as constants.
That is why the input that settles the question is a representative week of real production traffic, quiet hours included, replayed against the candidate pricing models while you record the invoice rather than the benchmark. Scope the whole model to that profiled workload and a unit metric such as cost per call, not to total spend.
| Workload profile | Pricing to ask for | Why |
|---|---|---|
| Steady, high-volume traffic with a flat floor | Reserve the floor, burst on-demand above it | Reserved capacity wins above sustained-utilisation break-even; peaks spill to on-demand |
| Spiky, low-volume or experimental traffic | Per-call (token or request) billing | No idle cost; you are not paying for capacity your traffic does not use |
| Batch or pipeline work where cold starts reach no user | Scale-to-zero billing | The first-request delay is acceptable, so idle capacity need not be bought |
| Model likely to be deprecated within the term | Short term, or a model-independent reservation | Reserved token capacity is in practice a bet on a model with a 12-18 month life |
| Very high sustained volume or strict data residency | Price an on-prem option alongside | Carnegie Mellon’s analysis finds on-prem viable mainly at ≥50M tokens/month or under residency mandates |
Where the decision turns into where the workload should run (sovereign cloud, hyperscaler or your own racks), the companion piece on modelling cost per request to justify compute and placement choices covers placement in depth.
How will the vendor’s bill grow with our volumes, and does the pricing have a cap?
A vendor’s bill grows with the unit it meters and in the shape its pricing structure gives that unit. A forecast you can defend therefore models each volume driver separately and applies the contract’s structure (linear, tiered or capped) to each, rather than drawing today’s invoice out in a straight line. LynxBenchAI does not forecast bills, but when a growth scenario includes moving part of the load to dedicated hardware, its published per-device results give the performance side of that scenario a measured starting point.
Two facts from the sources show why straight-line extrapolation misleads in both directions. Hyperscaler egress is tiered and the per-GB rate falls as monthly volume rises, so a linear projection at entry rates overstates list egress. A commitment tier, by contrast, is a floor and not a cap: Microsoft’s documentation for its commitment plans says usage above the quota is charged at the tier’s overage rate. The metering unit matters, not just the volume. Estuary (2026) shows that identical data workloads produce very different bills depending on whether a vendor meters by GB, rows, events, credits or running infrastructure.
The FinOps Foundation’s forecasting capability describes its most mature (Run) stage as a mix of rolling, trend-based and driver-based models, with forecast variance measured as (forecast − actual) ÷ forecast. Trend-based forecasting reads your history. Driver-based forecasting starts from the quantity that produces cost, which its working group expresses as a compute metric × time × rate.
Growth in catalogue size and user base
When growth comes from more content and more users (more titles, more renditions per title, more regional caches, more simultaneous streams), compute and storage-plus-egress should get separate growth curves. They respond to different drivers: egress is billed per GB of data leaving the network, not per request served. The Bex figures above already show the effect inside one bill, with egress rising from 1-2% of cost at low traffic to 23-31% at production volume. If your storage and transfer curve is steeper than compute, the cost model is transfer-dominated, and that line deserves the optimisation effort and the hardest negotiation. Re-derive the curves whenever catalogue size, the bitrate ladder or user geography shifts materially, since each one moves storage and egress without touching compute. For projecting cost per request at a multiple of today’s users, including batching gains, the cost-per-request model goes further than this section.
Growth in fleet size and sensor count
When growth comes from more devices or richer sensors, forecast raw data volume from its drivers: units in the fleet × sensors per unit × bitrate per sensor × hours recorded. Apply the vendor’s stated structure to that volume, and the question “how much will the bill grow if we add vehicles or higher-resolution cameras” becomes arithmetic instead of guesswork.
Put your trajectory into the RFP, not just today’s volume, and require each vendor to state whether its pricing is linear in ingested data, tiered or capped, along with tier boundaries and overage rates. Estuary’s guidance for comparable vendor pricing is to price every meter on the same assumptions, add the lines that sit outside the quote, and test the result against half and double your estimates. A disclosed pricing structure is not a fixed one, though: Google raised its peering egress rates in May 2026. Pricing in this category also moves faster than annual budget cycles, so put a quarterly review with utilisation data attached in the calendar.
Which hidden costs get left out of an AI inference TCO analysis?
The costs that surprise people are the ones not on the rate card: idle capacity, egress and storage, and the engineering time to operate the system. Estuary notes that destination compute and engineering time can exceed the vendor’s own charge. CIOPages’ inference-serving buyer guide gives a three-year formula that makes the first group explicit:
TCO = (Requests × Tokens-per-Request × Published Rate × 36 months) + Idle Capacity Hours × Hourly Rate + Engineering FTEs to Operate + Egress and Storage − Cached-Prefix Savings − Committed-Spend Discount
Idle hours come from your traffic week, not from the autoscaler’s defaults, because whoever configured the autoscaler decided the idle capacity, not whoever signed the contract. Engineering FTEs should include upgrades, schema or model changes, recovery and on-call. Egress and storage come from the breakdown above. Cached-prefix savings depend on what invalidates the cache, and the committed-spend discount only counts if you would meet the commitment anyway.
Folding rework, false rejects and downtime into the per-unit figure
For an inference system that makes decisions on a production line (inspection, grading, sorting), compute and power per unit understate the true cost, because the model’s errors create work downstream. ASQ’s cost-of-quality framework names the categories: internal failure costs such as scrap, rework and failure analysis, and external failure costs for defects that reach the customer. A false reject maps onto rework or re-grading. A missed defect maps onto external failure, and inspection-system outages map onto line downtime.
The per-unit figure then becomes (hardware amortisation + power + maintenance + false rejects × re-grading cost + missed defects × failure cost + downtime hours × downtime cost) ÷ units inspected. Keep maintenance as a separate line in the capex case so it stays visible to whoever approves the spend, and amortise it per unit only in the side-by-side comparison. Two cautions apply. Sparse or inconsistent records weaken the hidden-cost estimate. Across several plants, differing defect taxonomies and reject thresholds make the figures incomparable until they are aligned. This is a financial model, and it does not show that a given accelerator or precision meets the model’s accuracy and latency needs.
What one inference costs on our own GPU server once power, maintenance and depreciation are in
The on-prem cost per inference is the fully loaded hourly cost of the server divided by the inferences it serves per hour at your latency target and real utilisation. Carnegie Mellon’s model (Pan et al., 2025) builds the hourly side from hardware cost (GPU count × unit price) plus monthly electricity (GPU count × power draw × operating hours × electricity rate). Its authors flag staffing and maintenance as extensions the model still needs. Lenovo’s 2026 TCO paper shows what those extra lines look like for one configuration: $9.80 an hour of operating cost, made up of $5.45 amortised maintenance, $2.27 power and cooling and $2.08 colocation.
The remaining lines in the full model are these:
- Depreciation: capex spread over the life you assume.
- Power and cooling: GPU draw scaled by your facility’s PUE (total facility power ÷ IT power). Assuming peak GPU operation throughout, the Heron study (2025) puts five-year power cost at 12.4% of an H100’s $30K purchase price at US enterprise rates, and 27% in Germany.
- Colocation and bandwidth: rack, power contract and transit. A dedicated box on flat-rate bandwidth removes the metered egress line, though the GPU’s fixed cost, power and operations time remain.
- Failure and replacement: spares, RMA turnaround and the capacity you hold back to cover them, carried as its own line.
Lenovo’s own arithmetic shows how the per-unit figure falls out: an amortised $18.88 an hour at 32,955 tokens a second gives about $0.159 per million tokens for its eight-GPU H200 configuration. That figure assumes throughput parity with the cloud instance it is compared against. Lenovo also sells the hardware, so take the line-item structure from it and leave its headline ratios aside.
At what volume does on-prem become cheaper than the vendor’s per-call price?
On-prem becomes cheaper at the point where cumulative local cost (capex plus monthly operating cost × months) equals cumulative API cost for the same output; Pan et al. solve for that break-even month directly. Across 54 scenarios, their results put break-even within a few months for small models, around two years for medium ones and about five years for large ones, and they find on-prem viable mainly at 50 million tokens a month or more, or under strict data-residency mandates. Against aggressively priced providers, large-model break-even stretches to five to nine years. Lenovo’s worked case reaches break-even in about 5.2 months against on-demand cloud and about 8.5 months against a one-year reservation. The gap between those results is the point: break-even depends on which commercial baseline you compare against, how long you amortise and how busy the hardware stays. Before committing capex, validate the utilisation assumption on representative traffic, since Lenovo’s break-even holds only for high-utilisation environments.
Will the renewed tier still pencil out if we retrain quarterly or add a scoring model?
A tier pencils out only if it holds under the scenarios you expect over the full term: retraining compute at your planned cadence, higher inference volume and any new models, priced against each tier’s rates and contract headroom. Steady-state usage today is not enough. TechnoLynx’s Inference Cost-Cut Pack audits the cost and latency of the current serving path before those scenarios are priced.
Per-decision cost moves with two independent drivers. Retraining cadence adds training compute, which Fireworks AI bands by model size rather than per job. Volume growth adds inference calls, and what it does to unit cost depends on whether the tier is linear, tiered or capped. Contract terms decide the rest. Commitment plans on Microsoft’s platform run by calendar month, charge overage above quota and can be ended by the last day of a month. Reserved capacity tied to one model carries deprecation risk: TianPan (2026) gives a typical model life of 12 to 18 months, with notice periods often only a few weeks. The same analysis notes that Azure’s reservations are model-independent across supported models while Bedrock’s provisioned throughput is model-specific, and advises matching term length to model life rather than to the discount curve.
Scenario procedure for the renewal
- Price the current tier at today’s measured volume and cost per request.
- Add the volume your product plan assumes, then repeat at half and double that estimate.
- Add retraining compute at the planned cadence, on the meter it will actually be billed on.
- Add each new model you expect during the term, and check whether the tier’s capacity and model-count terms allow it.
- Choose the tier whose worst-case cost and model count stay within budget and contract scope for the whole term.
Treat any retraining scenario whose timing or scope is uncertain as a range. Renegotiation triggers may also sit in clauses that only legal review will surface, whatever the cost model says.
Tune the serving path before pricing a replacement
Suppose the renewal model shows the current deployment costing more per request than the benchmark. That gap is not automatically a reason to switch vendor, tier or model. Production AI cost and latency can often be improved without replacing the model, because the main levers (batching, caching, routing, quantisation, kernel work, serving configuration and runtime choice) sit below the model layer and can be tuned independently. Replacing the model is the most expensive lever, and it frequently leaves the workload economics unchanged. Profile and optimise the serving path before replacing anything. Profiling the serving path is part of TechnoLynx’s GPU performance engineering work.
Who can build our renewal cost model, and should we do it in-house?
TechnoLynx audits the serving path for cost and latency through its Inference Cost-Cut Pack before infrastructure replacement is recommended. Whether to build in-house comes down to whether your team can profile production traffic and run a fair, SLA-matched benchmark. If it can, the method above is yours to apply. If it cannot, or the renewal date is close, that work is what the pack covers. When choosing any partner, ask whether they measure your traffic or quote rate cards, and whether their benchmark honours your latency and availability targets.
The engagement’s unit of work is the serving path and its cost per request. A team with a mature FinOps practice and its own profiling capability may need only the FinOps Foundation’s unit-economics and forecasting frameworks, which already link unit metrics to vendor selection, renewals and commercial model choices. Legal review of renewal and exit clauses remains a separate track in every case.
FAQ
Which billing model fits an expensive workload that can be started and stopped with little notice?
Where a cold start never reaches a person, as in most batch and pipeline work, billing that scales to zero avoids paying for idle capacity, and you accept the first-request delay. Always-on or reserved capacity only makes sense once the workload has a steady floor you can keep busy.
How do you reduce inference costs without changing the model?
Profile the deployed serving path first. Batching, caching, routing, quantisation, kernel work, serving configuration and runtime choice can each be tuned without replacing the model.
