You don’t know what one request costs on self-hosted hardware until you know what load that hardware actually serves. On identical H100 hardware, effective cost spanned $0.21 to $15.25 per million output tokens across configurations and load levels, according to a 2026 concurrency-aware cost study. So the model has four parts: fixed cost spread over measured utilisation, an operations floor, a workflow multiplier on raw token volume, and a breakeven checked against your own traffic. A cost-per-request figure quoted without the request rate it was measured at is meaningless.
This applies to any team running inference as part of AI infrastructure and production SaaS, and it is the same discipline we apply to performance work in general: profile the real system first, because a rule of thumb only means something in the context profiling gives it.
How do you model cost per request for a hosted API versus self-hosting?
Cost per request for a hosted API is roughly linear: tokens times price, so it scales directly with volume. Self-hosting is a roughly fixed monthly cost whatever the usage, so its cost per request is that fixed cost divided by the requests you actually serve. TechnoLynx models cost per request for inference workloads through the Inference Cost-Cut Pack. The work starts with an audit of the workload you actually run.
The self-hosted side needs these buckets, the first three from one TCO guide for LLM inference:
- Serving: GPU amortisation or rental, power and hosting, divided by realistic utilisation rather than theoretical throughput.
- Operations floor: monitoring, upgrades, on-call and capacity buffer, which one 2026 guide puts at typically $1,500 to $2,500 a month for one production stack.
- Workflow multiplier: retries, RAG overhead and agent loops, which the same guide says multiply raw token volume by 2 to 50 times.
- One-off setup: this goes into a payback calculation, not the monthly operating line.
The payback arithmetic is simple. Monthly saving equals the comparable API operating cost minus the self-hosted operating cost, and setup payback in months equals one-off setup cost divided by that saving. If the saving is zero or negative, there is no payback. We count fine-tuning as one-off setup cost here. It is a real cost, and it only pays back if the monthly saving is positive.
Adding people cost honestly shifts self-hosting economics meaningfully. One 2026 breakdown estimates ongoing operations at 10 to 20 percent of one senior engineer, permanently, for model updates, serving upgrades, incident response and quality regression tracking. Another suggests multiplying raw GPU cost by 1.3 to 2.0 to get the true cost of self-hosting. A vendor-modelled sweep of 900 scenarios found people cost more than GPUs at small scale: adding a $1,500 monthly operations floor cut the share of scenarios where self-hosting beat one API tier from 84.4 percent to 67.3 percent.
If you want the measurement step on its own, before any modelling, we cover it in how to measure inference cost per request before any optimisation.
Worked example: the breakeven formula with staffing included
This example uses one published set of inputs, stated as assumptions. A single reserved H100 at $3.99 an hour for 720 hours is $2,873 a month. Electricity adds about $64, for a floor of about $2,937 before staffing. A quarter of one developer’s time at the US median salary adds roughly $2,833 a month.
| Step | Working | Result |
|---|---|---|
| Self-hosted monthly cost, hardware only | $2,873 + $64 | about $2,937 |
| Add 0.25 FTE | $2,937 + $2,833 | about $5,770 |
| Breakeven against an API at $0.375 per million tokens, hardware only | $2,937 ÷ ($0.375 × 30 days) per million | about 261M tokens/day |
| Same, with staffing | $5,770 ÷ ($0.375 × 30) per million | about 513M tokens/day |
The rule behind it: if monthly self-hosted TCO is below daily tokens × API rate × 30, you are past breakeven, and the daily token figure should come from your actual API usage logs, not estimates. In this example, staffing alone roughly doubles the volume you need.
At what daily query volume does hosting your own fine-tuned model become cheaper than the API?
There is no single threshold. The published estimates range from about 10 million to hundreds of millions of tokens a day. One 2026 analysis puts breakeven against Claude Sonnet 5 at around 10 to 15 million tokens a day on a Hetzner GEX131, if a 27B to 70B-class open model meets the quality bar. A vendor-modelled sweep of 900 scenarios gives a median of 11 to 12 million tokens a day against mid-tier APIs. Another source argues realistic breakevens often land in the hundreds of millions of tokens a day once idle time, staffing and compliance overhead are counted. A fourth says self-hosting starts to win above 100 million tokens a day with comparable quality, even after hidden costs.
The spread comes from three inputs: the API’s price per million tokens, the GPU configuration the model needs, and how well you use that hardware. Against Gemini Flash pricing, one estimate found a single A100 would have to run at maximum utilisation around the clock just to break even. The concurrency-aware study argues the crossover is a surface across five variables, not a single point.
Volume alone also doesn’t settle it. The open model has to meet your quality bar on your own prompts, and one guide recommends scoring your top 200 prompts through both candidates before deciding. A routing layer that sends simple, high-volume requests to a small self-hosted model and the rest to an API can bring the blended cost below the pure self-hosting breakeven. That changes the question from “which one” to “which share of traffic”.
Utilisation, request mix and duty cycle: the assumptions to check
The concurrency-aware study argues that utilisation is not an input you choose. It is an output of arrival rate, model architecture, quantisation and SLO, and calculators that take it as a field have the causal direction backwards. These are the assumptions we would test first, and how to check each one:
- Duty cycle. A GPU at rest costs the same as one at full load, and below about 60 percent utilisation the per-token economics look worse than the brochure suggests. At 50 percent utilisation the effective cost per token roughly doubles, and at 25 percent it quadruples.
- Request mix. Request size changes which GPU is cheapest. An A10G gave up to 2.6 times the tokens per dollar of an A100 for small requests, while the A100 reached up to 1.5 times the efficiency on large ones. In one measured sweep, RAG-style long-prompt traffic cost about 2.4 to 2.7 times the chat baseline at 25 requests per second, but less than chat at 1. Check it with a histogram of input and output lengths from logs.
- Latency SLO. Advertised cost floors often sit at loads no interactive product could ship. In one measurement, the cheapest point for an 8B model came with a 7.4-second p99 time to first token, and the cheapest cost within the SLA was higher. Check it by finding the highest load that still meets your SLO, and take cost from that point.
- Concurrency. Peak concurrent users decide GPU count more than total token volume does. Check peak concurrency, not daily totals.
- Burst headroom. In one serving study, 84 percent of SLO violations came from request bursts or clusters of large requests, and the authors suggest over-provisioning (for example 10 percent) to absorb them. That headroom is idle cost, and it belongs in the model.
One guide’s validation step is worth adopting as written: prove the unit economics with 30 days of real traffic before scaling.
What happens to the self-hosting breakeven if query volume doubles next quarter?
Doubling volume doesn’t move cost cleanly, because the cost curve is a steep drop followed by a plateau. On H100, effective cost dropped by roughly 20 times over the first 25 requests per second, then flattened as the engine saturated. The underutilisation penalty fell from 24.4 times the naive estimate at 1 request per second to 1.18 times at 25. Three cases follow from that curve:
| Where you are now | What doubling does |
|---|---|
| Low load, steep part of the curve | Cost per request falls sharply, and self-hosting can move from losing to winning. For Mixtral the crossover against lower-priced API tiers sat near 1.5 to 2 requests per second. |
| Mid load, flattening part | The penalty over the naive estimate is already down to 1.18 times at 25 requests per second. |
| Near saturation | Throughput plateaued at about 6,230 tokens per second from 100 requests per second up, while median time to first token rose into seconds. |
Scaling with only large GPUs often leaves resources underutilised, and mixing GPU types allows finer-grained scaling. Against cheap API tiers, the breakeven volume can exceed what one GPU can serve at all. A defensible forecast therefore states which part of the curve you are on today, and models the doubled volume as a new operating point on a measured curve rather than as a multiplier on today’s figure.
DGX Station capex versus shared-partition allocation
Public US listings for the GB300 DGX Station started at $92,887.47, most sat between $96,000 and $130,000, Dell listed one at $159,283, and lead times ran from about two to sixteen weeks. A European reseller gave a four-to-sixteen-week lead time depending on vendor and expected the price to rise to about $145,000 in 2027 on memory costs. Listed figures are base configurations: storage, an add-on GPU, warranty and on-site service cost extra. The system is rated at 1,600 W, and drawing the full figure needs a 115 to 240 V circuit.
Worked example: when the DGX Station capex pays back
Spread over 36 months, the cheapest listing is about $2,580 a month before power, storage or support. At $6.69 an hour for an on-demand B200, $92,887.47 buys about 13,900 GPU-hours, roughly nineteen months of never switching the machine off. That figure assumes 100 percent duty. This is why one analyst concludes ownership rarely wins below roughly 60 percent duty cycle. The comparison also sets owned desktop hardware against rented data-centre silicon, which is not like for like.
Shared allocations price idle time differently. At the University of Wyoming’s ARCC, an exclusive node reservation is a $100 base cost plus hours reserved, charged even if the reserved hours go unused. The San Diego Supercomputer Center’s condo model lists an 8×L40S node at $74,000, plus an annual GPU operations fee of $4,668 to $7,468 a node, over a nominal five-year term. Unused allocation expires at year end. Its hotel model is pay-as-you-go for bursty needs, with minimum purchases of $352 or $563 for 10,000 service units depending on affiliation.
Renting has its own trap. One 2026 analysis argues that with rental prices compressing, breakeven utilisation for owning an H100 against a committed rate near $2.10 an hour sits around 90 to 95 percent. That compares owning against a committed rental rate, while the 60 percent figure compares owning against on-demand rental. Both depend on a utilisation figure that should be measured, not taken from a slide. For the full renewal-time comparison, see TCO modelling for GPU infrastructure against API pricing.
Who can build a cost-per-request model we can defend to procurement?
TechnoLynx builds cost-per-request models for inference through the Inference Cost-Cut Pack, tying each figure to measured load: the request rate, request mix, SLO and duty cycle of your actual traffic, with breakeven against the API, rented and owned options you are weighing. Where the open question is which hardware to buy, LynxBenchAI publishes measured results across AI accelerators to inform that choice. It measures the hardware, not your workload.
The Inference Cost-Cut Pack covers the engineering, not a legal determination on data residency. That belongs with counsel or your compliance function, and the model takes their answer as an input. One guide’s rule: if your p95 volume is under 10M tokens a day against Sonnet 5, or under 25M against Haiku 4.5, the math almost certainly doesn’t work.
How sovereignty requirements change the comparison
Several of the sources treat hard residency requirements as a reason to self-host whatever the volume: one names residency that no API can satisfy as a standalone case for self-hosting, and a 2025 cost-benefit study notes that long break-even periods can still be accepted when privacy, sovereignty or lock-in concerns override cost. In our view the honest structure is to remove ineligible options first, then compare cost only among the options that remain, and to label any premium paid for sovereignty as exactly that, so nobody later mistakes it for an efficiency loss.
Price sovereign-cloud GPU capacity from the tender in front of you, using the same utilisation-adjusted method. On the evidence side, we would have a sovereign-cloud cost case record where the control plane is hosted, the full sub-processor chain and who can access inference logs. We would keep assembling that evidence into an audit-ready pack separate from the cost model. The model should cite the pack, not reproduce it.
Documenting the model so procurement and compliance can audit it
A reviewer should be able to change an input and see whether the conclusion survives. Make that easy. This checklist is what we would expect in the document itself:
- Measured load. Monthly input and output tokens, peak concurrent users, latency target, retry rate, cache hit rate and RAG overhead, each measured before self-hosting.
- Every price with its source. API rates, rental or purchase quotes, power tariff and operations fees, each dated, since prices move (one 2026 rent-or-own analysis shows rental compression reversing ownership cases within two years).
- Cost at the SLO-feasible load. Tokens per dollar taken at the highest load that still meets the SLO.
- People cost stated as a fraction of a named role, not left out.
- Sensitivity cases: each volume scenario read off a measured arrival-rate curve, not a fixed utilisation.
- Eligibility filter: which options residency rules excluded and on whose determination.
- A review date. One guide treats self-hosting as a decision to re-benchmark every six months as open and closed models improve.
The test for whether the document is finished is simple. A reviewer who has never seen your system should be able to trace each figure on the summary line back to a measurement or a dated quote, and say which input would have to change for the decision to flip. If that input is the utilisation figure and nobody measured it, the model isn’t ready, however good the spreadsheet looks.
