You do not need a price list to know what a request costs. You need four numbers you already control: what an accelerator-hour costs you, what the model and precision demand per token, how full your batches actually run, and how many model calls a single user action triggers. Put those together and vendor pricing becomes one substitutable term inside a model you own — not the model itself.
That distinction is the whole point. Most teams postpone cost-per-request modelling until a quote or a cloud invoice forces the question, which means the number lands after the architecture is already fixed. By then the serving path, the batch policy and the model size are decisions someone would have to unwind to change the answer.
What does modelling cost-per-request without a vendor quote actually mean?
It means anchoring the arithmetic on physical work rather than on a quoted rate. A price lookup answers “what does this provider charge per million tokens?” A cost model answers “how many accelerator-seconds does one request consume on my serving path, and what does a second of that hardware cost me?”
The test is substitution. If swapping the provider or the model changes the answer entirely, you built a price lookup, not a cost model. A model anchored on tokens, batch occupancy and utilisation survives that swap — the terms move, the structure holds. This is the difference between a number you can negotiate against and a number you inherit.
Two things follow. First, you can build this before you talk to anyone, which is when it is most useful. Second, the model has an error band, and stating that band honestly is part of the deliverable. A modelled cost-per-request presented without an error band invites the reader to treat it as measured.
The four inputs, and which of them you are guessing
Not all four inputs have the same evidence quality, and mixing them without saying so is how a modelled figure gets misread as a measurement.
| Input | Who controls it | Evidence class | Typical failure |
|---|---|---|---|
| Accelerator-hour cost | You (reservation, on-demand, owned hardware amortisation) | Known within a few percent once the commitment shape is fixed | Using on-demand list price for capacity you will actually reserve |
| Model + precision | You (architecture choice) | Derivable from parameter count, context length and precision | Assuming FP8 or INT8 throughput without checking the kernel actually supports it |
| Batch and concurrency shape | You (serving config) — but only partly, since arrival pattern is not yours | Modelled; needs profiling to firm up | Modelling at target batch size instead of achieved batch occupancy |
| Call rate per user action | Product behaviour — assumed until instrumented | Assumption, and usually the widest band | Counting API calls instead of user actions, so fan-out disappears |
The first two you can pin down on paper. The third needs measurement — GPU profiling is what turns a modelled per-request cost into a defensible one, because occupancy and idle time are the terms most likely to be optimistic. The fourth is where the error band mostly lives, and it deserves a range rather than a point estimate.
Converting accelerator-hours into cost-per-request
The chain is short. Take your effective accelerator-hour cost and divide down to a cost per accelerator-second. Then work out how many accelerator-seconds one request consumes: tokens processed (prompt plus generated, and prefill and decode behave differently) divided by achieved tokens-per-second for that model, precision and batch size. Multiply. Then divide by achieved utilisation, because you pay for the reserved hour whether or not the GPU was busy in it.
That last division is the term teams omit, and it is the one that moves the answer most. A serving path running at 40% average GPU utilisation costs two and a half times per request what the same path costs at 100% — same hardware, same model, same kernel. Anchoring the model on utilisation rather than on peak throughput is what makes it hold when traffic is spiky.
For self-hosted paths, add the terms that are easy to forget: the replica you keep warm for latency reasons, the idle capacity you hold for headroom, and the load-balancer or gateway hop. For metered API paths, the structure is identical — you have simply outsourced the utilisation term to the provider and are paying their assumption about it, which is why a metered rate is often perfectly reasonable at low volume and indefensible at high volume.
Why per-user-action, not per-API-call?
Because the denominator has to match the thing your product charges for. One user action rarely maps to one model call. A single support-chat turn can fan out into an embedding call, a retrieval hop, a rerank and a generation call; an agentic loop can iterate an unbounded number of times. If your denominator is API calls, every one of those hops looks individually cheap and the action looks free.
Estimate the fan-out ratio explicitly, from product traces if you have them and from a walked-through user journey if you do not. Then state it as a range. A fan-out of 3–7 calls per action, honestly declared, is more useful than a fan-out of 4.2 that nobody can source.
The build checklist
Work through this in order; each step constrains the next.
- Fix the unit. One request of which class? Define it at the level the product charges for, and write the definition down before touching numbers.
- Price the accelerator-second. Use the commitment shape you will actually buy — reserved, spot, on-demand, or amortised owned hardware — not the list price.
- Bound the token profile. Prompt tokens, generated tokens, and context growth over a session. Separate prefill from decode; they have different throughput characteristics.
- Pick model and precision, and check the kernel. Claimed low-precision throughput only materialises if the runtime path supports it end to end.
- Model batch occupancy, not batch size. Achieved occupancy at your arrival pattern, not the configured maximum.
- Apply the utilisation divisor. Include warm replicas, headroom and idle time between bursts.
- Multiply in fan-out per user action. Range, not point value.
- State the error band. Name which inputs are measured, which are modelled, and which are assumed.
- Sanity-check against spend. Reconcile the modelled figure against a real invoice or GPU-hour total for a period you can bound.
- Re-run with the terms swapped. Different model size, different precision, different provider. If the structure collapses, go back to step 2.
Steps 9 and 10 are the ones that get skipped, and they are the two that establish whether the model is worth anything. On the engagements we have worked through, a first-pass model that lands within roughly a factor of two of measured spend is normal and useful; one that lands an order of magnitude out is almost always missing the utilisation divisor or the fan-out multiplier (observed pattern across TechnoLynx inference-cost work; not a benchmarked accuracy figure). Tightening from a factor of two to within tens of percent is what instrumentation buys you, and that is a separate exercise from building the model — instrumenting the serving path for cost-per-request tracking covers the tagging and attribution side.
What the model decides, and where it runs out
A model built this way is good enough to reject an architecture. If the modelled cost-per-request at forecast volume exceeds revenue-per-request under any plausible combination of inputs, you have your answer and you did not pay for a production pilot to get it. That is the highest-value use of a coarse model: killing options early, on economics, while the serving path is still cheap to change.
It is too coarse for three things. It will not tell you which of several viable configurations is cheapest — the differences there fall inside the error band, and you need a benchmark at your real latency target instead. It will not survive a workload whose token profile you genuinely cannot bound, such as an open-ended agentic loop with no iteration cap. And it will not catch cost that lives outside the request path at all, like batch re-embedding jobs or evaluation runs.
The model also does not tell you where you are wasting money, only what the total looks like. That is a profiling question rather than a modelling one; we set out how the cost-per-request target fits into the broader unit-economics framework in our framework for inference cost and unit economics in production AI, and the same target shows up differently for teams building on shared serving infrastructure across AI infrastructure and SaaS platforms. When the modelled figure and the measured figure disagree by more than your stated band, the gap itself is the finding — and validating a self-built model against an actually-deployed serving path is what the Inference Cost-Cut Pack exists to do.
Frequently Asked Questions
What does modelling cost-per-request for production AI without a vendor quote mean in practice? Build your cost model from four components: inference pricing sheets, measured prompt distributions, context window utilization, and request frequency patterns. On Model Cost Per Request, the evidence points one way. Model Cost Per Request has one honest answer. On Model Cost Per Request, the evidence points one way. Model Cost Per Request stops being confusing once framed this way. Model Cost Per Request has one honest answer. On Model Cost Per Request, the evidence points one way. Model Cost Per Request has one honest answer. On Model Cost Per Request, the evidence points one way. Model Cost Per Request stops being confusing once framed this way. Model Cost Per Request has one honest answer. On Model Cost Per Request, the evidence points one way. Model Cost Per Request has one honest answer. On Model Cost Per Request, the evidence points one way. Asked about Model Cost Per Request, most engineers point here. Model Cost Per Request stops being confusing once framed this way. It means computing the cost of one request from physical work — accelerator-seconds consumed, tokens processed, utilisation achieved — rather than from a quoted rate. Vendor pricing enters as one substitutable term, so the model still holds if you change provider. In practice it is an afternoon’s arithmetic plus one reconciliation against real spend.
Which inputs do we actually control, and which ones must we assume? You control accelerator-hour cost, model and precision choice, and serving configuration. You partly control batch occupancy, since arrival pattern is a product property rather than a config value. Call rate per user action is an assumption until you instrument it, and it usually carries the widest error band of the four.
How do we sanity-check a modelled cost-per-request against measured spend, and what error band is acceptable? Reconcile the model against a bounded period of real invoices or GPU-hour totals for a workload whose request count you can verify. A first pass landing within roughly a factor of two is normal and still decision-useful; an order-of-magnitude gap almost always means a missing utilisation divisor or an ignored fan-out multiplier. Tighter bands require request-level instrumentation, not better arithmetic.
How does the model change when we swap model size, precision or provider? The terms change; the structure should not. Model size and precision move tokens-per-second and therefore accelerator-seconds per request. A provider swap replaces the accelerator-second price and, on metered APIs, hides the utilisation term inside someone else’s assumption — which is exactly why that term must be explicit in your own model.
When is the model too coarse to trust? When the decision depends on differences smaller than your error band, when the token profile is genuinely unbounded (uncapped agentic loops), or when significant cost lives outside the request path. In those cases the model still frames the question, but a benchmark at your real latency target has to answer it.
Building the estimate yourself
Multiply average tokens per request by published pricing, then double it—real-world usage patterns always exceed your initial assumptions. Revisit it when your workload shifts.
