A cost-per-request figure is only honest if its denominator is honest. Divide the monthly inference bill by the number of successful responses returned to users and you have quietly assigned zero cost to every timeout, every 5xx that triggered a client retry, every cancelled stream, and every call that burned prompt tokens before a guardrail rejected the output. The compute was consumed. The provider billed for it. It simply never appeared in the metric.

This is the failure mode we see most often once a team has done the hard work of standing up unit-economics tracking at all: the instrumentation is real, the dashboard is real, and the number is wrong in one specific direction — always optimistic, and most optimistic precisely when the system is behaving worst.

Why does a successful-only cost-per-request metric break under load?

Because failures and spend move together, and only one of them enters the calculation.

Picture a serving path where an upstream dependency degrades — a vector store slows, a GPU node gets evicted, an autoscaler lags behind a traffic step. Latency climbs past the client timeout. The client retries. Each retry issues a fresh prefill: the prompt tokens are re-encoded, the KV cache is rebuilt from scratch, GPU-seconds are consumed. Some of those retries also time out, so the client retries again. Meanwhile the successes that do get through are, on average, the cheaper short-context calls that finish inside the timeout budget.

The arithmetic that follows is the trap. Numerator up, denominator biased toward cheap successes, headline metric flat — or improving. A cost programme that cannot explain a bill spike coinciding with an error-rate spike is measuring the wrong requests. That sentence is the whole failure in one line, and it is the thing to put on the wall next to the dashboard.

The correct frame is that the billable unit is an attempt, not a delivery. The provider meters compute, not customer satisfaction. Any metric whose denominator is narrower than the set of compute-consuming events will drift from the invoice, and the drift is not random noise — it is structurally correlated with incidents.

Which failures actually cost money

Not every failure is billed, and treating them uniformly produces a different kind of wrong number. The distinction that matters is whether the request reached compute before it died.

Failure class	Billed compute?	Where the cost lands
Request rejected at the gateway (auth, rate limit, schema validation before dispatch)	No	Negligible — gateway CPU only
Guardrail or content filter rejection after generation	Yes, in full	Prefill + decode tokens paid, response discarded
Input-side guardrail rejection before dispatch	No	Negligible
Server-side timeout (deadline exceeded mid-generation)	Yes, partial to full	GPU-seconds burned up to the abort point
Client-side cancellation of a stream	Yes, up to cancel	Tokens already decoded are billed; some runtimes keep generating
Automatic retry after 5xx	Yes, once per attempt	Full prefill re-paid on every attempt
Queue-depth shedding (503 before scheduling)	No	Negligible, and this is the cheap failure you want
Speculative or hedged duplicate request	Yes, both copies	Deliberate spend; must be labelled, not hidden

Two things fall out of that table. First, shedding early is economically superior to timing out late — a 503 at the queue costs almost nothing while a deadline exceeded at 80% of a generation costs almost everything. Second, cancelled streams are the class teams most often assume are free. Whether the runtime actually stops decoding on client disconnect is a property of your stack, not a guarantee; in vLLM, SGLang and most managed endpoints it is a configuration and version question worth verifying rather than assuming.

Reconciling the metric against the invoice

The practical instrument is a two-number report rather than a single headline. Compute both:

• Delivered cost-per-request — billed spend divided by successful responses. This is the number that maps to customer value.
• Attempted cost-per-request — billed spend divided by every attempt that consumed compute, including retries, timeouts, cancellations and post-generation rejections.

The ratio between them is the waste ratio: the share of billed inference spend that produced no delivered response. Reported alongside the headline unit cost, it turns an invisible leak into a named line item.

Reconciliation then becomes a monthly closing exercise:

1. Sum billed spend from the provider invoice or, for self-hosted capacity, from GPU-hour allocation including idle reservation.
2. Sum instrumented cost across all attempts from your own telemetry.
3. Compare. A residual gap after both sides are computed over attempts points at compute you are not tracing at all — background jobs, evaluation runs, warm-up traffic, a sidecar model nobody owns.
4. Attribute the delivered/attempted gap by failure class using the table above.
5. Report waste ratio as a first-class metric next to the cost-per-request SLO.

In our experience the first honest reconciliation is uncomfortable, because step 3 usually finds something. That is the point: an unexplained gap between measured unit cost and the actual invoice is the most concrete symptom a platform team can bring to a cost review. It is also why the cost-per-request baseline in our [inference cost-cut audit](Inference Cost-Cut Pack) is computed over all attempts from the outset — surfacing retry and timeout waste as a named item rather than letting it dissolve into the headline figure.

Setting retry policy so it cannot compound the bill

Once waste is visible, retry configuration stops being a reliability-only decision and becomes a cost decision with a reliability constraint.

The compounding mechanism is straightforward. Fixed-interval retries from many clients synchronise; a degradation that would have recovered under a 1× load instead faces 3×, which prevents recovery, which triggers more retries. The bill grows superlinearly against a system that is delivering nothing.

What we pay close attention to on a serving path:

• Retry budgets over retry counts. Cap retries as a percentage of total traffic (a few percent) rather than as “3 attempts per request”. Under a broad degradation the budget exhausts and the storm cannot form.
• Exponential backoff with jitter, always. Fixed intervals guarantee synchronisation.
• Circuit breaking at the point where p95 latency starts climbing, not at the point where errors appear — by then the retries are already in flight.
• Timeout budgets that shrink down the call chain. If a RAG turn has a 10s user-facing deadline, the generation call cannot also have a 10s deadline; each hop must reserve headroom or the outer layer times out while the inner layer keeps billing.
• No retries on non-idempotent or already-partially-delivered streams. A retried stream re-pays prefill for output the user has already seen.
• Label hedged requests explicitly. Hedging is a legitimate latency technique that deliberately buys duplicate compute; it should show up as intentional spend, not as waste.

Instrumenting this means tagging at attempt granularity rather than at request granularity. Each attempt carries its own trace with an attempt index, a terminal-state label (delivered, timeout, cancelled, rejected-post-gen, shed), and its own token and GPU-time counters. Aggregating by the parent request ID then gives you both numbers for free. The tagging discipline itself — feature, model, tenant, cache-hit state — is the subject of our guidance on instrumenting production AI for cost-per-request tracking; attempt state is one more dimension on the same trace, not a separate pipeline.

What this does to the margin picture

Steady state and peak load produce different economics, and the delivered/attempted split is where the difference becomes legible. At steady state, waste ratio is usually a small, stable tax. Under peak load or partial degradation it can dominate: the feature’s gross margin is being set not by the model you chose but by the share of spend going to responses nobody received.

That reframes what a cost-per-request SLO is for. It is not a number to drive down in isolation; it is a control variable that has to survive the operational reality of the serving path rather than the happy path. The same discipline that makes cost-per-request robust to a model swap — which the production AI cost and infrastructure framework develops in full — has to be applied to the failure path, or the metric quietly stops describing the invoice.

For teams running self-hosted capacity, the attribution question goes one level deeper: profiling the serving path is what lets you assign GPU-seconds to aborted and retried calls rather than only to delivered ones. Without that, the waste ratio is an estimate rather than a measurement.

A reasonable place to set the investigation threshold is where waste stops being noise and starts being a budget line — but the right number depends on your traffic shape and how expensive your prefill is relative to decode. The more useful question is whether your waste ratio moves when your error rate does. If it does not, the instrumentation is still counting the wrong requests.

Frequently Asked Questions

What does it mean in practice to say cost-per-request optimisation misses the bill if you ignore failed requests?

Failed requests destroy your cost-per-request metrics by consuming resources without generating value. For Cost Per Request Optimisation, it helps to be precise. The useful way to read Cost Per Request Optimisation is this. For Cost Per Request Optimisation, it helps to be precise. Under the hood, Cost Per Request Optimisation is this. The useful way to read Cost Per Request Optimisation is this. For Cost Per Request Optimisation, it helps to be precise. The useful way to read Cost Per Request Optimisation is this. For Cost Per Request Optimisation, it helps to be precise. Under the hood, Cost Per Request Optimisation is this. The useful way to read Cost Per Request Optimisation is this. For Cost Per Request Optimisation, it helps to be precise. The useful way to read Cost Per Request Optimisation is this. For Cost Per Request Optimisation, it helps to be precise. With Cost Per Request Optimisation, the detail that matters is this. Under the hood, Cost Per Request Optimisation is this. Looking at Cost-Per-Request Optimisation Misses the Bill If, it means the denominator excludes compute you were charged for. Timeouts, retried 5xx calls, cancelled streams and post-generation guardrail rejections all consume GPU-seconds and tokens, so counting only delivered responses produces a unit cost that is systematically lower than the invoice implies — and most wrong during incidents., anything that reached compute is billed: server-side timeouts (up to the abort point), each retry attempt in full, cancelled streams up to the cancellation, and guardrail rejections that happen after generation. Failures that die before dispatch — gateway auth failures, input-side validation, queue shedding — cost effectively nothing.

How do we reconcile a measured cost-per-request against the actual provider invoice, and what does the gap tell us? Sum billed spend on one side, sum instrumented per-attempt cost on the other, and compare. The delivered-versus-attempted gap is your waste ratio, attributable by failure class; any residual beyond that points at compute you are not tracing at all, such as evaluation runs, warm-up traffic or an unowned sidecar model.

How should retry policy, backoff and timeout budgets be set so retries do not silently compound the bill during a degradation? Use retry budgets expressed as a share of total traffic rather than a fixed attempt count, apply exponential backoff with jitter so clients do not synchronise, break the circuit when p95 latency starts climbing rather than waiting for errors, and shrink timeout budgets down the call chain so inner hops cannot keep billing after the outer deadline has passed.

What waste-ratio threshold should trigger investigation, and how does it sit alongside the cost-per-request SLO? There is no universal threshold — it depends on traffic shape and how prefill-heavy your workload is. The operationally useful trigger is correlation rather than level: if waste ratio does not move when error rate moves, the instrumentation is wrong; if it does move and the movement is material against the cost-per-request SLO, the retry and timeout policy is the first thing to examine.

Calculate total cost including retries and failures

Multiply your per-request price by the inverse of your success rate to see what each completed request actually costs once you account for wasted inference on errors. If Cost Per Request Optimisation is on your roadmap, the next step is to map it onto your own constraints rather than copy a reference architecture.