A cost-per-request SLO is only as real as the telemetry behind it. If a cost figure cannot be decomposed back to the request that produced it, it cannot support an optimisation decision — you can only observe that spend moved, not why.
The naive version of this is familiar. Someone reads the monthly provider invoice, divides by request count, and reports the quotient as cost-per-request. The number is arithmetically fine and operationally useless: it cannot be sliced by feature, by model version, by tenant, or by retry path. When it rises 18% month over month, nobody can say whether a new feature shipped, a prompt template grew, a cache regressed, or a fallback model started absorbing traffic.
The instrumentation pattern below is the alternative. It is not complicated, but it has to be in place before the cost question gets asked, because instrumentation designed after the fact almost always misses retries, streaming partials, cache hits and fallback-model calls — precisely the paths where per-request economics degrade.
What does instrumenting production AI for cost-per-request tracking mean in practice?
It means three things happening on the same trace, in this order.
Tag at ingress. Every inference request gets labelled at the entry point of the serving path with the attributes you will later want to slice cost by. Not at the model call — at ingress, before any branching, so retries and fallbacks inherit the same request identity.
Emit consumption counters. On the same span, record what the request actually consumed: prompt tokens, completion tokens, GPU-milliseconds, accelerator ID, batch occupancy. These are physical quantities, not money.
Attribute cost onto the tags. Money is a derived layer. Provider spend maps to token counters via a rate table; self-hosted spend maps to GPU-time counters via an accelerator-hour cost. Keeping the physical and monetary layers separate is what lets a price change re-derive history instead of invalidating it.
The distinction that matters: the first two steps are engineering work on the serving path, the third is a join. Teams that try to make the serving path emit currency directly end up rebuilding it every time a contract, an instance type, or a discount tier changes.
Which attributes must be attached to every inference request?
This is the part worth over-specifying, because adding a tag retroactively gives you no history. In our experience the minimum useful set is smaller than teams expect and more structural than they expect.
| Tag | Why cost cannot be sliced without it | Cardinality risk |
|---|---|---|
feature_id |
Maps spend to the product surface that earns revenue; the unit of margin conversation | Low — bounded by product surface count |
model_id + model_version |
Separates a model swap from a workload change; without the version, an upgrade and a traffic shift look identical | Low |
prompt_version / template_version |
Prompt growth is the most common silent cost regression; untagged, it is invisible | Low if templates are registered |
tenant_id |
Required for per-customer gross margin and for finding the one account distorting the mean | High — hash or bucket for dashboards |
cache_state (hit / miss / partial) |
Cache hits consume near-zero model compute; mixing them into the mean understates true miss-path cost | Low |
attempt_no + terminal_status |
Distinguishes first attempt from retry, and completed from aborted or rejected | Low |
route_class (primary / fallback / degraded) |
Fallback models often have inverted price profiles; untagged fallback traffic is the classic invoice surprise | Low |
request_id (propagated) |
The join key. Everything above is worthless if the retry lands on a different trace | — |
Two notes from practice. First, tenant_id at full cardinality will hurt your metrics backend; keep it on the trace and on a sampled event stream, and aggregate to a bucketed dimension in the dashboard layer. Second, attempt_no and route_class are the two tags most often omitted in a first pass and most often responsible for a rebuild six months later.
Attributing provider spend and self-hosted GPU-hours
The two cost sources need different joins, and a hybrid deployment needs both running side by side.
For metered provider calls, the attribution is a rate lookup: prompt tokens × input rate + completion tokens × output rate, evaluated per model version and per contract period, then summed onto the request’s tags. Cached-prefix and batch-discount pricing tiers belong in the rate table, not in application code. The reconciliation test is simple — the sum of attributed request cost for a billing period should land within a small percentage of the invoice line for that model. If it does not, you have untagged traffic, and finding it is more valuable than the cost number itself.
For self-hosted serving, the physical quantity is GPU-time, and the honest denominator is reserved accelerator-hours rather than busy ones. Idle reservation is a real cost that a request-level view will otherwise discard. A workable approach: measure per-request GPU-milliseconds from the serving runtime (vLLM, SGLang and Triton all expose per-request timing and batch metadata; NVIDIA’s DCGM and standard CUDA-level profiling supply device-level utilisation to sanity-check against), then allocate the reservation cost of each accelerator across the requests it served in that window, pro-rata on GPU-time. Idle time becomes a visible residual line rather than a silent dilution. The per-request GPU counters this depends on come from the same profiling discipline described in our GPU profiling and optimisation work — attribution consumes profiling output, it does not replace it.
Under continuous batching, per-request GPU-time is an allocation, not a measurement. Say so on the dashboard. A batched request’s share of a step is an accounting convention, and different conventions shift individual request costs materially even when the aggregate is stable.
Accounting for retries, streams, cache hits and fallbacks
These four paths are where instrumentation designed late fails, so they deserve explicit rules rather than defaults.
- Retries attribute cost to every attempt that consumed compute, under one shared
request_id. Report attempt-level cost for waste analysis and request-level cost (the sum) for the unit-economics number. The gap between the two is retry overhead as a percentage of per-request cost. Where the honest denominator matters most — timeouts, 5xx storms, requests that burned prompt tokens before failing validation — we treat it as its own subject; see why failed requests belong in the cost denominator. - Streaming partials must be counted on abort. A cancelled stream has generated real completion tokens. Emit the counter on connection close, not only on successful completion, or every abandoned generation reads as free.
- Cache hits are recorded as requests with near-zero model cost but non-zero serving cost, and never dropped. Excluding hits inflates the reported miss-path cost; folding them silently into the mean makes cache regressions invisible. Report both blended cost-per-request and miss-path cost-per-request.
- Fallback calls carry the same
request_idwithroute_class: fallbackand their ownmodel_id. A request that hits a primary and then a fallback has two consumption events and one unit cost.
A minimum viable dashboard schema
One fact table, three derived views. That is enough to make the parent SLO measurable.
Fact grain: one row per attempt, keyed on (request_id, attempt_no), carrying the tag set above plus prompt_tokens, completion_tokens, gpu_ms, provider_cost, allocated_infra_cost, latency_ms, timestamp.
View 1 — unit cost. Cost-per-request and cost-per-token, grouped by feature_id × model_version, with p50/p95 latency on the same panel. Cost and latency must be readable together or teams will trade one for the other without noticing.
View 2 — waste. Retry overhead %, abort-attributed cost %, fallback share of spend, and cache hit rate — each expressed as a percentage of total attributed cost for the feature.
View 3 — coverage. Attributed spend ÷ invoiced-plus-reserved spend, per model and per period. This is the credibility panel; the other two views are only as trustworthy as this one.
Cost-per-token belongs beside cost-per-request rather than instead of it: token cost isolates model and prompt efficiency, request cost captures how many calls a product action provokes. When one customer-facing action starts fanning out into a variable number of calls, the request-level KPI stops being sufficient on its own — that transition is treated in graduating from cost-per-request to per-business-action economics.
Validating that coverage is high enough to decide on
Before anyone optimises against these numbers, three checks:
- Reconciliation. Attributed cost against the provider invoice and against reserved accelerator-hours, per period. Persistent unexplained residual means untagged traffic.
- Tag completeness. The share of attempts with a non-null value for each required tag. A tag that is 70% populated cannot support a slice; treat completeness per-tag, not as one global number.
- Attempt closure. The share of attempts with a
terminal_status. Open-ended attempts are usually aborted streams that never emitted their counter.
Our working threshold before we let a cost figure drive an architectural decision is high attribution coverage with per-tag completeness on the dimension being sliced (an internal engagement heuristic, not a published benchmark). Below that, the correct output of a cost review is an instrumentation backlog, not an optimisation plan. This is also why the [inference cost cut pack](Inference Cost-Cut Pack) treats telemetry as a precondition: findings on a buyer’s deployed serving path have to be re-measurable after a change, and they are not if coverage moved at the same time as the code.
What has to survive a model or provider swap
The test for a durable design is whether a provider migration invalidates your history. If cost is attributed onto workload-anchored tags — feature, tenant, route, cache state — and money is derived from a separate rate table, the swap changes one term and the trend line stays continuous. If model_id is baked into the metric name, or currency is emitted from application code, you rebuild.
That portability is the whole reason to keep the physical and monetary layers apart. Token and GPU-time counters describe the workload and stay valid for the life of the feature; rates are contractual and change without warning. We see the same pattern across AI infrastructure and SaaS platforms: the teams who can answer “why did unit cost move” are the ones who instrumented the workload rather than the invoice.
The open question we have not seen answered cleanly anywhere is batching attribution. Under continuous batching there is no neutral way to split a fused step’s GPU-time across the requests inside it, and the convention you pick shifts which request class looks expensive. Aggregate cost stays honest; per-request comparisons across very different sequence lengths do not. If someone has a convention that survives both scrutiny and a 10× concurrency change, we would like to see it.
Frequently Asked Questions
What does instrumenting production AI for cost-per-request tracking mean in practice? Every production AI request generates costs across inference compute, token consumption, and model serving—costs that remain invisible without per-request attribution. It means tagging every inference request at ingress with the attributes cost must later be sliced by, emitting token and GPU-time counters on the same trace, and attributing provider spend and self-hosted accelerator-hour cost back onto those tags. The physical counters and the monetary layer stay separate, so a price change re-derives history rather than invalidating it.
Which attributes must be attached to every inference request for cost attribution to be sliceable later? At minimum: feature, model ID and version, prompt or template version, tenant, cache state, attempt number and terminal status, route class (primary/fallback/degraded), and a propagated request ID as the join key. Attempt number and route class are the two most commonly omitted and the two most likely to force a rebuild later.
How do you attribute provider token spend and self-hosted GPU-hour cost back to individual requests? Provider spend is a rate lookup against per-model-version token counters, summed onto the request’s tags and reconciled against the invoice line. Self-hosted cost allocates each accelerator’s reserved hourly cost across the requests it served, pro-rata on measured GPU-time, leaving idle reservation as a visible residual rather than a silent dilution.
How should retries, streaming responses, cache hits and fallback-model calls be accounted for in per-request cost? Every attempt that consumed compute is attributed, under one shared request ID; the sum is the unit cost and the gap is retry overhead. Streaming counters must be emitted on abort, cache hits recorded as near-zero-model-cost requests rather than dropped, and fallback calls tagged with their own model ID under the original request.
What does a minimum viable dashboard schema for cost-per-request and cost-per-token look like? One fact table at attempt grain — keyed on request ID and attempt number, carrying the tag set plus token counts, GPU-milliseconds, provider cost, allocated infrastructure cost and latency — feeding three views: unit cost with p50/p95 latency, a waste view (retry, abort, fallback, cache), and an attribution-coverage view.
How do you validate that attribution coverage is high enough for the numbers to drive decisions? Run three checks: reconciliation of attributed cost against invoices and reserved accelerator-hours, per-tag completeness on the dimension being sliced, and attempt closure (share of attempts with a terminal status). If coverage is weak, the correct output of a cost review is an instrumentation backlog rather than an optimisation plan.
What instrumentation has to survive a model or provider swap without being rebuilt? The workload-anchored tags and the physical counters — feature, tenant, route, cache state, tokens, GPU-time. Rates belong in a separate, versioned table; if model IDs are baked into metric names or currency is emitted from application code, a migration breaks the trend line and the history goes with it.
Why per-request cost matters more than monthly totals
Aggregate spend hides the distribution—until you instrument per-request cost, you cannot see that 4% of queries consume 60% of your budget. Revisit it when your workload shifts.
