How to Measure Inference Cost-Per-Request Before Any Optimisation

Dividing last month’s GPU spend by last month’s request count gives you a number. It does not give you a baseline. The difference matters the moment someone asks whether a 20% improvement came from your change or from a quieter week.

A usable inference cost-per-request baseline is a record, not a figure: cost-per-request segmented by model, batch size and hardware tier, captured alongside p95 latency and GPU utilisation over a defined traffic window, with enough metadata that the same measurement can be repeated three weeks later and compared honestly. Everything downstream — bottleneck ranking, ROI modelling, the decision to fund or decline an optimisation — inherits the reproducibility of that record. Get it wrong and every later claim about savings is unfalsifiable.

This piece is about that first step only: what to instrument, what to segment by, and how to write it down. Where the money actually lands once you have the numbers is a separate question, and we cover the decomposition in where inference spend lands across model, serving path and overhead.

Why does dividing the GPU bill by request count fail?

It fails for one structural reason: an aggregate quotient has no attribution, so it cannot be differenced.

Suppose your bill-derived figure is $0.0041 per request. Next month it is $0.0034. What changed? Possibly your batching work. Possibly the traffic mix shifted toward shorter prompts. Possibly a reserved-instance discount kicked in, or a noisy retry loop got fixed by someone in a different team, or the month simply had fewer business days. The aggregate number moves for all of these reasons and reports none of them.

There is a second, quieter failure. Bill-derived cost-per-request usually spreads idle capacity evenly across all requests, which means an under-utilised GPU tier looks like an expensive model. Teams then optimise the model. In our engagements the more common pattern is that the accelerator was starved rather than saturated — a batching or host-side pre-processing problem wearing a model costume. Measurement that cannot separate those two cases will send the engineering weeks to the wrong place.

The correct frame: cost-per-request is not a property of a model. It is a property of a model running on a specific hardware tier under a specific batching regime at a specific traffic shape. Drop any of those qualifiers and the number stops being reproducible.

What to instrument before you touch the serving path

The instrumentation requirement is smaller than teams expect. You do not need a full observability rebuild; you need per-request records that can later be grouped.

Minimum fields on each served request:

Request class — an application-level label (chat, summarisation, batch scoring, embedding refresh). Cost per request is meaningless when a 40-token classification and a 4,000-token generation share one average.
Model / checkpoint identifier and revision. Silent checkpoint swaps invalidate baselines.
Hardware tier — instance type or accelerator SKU, plus how many devices served the request.
Realised batch size at execution time, not the configured maximum. The gap between the two is often the finding.
Input and output token counts for LLM workloads; frame count and resolution for vision workloads.
Queue time and execution time separately. Merged into one latency number they hide the batching story.
Timestamp, so the window is definable after the fact.

Alongside that, sample device-level telemetry on the same clock: GPU utilisation, memory occupancy, and achieved SM occupancy where you can get it. NVIDIA’s DCGM exporter or nvidia-smi sampling into Prometheus is sufficient for baselining; you do not need Nsight Systems until you move from baselining to profiling. Our colleagues’ profiling methodology for AI inference picks up exactly where this measurement stops.

One practical note on cost inputs. Convert to money at the effective hourly rate for the tier — the rate you actually pay after commitments and discounts — and record that rate in the baseline. When the rate changes later, a reviewer can normalise. When it is buried in a spreadsheet nobody kept, they cannot.

The baseline record: a template

This is the extractable artifact. Every row is one segment; the segmentation keys are the left three columns.

Field	Example value	Why it is in the record
Segment key: model + revision	`llama-3.1-8b-instruct @ r4`	Cost claims are void if the checkpoint moved
Segment key: hardware tier	`1× A10G, g5.xlarge`	Attributes spend to silicon, not to the model
Segment key: realised batch size band	`4–8`	Distinguishes batching headroom from model cost
Request class	`summarisation`	Prevents mix-shift from masquerading as improvement
Requests in window	`1,284,000`	Establishes statistical weight of the segment
Cost per request	`$0.0038`	The headline figure, now qualified
Cost per 1K output tokens	`$0.0091`	LLM workloads: the billable-unit view
p95 / p99 latency	`840 ms / 2,110 ms`	Cost reductions that break latency SLOs are not savings
Mean GPU utilisation	`41%`	Separates a saturated device from a starved one
Effective hourly tier rate	`$1.006/h`	Makes the money reconstructible
Window definition	`2026-08-04 → 2026-08-17, full days, UTC`	Makes re-measurement possible
Traffic-mix fingerprint	`class share: 62/24/14; median in-tokens 512`	Detects whether the later window is comparable
Excluded traffic	`synthetic health checks, warm-up`	Prevents silent denominator inflation

Two rows minimum per model — one per hardware tier if you run more than one — and one row per batch-size band where the bands actually differ in occupancy. Fewer than that and the record collapses back toward an aggregate.

Window length and the traffic-mix problem

The window has to cover at least one full business cycle for the workload. For most production serving paths that is two weeks: enough to include weekday peaks, weekend troughs, and at least one deployment. A three-day window on a workload with weekly seasonality produces a baseline that will appear to improve or degrade by 15–30% on re-measurement with no engineering change at all (observed across TechnoLynx engagements; not a published benchmark).

Traffic mix is the harder half. You cannot hold it constant, so record it and compare like-for-like:

Capture the mix fingerprint — request-class shares plus median input and output token counts — with every baseline and every re-measurement.
When comparing windows, report cost-per-request per request class, then recombine using the original window’s class weights. This is the same normalisation logic a price index uses, and it isolates your change from mix drift.
If a class share moved by more than roughly a fifth of its original value, say so explicitly in the comparison rather than presenting a single blended delta.
Re-measure the baseline whenever the model revision, runtime version, or instance family changes. Those are new segments, not the same segment with a new number.

What the baseline does and does not tell you

It tells you where to look. A segment showing 41% mean GPU utilisation with realised batch sizes clustered well below the configured maximum points at the serving path — queueing, batch formation, host-side pre-processing — and away from the model. A segment at 90%+ utilisation with tight batch occupancy and high cost-per-token points at the model and runtime instead: quantisation, kernel selection, a compiled graph via TensorRT or ONNX Runtime, or a genuinely smaller checkpoint.

What it does not tell you is which stage inside the identified layer is responsible. Baselining gives you a hypothesis with a cost attached; a kernel timeline gives you the mechanism. Nor does it tell you whether the fix is worth funding — that is a payback calculation against loaded engineering hours, and the rubric for deciding when inference cost work justifies the spend is the right next read.

The measurable output of doing this properly is narrow and useful: a record that converts “our inference is expensive” into “summarisation on the A10G tier costs $0.0038 per request at 41% utilisation, and here is the window it was measured over.” That sentence is fundable. The first one is not. It is also the baselining step the Inference Cost-Cut Pack performs before anything is profiled or ranked, and it is why our engineering engagements start with measurement rather than a proposal.

Frequently Asked Questions

What does measuring inference cost-per-request before any optimisation mean in practice?

In Measure Inference Cost Per, the short answer is as follows. Measure Inference Cost Per comes down to a few moving parts. It means producing a segmented baseline record rather than a single number: cost-per-request broken out by model revision, hardware tier, realised batch size and request class, captured with p95 latency and GPU utilisation over a defined window. The practical test is reproducibility — someone else should be able to re-run the same measurement in a month and know whether they are comparing like with like.

Which metrics and instrumentation do we need in place before we touch the serving path?

Per-request records carrying request class, model revision, hardware tier, realised batch size, token or frame counts, and queue time separated from execution time. Alongside them, device telemetry on the same clock — GPU utilisation and memory occupancy sampled into whatever time-series store you already run. DCGM plus Prometheus is enough; deeper profilers belong to the next stage, not to baselining.

How do we attribute cost across model, batch size and hardware tier rather than to one aggregate GPU bill?

Convert the effective hourly rate of each hardware tier into cost per unit of device time, then divide that time across requests according to their measured execution and queue occupancy on that tier — segmented by model and realised batch-size band. Record the effective rate in the baseline so the money can be reconstructed later. The aggregate bill remains the reconciliation check, not the attribution method.

Cost-per-request is what the business pays for one unit of work; cost-per-token is the billable-unit view that makes requests of different lengths comparable. Record both, because a change in output length moves cost-per-request while leaving cost-per-token flat, and an optimisation to the serving path can do the reverse. Reporting only one lets prompt or completion drift look like an engineering result.

How long a traffic window do we need, and how do we handle traffic-mix variation between measurements?

Two weeks covers the weekly seasonality most production serving paths carry; shorter windows on seasonal traffic can shift 15–30% on re-measurement with no code change (observed across TechnoLynx engagements; not a published benchmark). Handle mix variation by recording a mix fingerprint — class shares and median token counts — and comparing cost-per-request per class before recombining with the original window’s weights.

How do we record the baseline so a later before/after comparison is defensible to a finance or engineering reviewer?

Store the segment keys, the window definition, the effective hourly rate, the excluded traffic, and the mix fingerprint alongside the figures — not just the figures. A reviewer’s objections are almost always about the denominator, the window, or a silent checkpoint change, and a record that pre-answers those three is hard to dispute. Version the record next to the code that produced it.

What does the baseline tell us about whether the bottleneck is model, runtime or hardware — and what does it not tell us?

Utilisation and realised batch occupancy together narrow the layer: low utilisation with under-filled batches indicates the serving path, high utilisation with tight batches indicates the model or runtime. What the baseline cannot do is name the mechanism inside that layer — which kernel, which transform, which cache miss — and it says nothing about whether closing the gap is worth the engineering weeks.

Baseline economics: what to capture on day one

Record wall-clock time, token counts, and instance type for every request in the first week—optimization is guesswork without that reference distribution. Revisit it when your workload shifts.