A bad cost-per-request number almost always triggers the same reflex: change the model. Distil it, quantise it, drop to a cheaper provider tier. That reflex conflates two different levers, and it is why so many cost programmes spend a quarter on model bake-offs and end up with a saving they cannot hold.
Model choice sets a floor on per-request compute. The serving path determines how far above that floor you are actually operating. Those are separate quantities, and only one of them is usually the reason your number looks bad.
Why does replacing the model rarely fix cost-per-request?
Because the model change is not doing the work you think it is doing. When a team swaps a 70B model for a distilled 8B and cost-per-request drops by 60%, the honest question is: did anything change about how requests were batched, cached, or scheduled? Usually yes — a smaller model fits more concurrent sequences into the same VRAM, so effective batch size rises without anyone touching a config value. The saving came from batch density, not from the weights.
Here is the diagnostic that separates the two. If a model swap moved cost-per-request without any deliberate change to batching, caching, or scheduling, the earlier number was measuring serving inefficiency rather than the model. That single test reframes the whole programme, because serving inefficiency is recoverable at fixed model and fixed output quality — and the recovery is portable to whatever model you run next.
The corollary is uncomfortable for anyone mid-bake-off: a cost programme that begins and ends at model selection has no mechanism to hold the gain. The next model arrives, the serving path is still misconfigured, and the same 60% waste reappears under a different set of weights. We see this pattern regularly on inference paths that have been through two or three model generations without a single profiling pass.
Where the recoverable cost actually sits
Four sources of serving-attributable waste account for most of what teams misread as model cost. Each is measurable at fixed weights and fixed p95 latency, which is what makes them separable from the model floor at all.
Batching behaviour. The gap between configured maximum batch size and effective batch size under real traffic is often large. A queue that admits requests faster than it fills batches will run at a fraction of its configured density. Continuous batching in runtimes like vLLM, SGLang, or TensorRT-LLM closes part of this, but only if the concurrency and admission settings match your arrival pattern rather than a benchmark’s.
Cache hit rate. Prefix and KV caching turn repeated system prompts, few-shot blocks, and retrieval context into compute you pay for once. A low hit rate on a workload with a fat shared prefix is pure, recoverable spend — and it is invisible in any price-per-token comparison between models.
Padding and sequence-length waste. Requests bucketed into fixed shapes pay for tokens nobody asked for. Length-aware batching and dynamic shapes recover it. The model is not involved in the decision.
Idle accelerator time. GPU-seconds are billed whether or not a request is in flight. Utilisation during the serving window — not average daily utilisation — is the figure that maps to per-request cost. Reserved capacity sitting idle between bursts is a scheduling problem wearing a hardware-cost disguise.
Profiling the serving path is the step that turns these four from hypotheses into an attribution. Our work on GPU profiling and kernel-level performance analysis covers the mechanics; here the point is narrower — profiling exists to tell you how much of the bill belongs to serving before anyone opens a model comparison spreadsheet.
Which lever for which symptom
| Symptom in the deployed path | Attributable to | Right first lever | Wrong first lever |
|---|---|---|---|
| Effective batch size well below configured maximum | Serving | Admission control, continuous batching, concurrency tuning | Smaller model |
| Low prefix/KV cache hit rate on a shared system prompt | Serving | Prefix caching, prompt template stabilisation | Cheaper provider tier |
| High padded-token fraction per batch | Serving | Length-aware bucketing, dynamic shapes | Distillation |
| GPU idle for a large share of the serving window | Serving | Scheduling, co-location, autoscaling policy | Larger instance |
| Cost near the floor implied by current weights, at target p95 | Model | Quantisation, distillation, or a smaller model — evaluated on quality | Further serving tuning |
| Quality headroom above product requirement at high per-request compute | Model | Deliberate model downgrade with a quality gate | Anything cost-framed |
The last two rows matter as much as the first four. Model replacement is a legitimate lever — it is just a second lever, and the evidence that should precede it is an attribution showing you are already close to the floor the current model implies.
The measurement that makes this argument operational
The measurable outcome of a serving-first cost programme is a split: what fraction of current cost-per-request is recoverable without changing model weights or output quality. Producing that split requires holding two things fixed while you vary the serving path — the model, and p95 latency. Without the latency clamp, every “saving” is just a trade against responsiveness that the product will eventually reject.
Four figures carry the attribution:
- cache hit rate on the serving path
- effective batch size against configured maximum
- accelerator utilisation during the serving window
- cost-per-request before and after serving changes, at fixed model and fixed p95
Teams that quantify this split get something beyond the saving. They can defend a model decision on quality grounds instead of using cost as a proxy for it — which is the argument ML-Platform leads usually want to make anyway and cannot, because the cost data is entangled with the serving data.
Our [inference cost-cut pack](Inference Cost-Cut Pack) is built around this ordering: profile the deployed serving path, attribute cost-per-request to serving waste versus the model’s compute floor, and only then entertain a model-change recommendation. The broader question of how per-request economics sits inside an AI infrastructure and SaaS platform — where the serving path is shared across tenants and features — is where this attribution becomes a governance concern rather than an engineering one. For the full unit-economics frame that this boundary sits inside, see our treatment of cost-per-request as a production AI target, which covers when the KPI is the right one at all.
What this does not claim
It does not claim models never matter. Precision, parameter count, and architecture set a hard floor, and no amount of batching gets you below it. It does not claim serving tuning is cheap — a serving-path change touching admission control and caching is real engineering work with real regression risk.
What it claims is narrower and more useful: the order is fixed. Serving attribution first, model decision second, because the first is portable across model generations and the second is not.
The open question we keep returning to is how teams should re-run the attribution after a model change lands. A new model shifts the floor and the batch-density arithmetic simultaneously, and we have not yet seen a clean convention for re-baselining that does not quietly reset the cost programme to zero.
Frequently Asked Questions
What does “cost-per-request optimisation is not model replacement” mean in practice?
Cost Per Request Optimisation targets efficiency within your existing model architecture, not a wholesale switch to cheaper alternatives. On Cost Per Request Optimisation, the evidence points one way. The useful way to read Cost Per Request Optimisation is this. On Cost Per Request Optimisation, the evidence points one way. The useful way to read Cost Per Request Optimisation is this. Cost Per Request Optimisation answers cleanly when you separate two things. On Cost Per Request Optimisation, the evidence points one way. It means the two levers are separable and must be pulled in order. Model choice sets the compute floor per request; the serving path — batching, caching, padding, scheduling — determines how far above that floor you run. In practice, most of the gap between your current number and your target sits in the serving path, and closing it requires no change to weights or output quality.
Which cost-per-request reductions come from the serving path rather than the model?
Four categories: raising effective batch size toward the configured maximum, raising prefix and KV cache hit rate on shared prompt or retrieval context, cutting padded-token waste through length-aware batching, and reducing idle accelerator time inside the serving window. All four are measurable at fixed model and fixed p95 latency, which is exactly what makes them attributable to serving rather than to the model.
How do we tell how much of our current cost-per-request is serving waste versus the model’s compute floor?
Profile the deployed serving path and compute the split: cost recoverable without changing weights or quality, versus the floor the current model implies. The four working figures are cache hit rate, effective versus configured batch size, utilisation during the serving window, and before/after cost-per-request at fixed model and fixed p95. A useful shortcut test — if a past model swap moved cost without any batching, caching, or scheduling change, the earlier number was measuring serving inefficiency.
When is a model swap or quantisation actually the right lever, and what evidence should precede it?
When the attribution shows cost-per-request is already near the floor the current model implies at your target p95, or when you have measured quality headroom above the product requirement. The evidence that should precede it is the serving-versus-model split plus a quality gate for the candidate. Without those, a swap is a guess whose saving cannot be attributed.
Why do model-swap savings often fail to hold across the next model change?
Because the saving was frequently produced by an incidental change in batch density rather than by the weights, and the underlying serving misconfiguration is untouched. When the next model arrives, the same waste reappears. A programme that begins and ends at model selection has no mechanism to hold a gain across generations; a serving-path fix is portable and does.
How do we keep p95 latency and output quality fixed while attributing cost-per-request changes?
Clamp both as constraints on every measurement rather than reporting them as outcomes. Serving changes are accepted only if p95 stays within its SLO band, and model weights are held constant so output quality cannot drift. Any comparison that lets latency float is trading responsiveness for cost, not recovering waste.
What does a serving-first cost programme look like alongside a cost-per-request SLO?
The SLO is the target; the attribution is the mechanism. The programme profiles the serving path, publishes the serving-versus-model split, works the serving levers to the floor at fixed p95, and only then opens a model decision — evaluated on quality with cost as a constraint. The split gets re-run after any model change, because a new model moves both the floor and the batch arithmetic at once.
Why swapping models rarely solves the problem
Prompt bloat and inefficient context windows drive most cost overruns, not the model’s price per token. Revisit it when your workload shifts.
