The bill on an LLM serving deployment goes up, and the first hypothesis in the room is almost always the same: the model is too big. A worked audit answers that hypothesis with measurement rather than a shortlist of smaller checkpoints — and on LLM serving paths the dominant cost driver frequently turns out to be output token length or unbatched short requests holding GPU memory, not model capacity.
What follows is one audit walked end to end on a single LLM serving workload: what was baselined, what the profiler showed, which lever was ranked first, and what the ROI model said about the model swap that had already been half-approved. The shape of the engagement — baseline, profile, rank, price — is the same one described in the [inference cost audit methodology](Inference Cost-Cut Pack) we run as a fixed deliverable. This page is that methodology applied, not re-explained.
The workload as it arrived
A chat-and-summarisation product, self-hosted on a small GPU fleet, one strong instruct model behind a single endpoint. Traffic mixed short interactive turns with long document summarisations. The team’s plan was to move summarisation onto a smaller checkpoint, which meant an evaluation harness, a quality bar to re-establish, and several engineering weeks before the first invoice would change.
The stated problem was cost. The measured problem had not been defined yet, because the only number anyone could quote was the monthly GPU spend divided by request count — an aggregate that cannot attribute a change to a cause. Establishing that attribution is the whole first week’s work; the mechanics of doing it are covered in how to measure inference cost-per-request before any optimisation.
What was baselined before anything moved
Nothing gets optimised until there is a number that can be reproduced next month. On this workload the baseline captured, over a defined seven-day traffic window, per request class:
| Measure | Interactive turns | Summarisation |
|---|---|---|
| Median input tokens | ~340 | ~4,100 |
| Median output tokens | ~90 | ~620 |
| p95 time-to-first-token | 0.6 s | 3.9 s |
| p95 end-to-end latency | 4.1 s | 26 s |
| Cost-per-request (measured baseline) | low | ~9× the interactive class |
| Share of total requests | 78% | 22% |
| Share of measured GPU-seconds | 31% | 69% |
Two things fell out of the table immediately. Summarisation was a fifth of the traffic and two thirds of the spend, so any lever applied uniformly across the endpoint would be mostly wasted. And the interactive class, despite being cheap per request, was holding a disproportionate amount of KV-cache memory relative to the work it did — the memory ceiling, not the compute ceiling, was what capped concurrency.
How does cost-per-token become cost-per-request when lengths vary?
Cost-per-token is a billing convenience; cost-per-request is the operational unit, and the two only agree when length is stable. On this workload the mean masked a wide distribution: a summarisation request with a 4,100-token prompt and a 620-token completion does not cost seven times a 90-token completion, because prefill and decode consume the accelerator differently. Prefill is a dense, well-batched matrix workload that saturates the GPU; decode is a sequential, memory-bandwidth-bound step that generates one token at a time per sequence and leaves compute units idle unless many sequences are in flight together.
That is why we price a request as prefill cost plus decode cost, with decode weighted by output length and by how many concurrent sequences the batcher managed to keep resident. Where the tokenizer ratio itself is in question — different for code, non-English text, and heavy whitespace — the counting assumptions need their own check before the cost model is trusted.
What the profiler showed
Instrumenting the serving path with request-level traces alongside kernel-timeline and memory-occupancy capture produced a picture the dashboard could not:
- Decode dominated. Roughly three quarters of the measured GPU-seconds on the summarisation class were spent in the decode loop, not prefill. This is the signature of a decode-bound workload: kernel timelines show short, frequent attention and projection kernels with low arithmetic intensity, GPU utilisation reads moderately high while achieved FLOPS sit far below peak, and throughput scales with batch occupancy rather than with clock or core count. A prefill-bound workload looks like the inverse — long dense kernels, high achieved FLOPS, and time-to-first-token that tracks input length.
- Batch occupancy collapsed at peak. Average in-flight sequences during the busiest windows sat well below the configured maximum, because KV-cache reservation was sized for worst-case context length and the scheduler ran out of memory blocks long before it ran out of compute.
- Output length was uncapped. The summarisation prompt had no length instruction and no
max_tokensceiling appropriate to the product surface. A meaningful share of completions ran long past the point where the UI truncated them — tokens paid for, generated, and never displayed. - Short requests held memory. Interactive turns kept cache blocks reserved through idle think-time in a long-lived session pattern, which is exactly the “unbatched short requests holding GPU memory” case that keeps occupancy low without appearing on any compute metric.
None of these findings implicate model capacity. A smaller checkpoint would have reduced the per-token cost of a decode loop that was already running at a fraction of the hardware’s capability, while leaving occupancy, cache sizing and output length untouched.
The ranked roadmap and what the ROI model priced
Ranking is where an audit earns its fee, because the ordering is financial, not technical. Each candidate lever was priced against the measured baseline: expected movement in cost-per-1K-output-tokens and cost-per-request, engineering effort in loaded weeks, and regression risk.
| Rank | Lever | Cost term it attacks | Effort | Quality risk |
|---|---|---|---|---|
| 1 | Cap and instruct output length per request class | Decode token volume | Days | Low — bounded by product spec |
| 2 | Re-tune KV-cache block size and eviction; free cache on idle sessions | Batch occupancy at peak | 1–2 weeks | Low |
| 3 | Continuous batching config: separate scheduling for long-context and interactive classes | Occupancy + p95 TTFT | 2 weeks | Low |
| 4 | Quantised weights on the summarisation path with the same checkpoint | Decode bandwidth | 2–3 weeks | Medium — needs eval harness |
| 5 | Route interactive turns to a smaller model behind a calibrated quality threshold | Model compute on 78% of traffic | 4+ weeks | Medium–high |
| 6 | Replace the summarisation model outright | Model compute on 22% of traffic | 6+ weeks | High |
The first three levers touch configuration and prompt contracts. They do not change the model, they do not need a re-evaluation cycle, and together the ROI model projected them as carrying the majority of the recoverable spend on this workload — because they act directly on the two terms profiling had named as dominant. The model swap the team had almost approved sat at rank six, attacking the smaller share of traffic with the largest engineering cost and the largest quality exposure.
The avoided cost is the finding, not a footnote. Six-plus engineering weeks plus a full re-evaluation cycle, spent on a lever the profiler showed would not have moved the dominant cost term, is a real number that belongs on the same ledger as the savings.
When would model replacement have ranked first?
If prefill had dominated the GPU-seconds, if achieved FLOPS during decode had been close to peak, and if batch occupancy at peak had already been near the configured maximum, then the serving path would have been doing its job and the remaining cost would genuinely be model compute. In that case ranks one through three would return near-zero, and replacement — or routing at rank five — moves to the top of the list. Model replacement is sometimes the right answer. The audit’s contribution is that it becomes a measured decision with a priced alternative behind it, rather than the first guess that happened to have a name everyone recognised.
We see this ordering hold across most LLM serving audits we run, though the specific ranking always follows the traffic mix rather than a template. A workload with uniform short prompts and heavy concurrency produces a different table entirely. Where the ranked roadmap turns into a funded project is a separate question — one of engineering hours against sustained call volume over a payback window, which is the commercial conversation our R&D engagements are scoped around.
What the audit handed over
A baseline that can be re-run, a named bottleneck with kernel-level and memory-level evidence behind it, a ranked roadmap with effort and risk on every line, and an ROI model calibrated on that baseline rather than on a generic reduction percentage. The team owns the change; the audit owns the evidence.
The open question on this workload was not which lever to pull first — the table settled that. It was how far the interactive class could be routed away from the strong model before measured answer quality moved, and that needs traffic-labelled evaluation data nobody had yet collected. Naming what you cannot yet decide is part of the deliverable too.
Frequently Asked Questions
What does an inference cost audit applied to an LLM serving workload mean in practice?
Real production workloads reveal cost patterns that spreadsheets and benchmarks never capture. Inference Cost Audit Applied turns on one distinction. Asked about Inference Cost Audit Applied, most engineers point here. Inference Cost Audit Applied behaves predictably once you see the mechanism. Inference Cost Audit Applied turns on one distinction. Inference Cost Audit Applied to an makes this clear: it means walking the deployed serving path in order — prompt and context length, tokenisation, prefill, decode, KV-cache behaviour, batching and queueing, routing, runtime — and attributing measured spend to each stage before proposing any change. The output is a reproducible baseline, a named bottleneck, a ranked roadmap and a priced ROI model, not a set of best-practice recommendations., cost-per-request and cost-per-1K-output-tokens per request class, p95 and p99 time-to-first-token and end-to-end latency, tokens-per-second per GPU, and batch occupancy and GPU utilisation during peak windows — all captured over a defined traffic window. Splitting by request class matters more than the aggregate, because a minority class often carries the majority of GPU-seconds.
Which profiler findings distinguish a prefill-bound workload from a decode-bound one? A prefill-bound workload shows long dense kernels, achieved FLOPS near peak, and time-to-first-token scaling with input length. A decode-bound one shows short low-intensity kernels, achieved FLOPS well below peak despite respectable utilisation readings, and throughput that moves with batch occupancy rather than with compute capacity.
How do continuous batching, KV-cache sizing and request routing change cost without changing the model? All three act on how many sequences the accelerator serves concurrently, which is the term that sets realised cost-per-request on a decode-bound path. Right-sizing cache blocks and freeing them on idle sessions raises occupancy; scheduling long-context and interactive traffic separately stops one class starving the other; routing sends only the requests that need the strong model to it. None of these touch model weights.
At what point in this worked audit would model replacement have been the correct ranked lever? Once the serving path is already efficient — occupancy near its configured ceiling, achieved FLOPS close to peak, output length bounded — the remaining cost is genuinely model compute, and replacement or calibrated routing moves to the top of the ranking. On this workload those preconditions did not hold, which is why the swap ranked last.
Why token accounting alone misses half your spend
Every audit we’ve run surfaces at least one cost driver invisible to naive token counts. Inference Cost Audit Applied rewards teams that measure first and argue later — start with the smallest instrumented slice and let the numbers settle the design.
