An inference cost audit deliverable is evidence, not advice. If what lands at the end of the engagement is a prioritised list of best practices — batch more, cache more, quantise — with no measured baseline underneath it, the buyer has bought an opinion. The distinguishing property of a usable deliverable is that it can be re-run: the same instrumentation, the same traffic window definition, the same metric definitions, applied again after each change, producing a before/after figure instead of an assertion.
That property is what this article walks through. Not why the audit should precede a model-replacement decision, but what the artefact itself looks like when you open it.
What does an inference cost audit deliverable contain in practice?
Five sections, in this order, because each one depends on the last.
| Section | What it holds | Why it has to be there |
|---|---|---|
| Baseline | Cost-per-request, cost-per-token, p95 and p99 latency, GPU utilisation, idle-capacity share, for a named traffic window | Nothing later can be measured without it |
| Cost attribution | Spend assigned across tokenisation, prefill, decode, routing, and idle capacity | Names where the money goes, not just how much |
| Profiler findings | Traces and kernel/queue timelines that identify the bottleneck by stage | Turns “it feels slow” into a named constraint |
| Optimisation backlog | Ranked items, each with expected delta, confidence, and estimated engineering days | Makes the work fundable and refusable |
| ROI model | Projected savings against implementation cost, plus avoided-cost lines | Lets finance and platform reviewers check the arithmetic later |
A deliverable missing the first two sections is a recommendations deck. A deliverable missing the last two is a diagnostic report nobody can act on.
The baseline section: what is captured before anything moves
The baseline is the most boring part of the pack and the part that decides whether the rest of it means anything. Its job is to freeze a set of numbers with enough definitional precision that the same numbers can be produced again in six weeks by a different engineer.
In practice that means each metric ships with its scope written next to it. Cost-per-request is stated per model, per request class, and per hardware tier — an aggregate figure across mixed traffic will move on its own as traffic mix drifts, which makes it useless as a control. Cost-per-token is stated separately for input and output tokens, because the two have different cost profiles on almost every LLM serving path. Latency is p95 and p99, never mean. GPU utilisation is paired with idle-capacity share, since a fleet running at 40% average utilisation with reserved instances is carrying a cost line that no kernel optimisation will touch.
The traffic window is named explicitly: which days, which hours, whether a known peak or a known trough. We have seen more baseline disputes come from an unstated traffic window than from any measurement error — one party read a Tuesday afternoon, the other read a weekend.
Establishing that baseline is its own piece of work, and it is worth reading how a reproducible cost-per-request figure is instrumented alongside this section.
The cost-attribution view
Attribution is where the deliverable stops looking like a monitoring dashboard. Instead of one cost line for “inference”, the same spend is split across the stages of the deployed serving path, and each stage carries a share.
For an LLM workload the stages that usually get their own row are tokenisation and request preparation, prefill (processing the input context), decode (generating output tokens), routing between model tiers or endpoints, and idle capacity — reserved accelerator time that nothing occupied. The last row is the one that surprises people. It is not a stage of the request at all; it is the gap between provisioned capacity and used capacity, and on bursty workloads it can be the largest single line in the table.
Attribution changes the conversation because it changes which levers are visible. A team looking at one number has exactly one lever: replace the model. A team looking at five rows can see that decode dominates on long-output traffic, that prefill dominates on retrieval-augmented traffic with large contexts, or that neither dominates and the real problem is that the fleet is provisioned for a peak that arrives twice a week. The three-bucket version of this decomposition — model compute versus serving path versus overhead — sits one level above the stage-level view described here.
What a backlog entry actually looks like
The backlog is the section buyers read first and the section most likely to be soft. An entry that says “enable continuous batching — significant latency improvement” is not fundable. An entry is complete when it carries four things: the change, the expected delta against the baseline metric it moves, a confidence class, and an implementation estimate.
A worked entry, in the shape the pack uses:
- Change: enable continuous batching in the serving runtime for the chat request class.
- Metric moved: cost-per-request and p95 latency on that class only.
- Expected delta: stated as a range, with the basis named — derived from the measured queue-wait share in the profiler traces, not from a vendor benchmark.
- Confidence: high / medium / low, where high means the profiler directly measured the wasted time the change removes, and low means the mechanism is plausible but the trace did not isolate it.
- Implementation cost: engineering days, including the regression-testing and rollout work, not just the config change.
- Risk note: what could regress — here, tail latency for single-request traffic during low-volume hours.
Confidence is doing real work in that list. A backlog where every item is “high confidence” has not been graded. Grading it honestly is what lets a platform lead fund the top two items and defer the rest without arguing about the whole roadmap. Whether any given item clears the funding bar is a separate calculation, and we treat that as its own decision rubric rather than folding it into the backlog itself.
How the ROI model stays checkable
The ROI model in the pack is not a forecast in the marketing sense. It is an arithmetic statement whose inputs are all visible, so that anyone can substitute the real post-change figures later and see whether the projection held.
Concretely, it states the baseline cost-per-request, the projected cost-per-request if the top-ranked backlog items ship, the sustained call volume the saving is multiplied across, the payback window, and the loaded engineering cost on the other side of the ledger. Every one of those inputs is a number from the baseline or the backlog, not a new assumption introduced at the ROI stage. That is the discipline: if the model needs an input that was never measured, the input gets flagged as an estimate rather than quietly averaged in.
The model also carries an avoided-cost line, and this is the one that most often dominates. When profiling shows the bottleneck sits in the runtime, the batching strategy, or the idle-capacity share rather than in the model, the audit has removed the case for a model-replacement programme — the integration work, the re-evaluation of output quality, the migration risk. Naming that avoided cost explicitly is more defensible than folding it into a headline savings percentage.
Re-running the audit is the closing mechanism. Because the baseline was captured with named metric definitions and a named traffic window, the same measurement can be repeated after each shipped change, producing an attributable before/after delta per item. Teams that skip this step end up in the position the audit was commissioned to avoid: a set of optimisations shipped, a bill that moved, and no way to say which change caused what. The four-week audit cadence sets aside the handover explicitly so the re-run instrumentation stays with the team rather than leaving with the auditor.
The profiler-findings section, incidentally, is not something the audit invents — it is the output of standard GPU profiling practice applied to the deployed path, and the traces and utilisation figures come from that discipline rather than from a bespoke methodology.
When the deliverable says the model really is the problem
Sometimes it does. Profiling occasionally shows a well-batched serving path, high utilisation, negligible idle capacity, and a forward pass that is simply expensive for the output quality it produces. In that case the deliverable says so, and says it with the same structure: the attribution table shows model compute dominating, the backlog’s top entry is a model or precision change with its own expected delta and implementation estimate, and the ROI model prices the replacement programme honestly rather than treating it as free.
An audit that can never conclude “replace the model” is not an audit. The point of the baseline and the attribution view is that when the conclusion does arrive, it arrives with evidence attached — which is a different object from the same conclusion reached by reading a monthly invoice. Where the audit’s scope stops and the buyer’s own execution starts is set out in what an inference cost audit is not.
The contents described here are what the Inference Cost-Cut Pack hands over, and the same evidence structure underpins the broader R&D and consulting engagements we run around production inference.
The open question in most engagements is not what the deliverable should contain. It is who owns the re-run after month three, when the traffic mix has drifted and the baseline no longer describes the system it was measured on.
Frequently Asked Questions
What does an inference cost audit deliverable contain in practice, and how does it connect to ROI?
What should you expect when the audit finishes? The mechanics of Inference Cost Audit Deliverable are worth stating plainly. For Inference Cost Audit Deliverable, it helps to be precise. The mechanics of Inference Cost Audit Deliverable are worth stating plainly. Inference Cost Audit Deliverable comes down to a few moving parts. For Inference Cost Audit Deliverable, it helps to be precise. The mechanics of Inference Cost Audit Deliverable are worth stating plainly. For Inference Cost Audit Deliverable, it helps to be precise. The mechanics of Inference Cost Audit Deliverable are worth stating plainly. Inference Cost Audit Deliverable comes down to a few moving parts. For Inference Cost Audit Deliverable, it helps to be precise. The mechanics of Inference Cost Audit Deliverable are worth stating plainly. For Inference Cost Audit Deliverable, it helps to be precise. The mechanics of Inference Cost Audit Deliverable are worth stating plainly. Asked about Inference Cost Audit Deliverable, most engineers point here. Inference Cost Audit Deliverable comes down to a few moving parts. When applied to What an Inference Cost Audit Deliverable, five sections: a measured baseline, a cost-attribution view across the serving path, profiler findings that name the bottleneck, a ranked optimisation backlog, and an ROI model. The ROI connection is mechanical rather than rhetorical — every input to the model is a number already stated in the baseline or the backlog, so the projection can be checked against real post-change figures., cost-per-request and cost-per-token (input and output stated separately), p95 and p99 latency, GPU utilisation, and idle-capacity share — each scoped to a named model, request class, hardware tier, and traffic window. The scoping matters as much as the numbers, because an unscoped aggregate drifts with traffic mix and cannot serve as a control.
How does a cost-attribution view assign spend across tokenisation, prefill, decode, routing, and idle capacity?
It splits the single inference cost line into one row per stage of the deployed serving path, so each stage carries a share of spend. Idle capacity is included as a row even though it is not a request stage, because reserved accelerator time that nothing occupied is often the largest single line on bursty workloads.
What does a ranked optimisation backlog entry look like — expected delta, confidence, and implementation cost?
A complete entry names the change, the baseline metric it moves, an expected delta stated as a range with its measurement basis, a confidence class, an implementation estimate in engineering days including rollout and regression testing, and a regression risk note. Confidence is graded honestly — high means the profiler directly measured the waste the change removes; low means the mechanism is plausible but was not isolated in the trace.
How is the ROI model calibrated so the projected savings can be checked after the work ships?
Every input is drawn from measurements already in the pack: baseline cost-per-request, projected post-change cost-per-request, sustained call volume, payback window, and loaded engineering cost. Anything that was not measured is flagged as an estimate rather than averaged in silently, so a reviewer can substitute actuals and see whether the projection held.
How do we re-run the audit after each change to prove the before/after delta?
Because the baseline was captured with explicit metric definitions and a named traffic window, the same instrumentation can be pointed at the system again after each shipped change. That produces an attributable per-item delta instead of one aggregate bill movement that no one can decompose.
What does the deliverable say when profiling shows model replacement genuinely is the right lever?
It says so directly, with the attribution table showing model compute dominating, a well-batched path, high utilisation and negligible idle capacity. The replacement then appears as the top backlog entry with its own expected delta and implementation estimate, and the ROI model prices the migration and re-evaluation work rather than treating the swap as free.
Building your deliverable around decisions, not data
Start with the three cost reduction levers your team can actually pull this quarter, then work backward to the metrics that matter. That answer is workload-specific, and it is worth writing down before you build.
