An inference cost audit ends with a measured baseline, a named bottleneck, a ranked optimisation roadmap and a calibrated ROI model. It does not end with a new model deployed in your serving path. That distinction is not a commercial hedge — it is the whole reason the audit is worth commissioning, and the engagements that go wrong are almost always the ones where the buyer read “audit” and heard “someone will make our model cheaper for us”.
The failure has a predictable shape. A team under cost pressure signs off the audit, then quietly defers its own optimisation planning until the report lands, because the report is assumed to be the fix. Four weeks later the evidence arrives, everyone agrees it is good evidence, and nobody has release capacity booked to act on it. The audit did its job. The handover failed anyway.
What “model replacement is out of scope” means in practice
It means the audit will investigate, measure and price a model change — and will not perform one.
Replacement is sometimes the correct lever. A smaller checkpoint, a distilled variant, or a different API tier can genuinely be the cheapest path to a lower cost-per-request. The audit exists to establish whether that is true before anyone pays for the integration, because a model swap is not a config change: it drags evaluation, regression testing, prompt or post-processing rework, and a re-baselining exercise behind it. In our experience those trailing costs are routinely larger than the model-line saving that motivated the swap (observed across TechnoLynx engagements; not a published benchmark).
So the boundary sits at the point where evidence becomes execution. Everything up to “here is what a replacement would have to beat, and what it would cost you to find out” is inside scope. Everything after — choosing the checkpoint, wiring it in, revalidating quality — belongs to the platform team, on their release calendar, with their ownership of the regression surface.
The scope boundary, line by line
| Concern | In scope for the audit | Out of scope |
|---|---|---|
| Baseline | Cost-per-request, cost-per-token, p95 latency, GPU utilisation over a defined traffic window | Ongoing production monitoring |
| Serving path | Profiling batching, queueing, tokenisation, caching, routing, runtime and kernel time | Rewriting the serving code |
| Quantisation | Measuring headroom, estimating saving and accuracy risk | Producing and validating the quantised artifact |
| Runtime / hardware | Naming candidate ports (ONNX Runtime, TensorRT, different accelerator tier) with expected effect | Executing the port |
| Model | Quantifying what a replacement must beat on cost and quality | Selecting, integrating or evaluating a replacement |
| Commercial | A calibrated ROI model per roadmap item | Cloud bill negotiation or committed-use purchasing |
| Delivery | Ranked roadmap with estimated effort and expected saving per item | Implementing the roadmap |
Read the right-hand column as a work list that stays with you, not as a list of things nobody will do. The point of separating them is that you can price the next step yourself, from numbers you can reproduce, rather than approving a change on somebody else’s confidence.
Why an evidence-only deliverable is the more useful one
The measurable outcome of a correctly scoped audit is scope-fit rather than speed. You get cost-per-request and p95 latency baselines, GPU utilisation, a bottleneck map, and a ranked roadmap where each item carries an effort estimate and an expected saving. That is enough to sequence batching, caching, routing, quantisation or runtime work against your own release train — and enough to decline three of the seven items because the payback window is longer than your planning horizon.
The avoided cost is specific: a model-replacement project started on a guess. Integration, evaluation and regression testing on a swap that profiling would have shown to be irrelevant is the single most expensive way to discover that the model was never the constraint. We see teams arrive at that discovery late, usually after the new checkpoint is already in staging.
There is a second reason the boundary holds. An audit that also implements has an incentive problem — the party measuring the bottleneck is the party quoting for the fix. Keeping the deliverable to evidence means the roadmap can honestly rank “do nothing on the model” first, which it often should. We explore how that audit is structured week by week, and what its exit criteria are, in our four-week inference cost audit cadence; the Inference Cost-Cut Pack is the packaged form of the same engagement, and its deliverable list stops exactly where the table above says it does.
How this differs from a FinOps review
Both look at a bill. They look at different things in it.
A cloud FinOps review or a bill-negotiation exercise works on the commercial layer: instance families, reserved capacity, committed-use discounts, waste from idle resources, tagging hygiene. It can reduce what you pay for the same compute. An inference cost audit works on the technical layer: how much compute a request actually consumes, and why. It can reduce the compute itself.
They compose well and answer to different owners. Negotiating a 20% rate reduction on a serving path that is 40% idle because batching is misconfigured leaves most of the money on the table — and the reverse is also true, which is why the audit’s ROI model is expressed in cost-per-request rather than in monthly spend. The unit-economics framing behind that choice sits with our work on AI infrastructure unit economics, which defines the KPI the audit reports against.
Sequencing against a migration you have already planned
If a model migration is already committed, run the audit first — but shorten it. The useful output in that situation is not a full roadmap; it is a baseline good enough that the migration can be judged against it afterwards. Without one, the migration will be declared a success on the model line while total cost-per-request stays flat, and nobody will be able to prove otherwise.
If the migration is planned but not committed, the audit is the cheaper decision. It costs a few engineering weeks to learn whether the checkpoint is the constraint. It costs a quarter to learn the same thing by shipping.
Frequently Asked Questions
What does “model replacement is out of scope” mean in practice for an inference cost audit?
Model replacement falls outside the scope of a proper inference cost audit. Inference Cost Audit comes down to a few moving parts. Inference Cost Audit is one of those terms that hides a simple idea. Inference Cost Audit comes down to a few moving parts. The mechanics of Inference Cost Audit are worth stating plainly. Inference Cost Audit is one of those terms that hides a simple idea. Inference Cost Audit comes down to a few moving parts. Inference Cost Audit is one of those terms that hides a simple idea. Inference Cost Audit comes down to a few moving parts. The mechanics of Inference Cost Audit are worth stating plainly. Inference Cost Audit is one of those terms that hides a simple idea. Inference Cost Audit comes down to a few moving parts. Inference Cost Audit is one of those terms that hides a simple idea. Inference Cost Audit comes down to a few moving parts. Inference Cost Audit is simpler than it looks. The mechanics of Inference Cost Audit are worth stating plainly. It means the audit measures, profiles and prices a possible model change but does not carry one out. Selecting a checkpoint, integrating it, and revalidating output quality stay with your platform team, because that work owns a regression surface only you can accept. The audit’s job is to tell you whether the model is worth touching at all.
What exactly is handed over at the end of the audit, and in what form?
A measured baseline (cost-per-request, cost-per-token, p95 latency, GPU utilisation over a defined traffic window), a cost-attribution view across the serving path, a named bottleneck, and a ranked optimisation roadmap where each item carries an effort estimate and an expected saving. The ROI model is calibrated to your own call volume, so you can price the next step without re-deriving the numbers.
If profiling shows the model really is the bottleneck, what does the audit produce instead of a replacement?
A target. The audit quantifies what a replacement would need to beat — on cost-per-request, on p95 latency, and on the quality threshold your traffic requires — plus an estimate of the integration, evaluation and regression-testing effort that replacement would carry. That turns a model swap from a hunch into a costed proposal you can approve or reject.
Which serving-path levers are in scope, and which are not?
In scope for investigation: batching strategy, queueing, tokenisation, caching, request routing, quantisation headroom, runtime choice, and kernel-level time. Out of scope for execution: all of them. The audit measures each lever’s contribution and expected saving; implementing the chosen ones is platform-team work, sequenced against your release calendar.
Who owns implementing the ranked optimisation roadmap after handover?
Your engineering team, and that is deliberate. The roadmap is ordered so it can be absorbed incrementally rather than as a single project, and items whose payback window exceeds your planning horizon can be declined without unravelling the rest. Booking release capacity before the report lands is the single change that most improves handover.
How is an inference cost audit different from a cloud FinOps review?
A FinOps review reduces the price you pay for compute — instance mix, committed-use discounts, idle waste. An inference cost audit reduces the compute a request consumes. They are complementary, but a rate discount on a badly batched serving path leaves most of the available saving untouched.
How should we sequence an audit against an already-planned model migration?
Run a shortened audit first, purely to establish a reproducible baseline. Otherwise the migration gets judged on the model line alone, and a flat total cost-per-request will be indistinguishable from a win. If the migration is planned but not yet committed, run the full audit — a few engineering weeks is a cheaper way to test the assumption than a quarter of integration work.
If the boundary still feels restrictive, the question worth asking your prospective auditor is a simple one: what does your deliverable say when the honest answer is do nothing to the model? An engagement that cannot produce that answer was never measuring anything. Scoping and delivery detail for this kind of work sits with our R&D and consulting services.
Cost audits versus model selection: scope boundaries
An inference cost audit quantifies waste in serving infrastructure—batching gaps, idle GPU time, over-provisioned replicas—but it does not compare alternative model architectures or recommend distillation; that belongs in a separate model-selection exercise. Revisit it when your workload shifts.
