An inference cost audit that runs “until we find something” tends to do one of two things: never finish, or finish with a document nobody will sign. The fix is boring and structural — put a clock on it, and give each week an exit condition that has to be true before the next week starts. Four weeks: instrument, profile, rank, hand off.
The point of the box is not speed. It is comparability. A four-week cadence bounds the diagnostic spend so it can be weighed against the cost of the optimisation or replacement project it informs, and it forces every finding to arrive in a form the platform team can re-run after we leave.
What does a four-week inference cost audit look like, week by week?
Each week produces one artifact and one gate. If the gate does not close, the week extends or the audit re-scopes — it does not roll forward on a shaky foundation.
| Week | Work | Output | Exit criterion |
|---|---|---|---|
| 1 — Instrumentation | Wire cost, latency and utilisation telemetry into the deployed serving path | Reproducible baseline: cost-per-request, p95 latency, GPU utilisation over a defined traffic window | The baseline can be regenerated by the client team without us, and two independent runs agree |
| 2 — Profiling | Profile the instrumented path by stage — queueing, batching, tokenisation, prefill/decode, kernels, post-processing | Bottleneck map attributing time and spend to named stages | One dominant constraint is named, with evidence, not a list of suspicions |
| 3 — Recommendations | Prototype or bound the candidate levers; re-measure against the week-one baseline | Ranked optimisation roadmap with a calibrated ROI model per item | Each ranked item carries a measured or bounded delta and an engineering-cost estimate |
| 4 — Handoff | Transfer dashboards, profiler configs, baseline scripts and the ROI model to the owning team | Decision document plus live instrumentation | A named owner on the platform team can reproduce the baseline and re-run the comparison |
Week one is the divergence point
The practical side of Run Four Week AI starts here. If you cannot say what a request costs today, at what p95, at what utilisation, then a 30% improvement in week three is a number with no denominator.
Instrumentation week is unglamorous. It means attributing spend per model, per batch size, per hardware tier and per request class rather than dividing the monthly GPU bill by the request count — we cover the mechanics of that in measuring inference cost-per-request before any optimisation. It means pinning a traffic window that is representative rather than convenient. It usually means discovering that the serving path has a stage nobody owns.
The gate is reproducibility, not completeness. Two runs of the baseline on comparable traffic should agree closely enough that a later delta is legible. If they do not, week one is not finished, and extending it is cheaper than discovering the problem in week three.
Week two: profiling the deployed path, not a benchmark rig
Profiling week takes the instrumented path and asks where the time and money actually go. The stages worth separating are request queueing, batching behaviour, tokenisation, prefill versus decode, runtime and kernel execution, memory transfer, and post-processing. On GPU-served workloads this is standard profiling methodology — NVIDIA Nsight Systems for kernel timelines, PyTorch’s profiler for operator-level attribution, framework-level traces for the request path — applied to the production serving stack rather than a synthetic harness.
The discipline here is resolution. An averaged latency figure hides padding waste and tail behaviour that a stage-level profile exposes immediately. What we are looking for by end of week is a defensible sentence of the form “X% of realised cost-per-request is attributable to stage Y under this traffic mix” — not a ranked list of things that looked suboptimal.
If profiling week ends without a dominant constraint, that is a finding, not a failure. Some deployments are genuinely flat, with cost spread evenly across stages. That result changes the week-three roadmap: flat cost profiles usually point at architecture or hardware-tier decisions rather than targeted optimisation.
Week three: ranking, and what makes an ROI model calibrated
Recommendations week converts the bottleneck map into a ranked roadmap. Ranking is by expected saving per engineering week, not by size of the measured inefficiency — a 40% headroom on a stage that costs 5% of the bill ranks below a 10% improvement on the stage that costs 60%.
An ROI model is calibrated when its inputs are measured rather than assumed:
- The cost-per-request delta comes from re-measuring a candidate lever against the week-one baseline, not from a vendor’s published speedup.
- Call volume comes from the same traffic window the baseline used, with an explicit assumption about growth.
- Engineering cost is loaded hours plus regression testing and rollout, not the happy-path implementation estimate.
- Quality impact is bounded where a lever touches numerics or model selection — quantisation and routing both need an accuracy floor stated before the saving is counted.
Where a lever cannot be prototyped inside the week, it is bounded rather than estimated: a stated best case and worst case, with the measurement that would collapse the range. That is honest, and it keeps the roadmap falsifiable.
Model replacement is allowed to appear on this roadmap. The audit does not presuppose that the serving path is the answer — it presupposes that the serving path is measured first. Sometimes the evidence says the checkpoint is the constraint, and the ranked roadmap says so with numbers behind it.
Week four: the handoff that keeps the evidence alive
The failure mode we see most often in diagnostic engagements is decay — the findings live in a slide deck, the dashboards were built on a consultant’s laptop, and six months later nobody can reproduce the baseline to check whether the optimisation worked. Handoff week exists to prevent that.
What transfers: the instrumentation itself (dashboards, exporters, cost-attribution queries), the profiler configurations used in week two, the baseline capture scripts, the bottleneck map, and the ROI model as a working spreadsheet or notebook with its inputs exposed. What also transfers is an owner — a named person on the platform team who has run the baseline once, with us watching, before the engagement closes.
That handoff is the deliverable shape behind the [inference cost audit pack](Inference Cost-Cut Pack), and the same principle applies to how we scope R&D engagements generally: the client team should be able to re-run the measurement without us.
When four weeks is the wrong shape
Four weeks fits a single deployed serving path with existing observability and a team that can grant access in the first days. It is the wrong shape in three recognisable cases.
- Too long: a single-model deployment with a suspected obvious constraint and good existing telemetry. A one-week scoping pass may produce the same decision at a quarter of the cost.
- Too short: a multi-tenant platform serving many model families, or a pipeline where pre- and post-processing sit in a different team’s codebase. Instrumentation alone can consume the four weeks.
- Wrong instrument entirely: if the organisation has not decided whether it will fund any optimisation work, the audit produces a decision document into a vacuum. Settle the decision rubric on whether optimisation justifies the spend first.
A time-boxed audit does not guarantee a saving. It guarantees a decision document that names which lever to pull and what it is worth — including the case where the answer is “do nothing yet”.
Frequently Asked Questions
What does running a four-week AI inference cost audit mean in practice, week by week?
Instrument your serving stack during week one to lock in reproducible measurements: cost-per-request, p95 latency, and GPU utilisation form your audit baseline. Week two profiles that instrumented path stage by stage to name the dominant bottleneck. Week three converts findings into a ranked optimisation roadmap with a calibrated ROI model. Week four transfers the instrumentation, evidence and model to a named owner on the platform team.
What has to be instrumented in week one before profiling can produce trustworthy findings?
Cost attributed per model, per batch size, per hardware tier and per request class, captured alongside p95 latency and GPU utilisation across a defined, representative traffic window. The gate is reproducibility: the client team must be able to regenerate the baseline independently, and two runs on comparable traffic must agree. Without that, a later improvement cannot be attributed to anything.
What does the profiling week actually look at?
It separates the serving path into request queueing, batching behaviour, tokenisation, prefill versus decode, runtime and kernel execution, memory transfer, and post-processing — then attributes realised time and spend to those stages. Batching, caching, routing, quantisation, runtime and kernel work are all candidate levers, but week two’s job is to establish which stage the money is in, not to pick a lever.
How do week-three recommendations get ranked, and what makes the ROI model calibrated rather than assumed?
Ranking is by expected saving per engineering week, so a large inefficiency on a cheap stage ranks below a modest one on the dominant stage. A model is calibrated when the cost delta is re-measured against the week-one baseline rather than taken from vendor claims, volume comes from the measured traffic window, engineering cost is loaded and includes regression testing, and any lever touching numerics carries a stated accuracy floor.
What is handed over in week four, and who on the platform team has to own it afterwards?
Dashboards and cost-attribution queries, profiler configurations, baseline capture scripts, the bottleneck map, and the ROI model with its inputs exposed. A named platform engineer runs the baseline once before the engagement closes, with us present, so the evidence survives as live instrumentation rather than a slide deck.
What are the exit criteria for each week, and what should happen if a week’s criteria are not met?
Week one closes on a reproducible baseline, week two on a named dominant constraint with evidence, week three on ranked items each carrying a measured or bounded delta plus an engineering-cost estimate, week four on a client-run baseline reproduction. If a gate does not close, the week extends or the audit re-scopes — rolling forward on an unverified baseline invalidates everything downstream.
When is four weeks the wrong shape?
It is too long for a single-model deployment with strong existing telemetry and one suspected constraint, where a one-week scoping pass may reach the same decision. It is too short for multi-tenant platforms or pipelines whose pre- and post-processing live in another team’s codebase, where instrumentation alone can absorb the four weeks. And it is the wrong instrument entirely if no optimisation work would be funded regardless of what the audit finds.
Four weeks: enough time to find the expensive surprises
Week one uncovers the baseline, week two finds the outliers, week three tests hypotheses, and week four documents the plan—or proves there isn’t a problem worth solving.
