The GPU invoice arrives as one number. Divide it by request count and you get a cost-per-request that looks like a property of the model — which is exactly why so many cost programmes begin and end with model replacement. It is the only component in the stack everyone can name.
The bill is separable, though, and separating it changes which levers are visible. Three buckets do the work: model compute (what the forward pass actually costs on the accelerator), serving path (batching, queueing, tokenisation, pre- and post-processing, network hops, cache behaviour), and platform overhead (idle GPU time, over-provisioned replicas, autoscaling lag, logging and observability, egress, orchestration). In deployed serving paths that reach an audit, model compute is frequently not the largest of the three — idle capacity and serving-path inefficiency often are. That is the claim worth testing on your own numbers before anyone signs a migration project.
What belongs in which bucket?
Attribution fails most often at the boundaries, so the boundaries need to be stated rather than assumed. The taxonomy below is the one we use to structure a baseline; the measurement source column matters as much as the line item, because a bucket without a named source is an estimate wearing a percentage sign.
| Bucket | Line items | Typical measurement source |
|---|---|---|
| Model compute | Forward-pass kernel time, prefill and decode GPU time, attention and GEMM kernels, weight and activation memory traffic | Kernel-level profiler (Nsight Systems, PyTorch profiler, TensorRT layer timings) |
| Serving path | Batch occupancy and padding waste, queue wait, tokenisation, image decode and resize, serialisation, retrieval or tool calls, cache hit rate, inter-service hops | Request traces plus server metrics (Triton / vLLM / TGI batch and queue counters, OpenTelemetry spans) |
| Platform overhead | Paid-but-idle GPU-hours, replica count above measured demand, autoscaling lag and warm pools, log and metric shipping, egress, orchestration control plane | Cloud billing joined to scheduler and autoscaler records (Kubernetes requests vs usage, DCGM utilisation) |
Two rules keep the split honest. First, GPU time that is paid for but not executing a forward pass is overhead, not model compute — this is where the “our model is expensive” narrative usually comes from. Second, work done on the host to prepare a batch belongs to the serving path even when it blocks the GPU; a starved accelerator is a serving-path defect that shows up on the model’s line.
Turning the taxonomy into measured shares
You rarely need new tooling to get a first pass. Paid GPU-hours come from the invoice; achieved utilisation comes from DCGM or the cloud’s own GPU metrics; the ratio between them gives the idle share directly. Batch occupancy and queue wait come from the inference server’s existing counters. Kernel time comes from a short profiling window on representative traffic rather than a continuous trace.
A defensible breakdown has four properties:
- Every bucket carries a percentage share and the source that produced it.
- The traffic window is defined and representative — a Tuesday afternoon is not a month.
- Cost-per-request (and cost-per-token, for LLM workloads) reconciles back to the invoice within a stated tolerance.
- Any bucket the current instrumentation cannot resolve is marked unmeasured rather than back-filled with an assumption.
That last point is the one finance reviewers respect most. Saying “we cannot yet separate retrieval latency from decode time, so 18% of spend is unattributed” is a stronger position than a tidy pie chart built on a guess.
For LLM deployments the mapping is direct but asymmetric. Cost-per-token splits into prefill and decode on the model line, prompt construction and retrieval on the serving line, and idle decode capacity between bursts on the overhead line. Output length is a serving-path decision — a generation cap is a cost control, not a model change. Where a full walkthrough is useful, the parent methodology’s inference cost audit methodology governs how profiling is sequenced and what the resulting roadmap has to contain; this taxonomy is the finance-facing view of the same evidence. The cost-per-request measurement baseline is the prerequisite — attribution without a reproducible denominator is arithmetic on sand.
Presenting it so a reviewer can decide
A breakdown earns its keep when it converts into a ranked set of options with payback periods attached. The reviewer is not choosing between architectures; they are choosing between engineering weeks. So each bucket should arrive with the shortest-payback intervention that touches it: right-sizing replicas and consolidating models onto shared accelerators for overhead, batching and cache work for the serving path, quantisation or runtime porting for model compute.
In our experience the sequencing argument sells itself once the shares are visible. When idle capacity is 40% of spend, no reviewer approves a six-week quantisation project first — and no engineer has to argue for the boring fix, because the line item does it for them. The [inference cost audit pack](Inference Cost-Cut Pack) is structured around exactly this breakdown, and the broader R&D engagement scoping that surrounds it treats the attributed baseline as the entry condition rather than a deliverable to be negotiated later.
One honest caveat: the three-bucket split is a reporting frame, not a physical partition. Batching sits between the serving path and model compute; an autoscaling policy sits between overhead and serving. Where a cost genuinely straddles two buckets, assign it to the one whose owner can change it, and say so in a footnote. Attribution is useful because it points at a decision-maker, not because it is metaphysically clean.
Frequently Asked Questions
What does “where inference spend actually lands — model vs serving vs overhead” mean in practice? Stripped down, Inference Spend Actually Lands is the following. Asked about Inference Spend Actually Lands, most engineers point here. Inference Spend Actually Lands is one of those terms that hides a simple idea. It means decomposing a single GPU invoice into three separately measurable buckets instead of treating the whole bill as the model’s cost. Model compute is forward-pass time on the accelerator; the serving path is everything around it that shapes how efficiently that time is used; platform overhead is capacity paid for but not used for inference. Each bucket has different owners and different levers.
Which line items belong in model compute, which in the serving path, and which are platform overhead? Kernel execution, prefill and decode GPU time, and weight or activation memory traffic are model compute. Batching and padding, queue wait, tokenisation, image decode, retrieval calls, caching, and service hops are the serving path. Idle GPU-hours, replicas above measured demand, autoscaling lag, log shipping, egress, and orchestration are overhead. Host-side work that starves the GPU belongs to the serving path, not the model.
How do we measure each bucket’s share of cost-per-request with the instrumentation we already have? Most teams can get a first pass without new tooling: the invoice gives paid GPU-hours, DCGM or cloud GPU metrics give achieved utilisation, and the gap is the idle share. Inference-server counters supply batch occupancy and queue wait; a short profiling window on representative traffic supplies kernel time. Anything those sources cannot resolve is reported as unattributed.
How much of our GPU bill is idle or over-provisioned capacity rather than inference work? That is a direct calculation, not an estimate: compare paid GPU-hours against GPU-hours actually executing forward passes over a defined window, then add replicas running above measured demand. In serving paths that reach an audit, this bucket is frequently larger than the model’s own compute — which is why we measure it before discussing model changes.
For an LLM deployment, how does cost-per-token map onto this three-bucket breakdown? Prefill and decode GPU time land on the model line. Prompt construction, retrieval, tokenisation, continuous-batching efficiency, and KV-cache reuse land on the serving line. Capacity held idle between traffic bursts lands on overhead. Output token length is a serving-path control, so a generation cap is a cost lever that requires no model change.
What does the breakdown look like when serving-path inefficiency, not the model, is the dominant cost? It shows low batch occupancy or heavy padding waste, a poor cache hit rate, or significant queue wait, while kernel time per request is unremarkable. The signature is a GPU that is busy but not productive — occupied by padded batches or starved by host-side pre-processing. The remedy is batching and caching work, not a smaller checkpoint.
How do we present this breakdown so a finance reviewer can approve or reject an optimisation on payback period? Give each bucket a measured percentage share, a named measurement source, and the shortest-payback intervention that touches it, priced in loaded engineering hours against projected saving over a stated window. The reviewer then compares line items rather than opinions, and can reject an option because the payback is too long — which is a legitimate outcome, not a failed audit.
Breaking down your invoice by component
Model compute rarely accounts for more than 60% of total inference spend—the remaining budget vanishes into serialization, network hops, and orchestration overhead. Revisit it when your workload shifts.
