A production AI feature does not become unsustainable when the monthly cloud bill goes up. It becomes unsustainable the moment cost-per-request starts growing faster than revenue-per-request — and a single line item on a cloud invoice will never tell you when that crossover happened. That is the decision this article is about: what optimisation target a production AI team should actually set. The default answer — watch cloud spend, renegotiate the vendor contract when it gets uncomfortable — is the wrong target. Not because cloud spend doesn’t matter, but because it is measured at the wrong granularity to protect the thing you care about, which is the margin on each AI-powered interaction your product serves. We work with teams who are confident their inference costs are under control because the finance dashboard is flat month over month. Then a model swap, a context-window change, or a 3x traffic spike turns a healthy feature into a loss-leader, and nobody saw it coming because the aggregate number moved slowly while the per-request number moved fast. The aggregate hid the structural problem. Why Generic Cloud Spend Is the Wrong Target for AI Workloads Cloud spend is a FinOps metric. It answers “how much are we paying the vendor this month,” and it responds to FinOps levers: reserved capacity, committed-use discounts, rightsizing idle resources, renegotiating rates. Those are real savings and worth pursuing. But none of them tell you whether a specific AI feature is economically viable. Here is the test we use to separate the two. A cost programme that survives a model swap without changing your per-request economics was never a production-AI cost programme — it was a FinOps initiative. If you switch from one model to another with twice the parameter count and your optimisation target doesn’t register the change, your target is measuring the wrong thing. The reason is structural. AI inference cost is dominated by per-request compute: tokens generated, sequence length, batch occupancy, the precision the model runs at, and how well the serving path keeps the accelerator busy. A single line-item cloud total averages all of that across every feature, every traffic pattern, and every off-peak idle hour. Two features can share an identical monthly bill while one runs at a 70% gross margin and the other is underwater on every call. The aggregate cannot distinguish them. This is the same root-cause pattern we see in why most enterprise AI projects fail: the organisation optimises a proxy metric that is easy to measure instead of the workload-level metric that actually governs the outcome. Cloud spend is the easy proxy. Cost-per-request is the workload-level truth. How Do You Measure Cost-Per-Request for an AI Feature? Cost-per-request is the fully-loaded compute cost of serving one user-facing AI interaction. The honest version includes more than the raw GPU-second: Compute time on the accelerator — the dominant term for most LLM and vision workloads, driven by tokens processed and how efficiently the serving runtime batches. Idle and under-utilisation overhead — capacity you provisioned but didn’t saturate. A request that runs on a GPU at 30% utilisation carries the cost of the 70% you paid for and didn’t use. Supporting infrastructure — vector store reads for retrieval-augmented generation, embedding calls, orchestration, network egress. Retries and fallbacks — a request that fails validation and re-runs costs you twice for one delivered result. You cannot derive any of this from the cloud invoice. It requires profiling the deployed serving path — measuring where time and memory actually go per request, not where you assume they go. Profile-first measurement is the methodology that underwrites the entire unit-economics framing; for the engineering mechanics of that profiling step, see how we approach GPU performance profiling and optimisation for production serving paths. Cost-per-token is the finer-grained sibling. For generative workloads, most of the cost-per-request variance comes from how many tokens a request consumes — so cost-per-token (input and output priced separately, because output generation is typically the expensive half) is the lever you tune, and cost-per-request is the KPI the product margin actually depends on. You set the target in cost-per-request because that’s what maps to a billable interaction; you optimise in cost-per-token because that’s where the compute goes. The unit economics of production AI explainer walks through that relationship in more detail. A Worked Cost-Per-Request Example (Explicit Assumptions) The numbers below are illustrative — they show the arithmetic, not a benchmarked rate for any specific deployment. Assume a customer-support summarisation feature: Average request: 1,200 input tokens, 300 output tokens. Serving on a GPU instance you’ve measured at a sustained 18 requests/second under realistic load (a profiled observed-pattern figure, not a spec sheet peak). Instance cost: on the order of $3/hour for the accelerator and supporting node. That is roughly $3 ÷ (18 × 3,600) ≈ $0.000046 of raw compute per request. Add idle headroom (say you provision for 2x peak, so half your capacity is unused at average load) and the effective compute cost roughly doubles to ~$0.0001. Add a 5% retry rate and RAG retrieval overhead, and you land in the rough vicinity of $0.00012 per delivered request. Now the decision becomes legible: if that feature is bundled into a plan that earns $0.002 of attributable revenue per support interaction, you have healthy headroom. If a model upgrade triples output tokens and halves throughput, cost-per-request jumps toward $0.0005 — still positive, but the margin compression is now visible at the moment it happens, not three months later in a slow-moving invoice. Cloud-Spend Target vs Cost-Per-Request Target Dimension Cloud-spend target (FinOps) Cost-per-request target (production AI) What it measures Total vendor invoice over a period Fully-loaded compute cost of one served interaction Granularity Account / project aggregate Per workload, per feature Maps to Cash outflow Product gross margin Survives a model swap No — change is averaged away Yes — registers immediately Primary levers Reserved capacity, rate negotiation, rightsizing Tokens, batching, precision, utilisation, retries Early-warning value Low — slow-moving aggregate High — crosses an SLO threshold per feature Who owns it Finance / FinOps ML platform / VP Eng Both columns are legitimate. The point is not that FinOps is wrong — it’s that FinOps is the wrong optimisation target for a production AI feature’s viability. They answer different questions and you need both. When Does an AI Feature Cross From Acceptable to Unsustainable? The crossover is not a dollar amount. It’s the point where cost-per-request grows faster than revenue-per-request, sustained over a real traffic window. Three things commonly trigger it: A model swap — adopting a larger or more capable model for quality reasons, without re-checking that the per-request economics still close. The quality win is visible in evaluations; the cost regression hides in the aggregate until traffic scales. A usage-pattern shift — users discover the feature and start sending longer prompts, or invoking it more often per session. Output tokens climb, and because output generation dominates cost-per-token, the per-request cost climbs with them. A provider or pricing change — moving between hosted API providers, or a provider repricing input versus output tokens. AI API pricing differs not just in headline rate but in how input and output are priced relative to each other, so the same workload can have meaningfully different cost-per-request on two providers even at similar advertised rates. Your target has to be set against the provider and pricing structure you actually run on. The gross-margin impact of getting this wrong is not marginal. When a feature with thin attributable revenue runs underwater on every call, scaling it increases losses — growth becomes the enemy. Across engagements where teams discovered the problem late, the recurring damage was the same shape: the feature that was supposed to drive expansion revenue was quietly eroding the margin it was meant to grow (observed-pattern; not a benchmarked figure). How Do You Set a Cost-Per-Request SLO? Treat cost-per-request like a latency SLO: a measured threshold, owned by the platform team, that gates what ships and what scales. A workable sequence: Establish attributable revenue-per-request for the feature. You cannot set a viability threshold without knowing what one interaction is worth to the business. Profile the current serving path to get an honest fully-loaded cost-per-request — including idle overhead and retries, not just the GPU-second. Set the SLO as a fraction of revenue-per-request that preserves your target gross margin, and pin a companion p95-latency budget — because the cheapest configuration that blows the latency budget isn’t actually a valid option. Wire it into the release gate. A model swap or config change that pushes cost-per-request past the SLO doesn’t ship until it’s brought back under, the same way you’d block a regression in accuracy or latency. This is where the cost target connects to broader release-readiness decisions for AI features. Re-measure on every change to the model, the prompt template, or the traffic mix — the three variables that move per-request economics fastest. The SLO is what turns a vague “keep costs down” into a concrete, workload-anchored target that a model swap cannot silently violate. FAQ Why is generic cloud spend the wrong target for AI workloads? Cloud spend is a FinOps aggregate that averages every feature, traffic pattern, and idle hour into one number, so it responds to rate negotiation and rightsizing but cannot tell you whether a specific AI feature is economically viable. Two features can share an identical monthly bill while one is profitable and the other loses money on every call. A cost programme that survives a model swap without changing your per-request economics was a FinOps initiative, not a production-AI cost programme. How do we measure cost-per-request for an AI feature? Cost-per-request is the fully-loaded compute cost of serving one user-facing interaction: accelerator compute time, idle and under-utilisation overhead, supporting infrastructure like RAG retrieval and embeddings, and the cost of retries. You cannot derive it from the cloud invoice — it requires profiling the deployed serving path to measure where time and memory actually go per request. What gross-margin impact does poor inference economics have? When a feature with thin attributable revenue runs underwater on every call, scaling it increases losses rather than revenue, so growth erodes margin instead of building it. The damage is often invisible in the aggregate cloud bill while it is acute at the per-request level, which is why teams discover it late. When do AI features cross the threshold from acceptable to unsustainable? The crossover is not a fixed dollar amount — it is the point where cost-per-request grows faster than revenue-per-request over a real traffic window. It is most commonly triggered by a model swap, a usage-pattern shift toward longer or more frequent prompts, or a provider/pricing change. How do we set a cost-per-request SLO? Establish attributable revenue-per-request, profile the current serving path for an honest fully-loaded cost, then set the SLO as the fraction of revenue that preserves your target gross margin, pinned with a companion p95-latency budget. Wire it into the release gate so a model or config change that breaches the threshold doesn’t ship until it’s brought back under, and re-measure on every change to the model, prompt template, or traffic mix. How does cost-per-token relate to cost-per-request when setting a unit-economics target? Cost-per-token is the lever you tune — most cost-per-request variance for generative workloads comes from token count, with output generation typically the expensive half. Cost-per-request is the KPI the product margin actually depends on, because it maps to a billable interaction. You set the target in cost-per-request and optimise in cost-per-token. How do AI API pricing differences affect the cost-per-request target for a production feature? Providers differ not only in headline rate but in how they price input versus output tokens relative to each other, so the same workload can land at a meaningfully different cost-per-request on two providers even at similar advertised rates. The target therefore has to be set against the specific provider and pricing structure you actually run on, and re-checked whenever you switch providers or a provider reprices. Setting the Target Is the Start, Not the Finish Choosing cost-per-request over cloud spend is the decision; applying it to a deployed serving path is the work. The threshold tells you whether a feature is viable — it doesn’t tell you where the cost is going or which lever to pull, which is where profiling, batching, precision, and serving-path engineering come in. If you want to see cost-per-request and cost-per-token compared across real configurations rather than in the abstract, the inference benchmarking examples make the comparison concrete. For teams already past the decision and ready to act on a deployed path, the Inference Cost-Cut Pack applies this framing to your serving stack: profile, set the cost-per-request SLO, and close the gap. The named failure class here is the FinOps-disguised-as-AI-cost-programme — a target that averages away the one signal that predicts when a feature stops paying for itself.
