Cost-per-request is the right first KPI, and it should stay in place. It stops being sufficient the moment a single customer-facing action stops mapping to a single model call. Once an agentic loop, a reranking pass, a multi-turn resolution flow, or a retry policy sits between the user and the model, cost-per-request measures the engineering path while your margin lives one level up — at the ticket, the report, the completed session.
That gap is where cost programmes quietly fail. A team runs an optimisation pass, watches cost-per-request drop by a third, and reports a win. Meanwhile the average number of calls per resolved support ticket has drifted from four to nine, because a new planner step retries on low-confidence answers. Per-call cost fell. Cost per resolved ticket rose. Both numbers are correct, and only one of them is the number Finance is going to ask about.
What does graduating to unit economics per business action actually change?
It does not mean retiring cost-per-request. Cost-per-request stays as the engineering control variable; cost-per-business-action becomes the margin-facing KPI, and a measured fan-out ratio links the two. That is the whole graduation, and it is deliberately additive — the per-call number is what an engineer can move with batching, caching, quantisation or a compiler pass, and it survives a model swap because it is anchored to the workload rather than to a vendor’s price list.
The rollup itself is arithmetic:
cost-per-business-action = cost-per-request × calls-per-action
The hard part is the second term. If you cannot state today how many model calls a single billable customer action consumes, a falling cost-per-request tells you nothing about product margin. That sentence is the test. It is also, in our experience, the question that most cost dashboards cannot answer without a week of instrumentation work.
The divergence signals
Not every workload needs the second metric. A single-shot classifier behind a form field has a fan-out of one, and adding a per-action KPI just duplicates the per-request one with extra reporting overhead. The decision is about workload shape, not about maturity.
| Signal in your workload | Stay on cost-per-request | Add cost-per-business-action |
|---|---|---|
| Calls per customer action | Fixed at one | Variable, or >1 on average |
| Agentic / planner loops | None | Present, with data-dependent iteration counts |
| Retrieval + rerank hops | None or fixed | Optional hops gated on confidence |
| Retry policy | Client-side, rare | Server-side, automatic, cost-bearing |
| Who asks for the number | Platform team | Finance, pricing, or a per-seat margin review |
| What the product bills for | API calls | Tickets, documents, sessions, outcomes |
| Effect of an optimisation pass | Directly visible in margin | Can be fully cancelled by fan-out drift |
Two or more rows in the right-hand column is the graduation point. One row is a watch item.
Defining the action without inventing a vanity metric
The failure mode here is picking a unit that flatters the dashboard. “Interactions served” is not a business action. The discipline is simple: the business action is the unit your product already bills, contracts, or reports on to a customer. If your pricing page says “per resolved ticket”, the action is a resolved ticket. If your contract counts generated documents, the action is a generated document. If nothing external counts anything, you do not have a business action yet — you have a usage metric, and cost-per-request remains the only honest number you have.
Two constraints keep the definition usable:
- The action must have a countable completion event in your telemetry, not just a start event. Abandoned sessions consume calls and must land somewhere.
- The action must be attributable to a single feature or product surface. An action that spans three features produces a cost number no one owns.
Measuring fan-out rather than estimating it
Fan-out is measured on the deployed serving path, not derived from an architecture diagram. The practical approach is to propagate a single action identifier from the ingress that starts the customer action through every downstream model call — embedding, retrieval, rerank, generation, guardrail, summarisation — and to attach it to the same trace that already carries token counts and GPU-time counters. In a typical stack that means an OpenTelemetry baggage field surviving across service hops, with the counters emitted alongside spans rather than in a separate metrics pipeline where they cannot be joined back to the action.
Once the identifier is in place, three numbers become available: mean calls per action, the 90th-percentile fan-out, and the distribution’s shape. Teams that instrument this typically find the largest cost driver is fan-out variance rather than per-call price (an observed pattern across our inference-cost work, not a benchmarked rate). The mean tells you what the feature costs. The p90 tells you which actions are unsustainable — the long resolution loops, the documents that trigger six regeneration passes, the sessions where the planner never converges. Reporting p95 latency and cost at the 90th-percentile fan-out case is where those hide.
The audit work behind this is the same profiling exercise that underwrites the per-call figure in the first place; we develop how a per-request measurement of the deployed serving path is produced in our [inference cost reduction pack](Inference Cost-Cut Pack), and the broader margin framing for AI-native platforms sits with our work on AI infrastructure and SaaS economics. The parent argument — why per-request unit economics is the correct starting frame at all — is developed in our treatment of unit economics for production AI.
What each side owns afterwards
Splitting the metric splits accountability, and that is a feature. Engineering owns cost-per-request and defends it against regressions in batching, cache hit rate, padding waste and idle GPU time. Product and finance own cost-per-business-action, which they move by changing the action mix, tightening loop termination conditions, or repricing. Both sides watch the fan-out ratio, because it is the only term that neither owns alone — a planner change made for quality reasons is an engineering decision with a direct margin consequence.
That division also settles the sequencing question. When cost-per-business-action is too high, the first thing to check is not per-call price. Check fan-out first, because it is usually the term with the widest variance and the cheapest fix (a termination condition, a confidence threshold, a cached retrieval step). Then check action mix — whether a small share of pathological actions is carrying most of the cost. Per-call price is the third lever, not because it does not matter but because a 20% cheaper call inside a loop that doubled in length is a net loss.
When graduating is premature
Three cases where the second metric should wait. Fan-out is genuinely fixed at one, and the rollup is a rename rather than a measurement. Telemetry cannot yet attribute cost by feature, model version and retry path — in which case instrumenting the per-request layer properly comes first, since a per-action number built on an unattributable per-request number inherits every one of its errors. Or the product has no externally countable action, which usually means pricing has not been decided; inventing a unit ahead of pricing produces a KPI that will be redefined within a quarter and cannot be compared over time.
In all three cases cost-per-request remains the honest number, and saying so is better than shipping a margin metric no one can reproduce.
Frequently Asked Questions
1. ROI: what does graduating from cost-per-request to unit economics per business action mean in practice?
Once AI costs exceed 15% of gross margin or you’re fielding pricing questions from finance, request-level metrics no longer suffice. Graduate Cost Per Request rewards a careful definition. Graduate Cost Per Request turns on one distinction. Graduate Cost Per Request rewards a careful definition. Asked about Graduate Cost Per Request, most engineers point here. Graduate Cost Per Request turns on one distinction. Graduate Cost Per Request rewards a careful definition. Graduate Cost Per Request turns on one distinction. Graduate Cost Per Request rewards a careful definition. Asked about Graduate Cost Per Request, most engineers point here. Graduate Cost Per Request turns on one distinction. Graduate Cost Per Request rewards a careful definition. Graduate Cost Per Request turns on one distinction. Graduate Cost Per Request rewards a careful definition. Graduate Cost Per Request has one honest answer. Asked about Graduate Cost Per Request, most engineers point here. It means keeping cost-per-request as the engineering control variable and adding cost-per-business-action as the margin-facing KPI, with a measured call fan-out ratio linking the two. In practice the deliverable is three auditable numbers: cost-per-action before and after a change, calls per action, and gross margin per AI feature at the current action mix.
2. What signals tell us cost-per-request has stopped being the right optimisation target?
The clearest signal is a variable number of model calls per customer action — agentic loops, confidence-gated reranking, multi-turn resolution, or server-side retries. A second signal is organisational: when Finance rather than the platform team starts asking for the number, the question being asked is about margin per action, not per call.
3. How do we define a “business action” for an AI feature without inventing a vanity metric?
Use the unit your product already bills, contracts, or reports to customers — a resolved ticket, a generated document, a completed session. It must have a countable completion event in telemetry, not only a start event, and it must map to one feature so a single team owns the resulting cost figure.
4. How do we measure the call fan-out ratio between a customer action and the model calls it triggers?
Propagate one action identifier from the ingress through every downstream model call and attach it to the trace that already carries token and GPU-time counters. That yields mean calls per action, the p90 fan-out, and the distribution shape — measured on the deployed serving path rather than estimated from an architecture diagram.
5. Which KPI do engineering and finance each own once both metrics are in place?
Engineering owns cost-per-request and defends it against serving-path regressions. Product and finance own cost-per-business-action, moved through action mix, loop termination rules, or pricing. The fan-out ratio is jointly watched, because quality-driven changes to a loop land directly on margin.
6. How does the graduation change what we optimise first — per-call price, fan-out, or action mix?
Fan-out first, because it usually has the widest variance and the cheapest remedies, such as a termination condition or a cached retrieval hop. Action mix second, to find the small share of pathological actions carrying most of the cost. Per-call price third — a cheaper call inside a loop that has doubled in length is a net loss.
7. When is graduating premature, and cost-per-request still the only honest number we have?
When fan-out is genuinely fixed at one, when telemetry cannot yet attribute cost by feature, model version and retry path, or when the product has no externally countable action because pricing is undecided. In those cases a per-action figure inherits the errors underneath it or gets redefined within a quarter.
Volume changes everything about your math
Once you cross 10 million requests per month, batch processing and caching investments pay for themselves within a quarter. Revisit it when your workload shifts.
