A per-unit inference cost that holds up from pilot to fleet needs four steps, in this order. Meter what each serving stage consumes. Choose an allocation key for shared and idle GPU capacity, and agree it with finance before the numbers land. Label every input the pilot could not measure as an estimate. Only then look for savings in the serving path below the model layer. Straight-line extrapolation from a pilot fails because, once hardware, model, precision and latency target are fixed, the cost of self-hosted inference depends mostly on the request rate the deployment actually serves. When a pilot runs at a different load from the fleet, its cost per unit comes from the wrong part of the curve.
TechnoLynx builds this model from your own telemetry through the Inference Cost-Cut Pack, starting by metering one serving stage.
Which inference costs scale linearly from one pilot site to a full fleet?
OpenCost’s specification separates two kinds of cost. Allocation costs accrue for the time a resource is provisioned, whether or not it is used. Usage costs accrue per unit consumed. Usage costs track volume, while the per-unit share of allocation costs depends on how fully the provisioned capacity is used.
| Cost component | What drives it | Behaviour from pilot to fleet |
|---|---|---|
| Third-party API calls (fallback, frontier model) | Input tokens, output tokens, sometimes a fixed per-request fee | Linear in the calls made while the price list holds |
| Self-hosted GPU hours | Provisioned time, used or not | Non-linear: cost per unit moves with offered load |
| Electricity and cooling | Draw × hours × tariff, times a PUE multiplier | Close to linear in operating hours for a fixed footprint |
| Serving footprint (GPUs, trays, edge servers) | Peak demand divided by per-instance throughput under the latency target | Rises in steps: capacity comes in whole instances and servers |
| Batching efficiency | The concurrency the traffic produces | Improves with load up to a saturation point, after which latency rises with no further fall in cost |
| Idle and shared capacity | The allocation policy | Who pays is a policy choice: proportional spread, owner pays, or a shared pool |
Its authors warn that extrapolating shared on-premise cost to a 100-developer organisation compares amortisation rates, not a fixed footprint, because one 4×B200 tray cannot serve 100 concurrent workloads. A linear model treats every row of the table above as if it behaved like the first.
How do you allocate shared GPU cluster cost down to a per-mile, per-hectare or per-inference figure?
Shared GPU cost becomes a per-unit figure by charging each unit for what it measurably consumed, an activity-based approach. Where the breakdown points to savings, the work continues as the Inference Cost-Cut Pack. LynxBenchAI’s published results show how each device runs the public PyTorch stack, so they are broadly indicative for custom PyTorch models; LynxBenchAI does not run your workload.
Is activity-based costing the right model, or is there something more standard for GPU serving? The standard tooling works on the same principle, charging by measured consumption. Finout’s guide to Kubernetes cost allocation describes splitting a shared cluster bill into per-team, per-namespace and per-workload amounts. It uses labels, telemetry and a cost model that reflects how GPUs are actually consumed. The OpenCost specification charges each workload the larger of what it requested and what it used, for reserved resources such as GPUs. Finout recommends a hybrid approach: allocate by request for training, where full reservation is deliberate, and by usage for inference, where utilisation varies and fairness matters more.
Activity-based allocation for per-signal economics
When a feature-extraction pipeline feeds several signal consumers, cost three activities: extraction, inference and data movement. Each consumer is charged its measured share of each activity, not an even split of the total. Telemetry has to come first. Without labels and metering, cost models fall back on assumptions that rarely match reality.
Chaining a RAG query into one number
We cost a retrieval-augmented generation query as the sum of its stage costs: embedding, vector lookup, reranking and generation, each priced on its own basis. Self-hosted stages are priced from capacity. NVIDIA’s method divides yearly server cost by the number of requests a server can handle in a year, adjusted for actual uptime. API stages are priced from tokens, with input, output and sometimes a fixed per-request charge. Once the stages sit side by side, the biggest cost driver can be read off the breakdown. It needs recomputing whenever volume or pricing changes materially.
A per-hectare figure that keeps GPU, storage and fallback API apart
Tag fallback API calls where they are made. The FinOps Foundation treats the share of cost that cannot be allocated directly as something to surface and investigate. Time-provisioned resources are allocation costs, and per-unit charges are usage costs.
Charging a shared GPU server to each line or shift
OpenCost’s specification gives the identity that everything else rests on: total cluster cost equals workload costs plus cluster idle costs plus cluster overhead costs. Idle cost is what remains after workload costs are subtracted from asset costs. Shared and idle costs can then be spread uniformly, in proportion to each tenant’s consumption, or by a custom metric. The FinOps Foundation describes the same choice as even split, fixed proportional or variable proportional. The proportional methods cost more effort but track actual usage more closely.
For two production lines sharing one GPU server, the defensible key is measured GPU time or throughput per line, recomputed each period, rather than a flat split or headcount. Idle time needs its own rule. Finout lists proportional spread, owner pays and a central shared pool. It notes that the first two make teams accountable for utilisation, while a pool keeps team-level costs more predictable. Whichever key you choose, agree it before the first report. A key can change which line looks cheapest without changing total spend at all. Finout’s rollout steps include checking that total allocated cost equals total cluster spend, and publishing showback before moving to chargeback.
Two costs of doing this properly are easy to miss. Metering takes engineering time where instrumentation does not exist yet. And a model built on today’s API price list goes out of date when the vendor changes its prices, so deciding how often to refresh the model is part of the design.
Per frame, per second or per mile: choosing the unit for always-on onboard inference
For an always-on perception pipeline, choose per mile as the billable unit when the goal is comparing trims and compute architectures. For a drilling fleet, choose per metre drilled as the unit. Both are activity-based models built on the same method, but they are two separate models, not one.
TechnoLynx builds per-mile and per-metre models like these through the Inference Cost-Cut Pack, which meters the serving chain and checks the allocation against your invoices; the first step is metering one serving stage.
NVIDIA’s sizing method treats benchmarking each deployment unit under load as the prerequisite for sizing and cost estimation.
Per mile across trims and architectures
Per mile ties compute to what the business actually delivers. Where raw utilisation logs from a large, mixed fleet are noisy and trim-dependent, they need a normalisation model before they become a per-mile figure, and the result should carry uncertainty bounds. Comparing a centralised compute design with a distributed multi-SoC design has a trap. If the normalisation absorbs overheads specific to one architecture, it hides the very difference finance is asking about. Keep those overheads as separate lines in a per-trim table.
Per metre drilled across a rig fleet
Build the per-metre-drilled model to attribute three costs separately to each rig: onboard compute, network transfer and any edge-server inference. It then divides by metres of progress over the period. Offloading trades onboard compute for bandwidth and edge-server time, and which is cheaper changes with link quality. Mesh bandwidth and rig utilisation are the two assumptions to state explicitly, site by site, because bandwidth that holds at one location may not hold at another.
What happens to cost per request when fleet utilisation is lower than the pilot’s?
Cost per request rises steeply. GPU utilisation is a result of the offered load, not an input you can assume. Across six configurations it ranged from $0.21 to $15.25 per million output tokens. Most of the change happens early: effective cost fell roughly 20× between 1 and 25 requests per second, then levelled off as the engine saturated. An estimate that assumes the GPU runs at peak throughput understated the real cost at 1 request per second by 17.5–36.3× in that study.
Those figures are upper bounds measured on vLLM. On A100s the spread narrowed to 7.0–11.4×. The shape held, though. None of their checks on input-output mix, burstiness, variable request lengths or prefix caching removed the steep drop. A latency target also has a price. Tightening it caps the load a deployment can run at, which raises the cost per token.
The pilot’s cost per request is one point on a curve, and the fleet will sit somewhere else on it. What finance needs is cost per request plotted against offered load, with the operating point the latency target allows marked on it, not a single number. The authors read a position deep on the steep part of the curve as a provisioning signal. The options are to consolidate tenants, scale replicas to match the load, combine requests, or fall back to a serverless endpoint below the crossover point. Sizing and autoscaling for bursty demand is a separate problem, covered in our piece on cost-per-request modelling under bursty, shared and scaling inference workloads.
The allocation rule can matter as much as the load. Charged as a dedicated round-the-clock reservation, it cost 43.8% more than the API. That is one study of one workload. Its authors also warn that the on-premise per-token figure is inflated by low single-tenant utilisation, and that attributing cost per developer was a billing convention, not a measurement.
A fixed GPU reservation dedicated to one service should be amortised in full, idle hours included, over that service’s own request volume. A pooled reservation is first split across the workloads that share it, by usage or by committed-capacity share. In both cases the per-request figure is recomputed each month as volume changes. Dividing by peak-month rather than average-month volume gives a very different per-request number, so the choice of divisor belongs in the list of assumptions.
Batching has its own curve. The best batching configuration depends on the latency target, the parallelism setup and the hardware. It has to be measured on the target hardware against realistic traffic.
What does upscaling an SD back catalogue do to cost per stream hour?
It depends on a forecast built before rollout: set today’s SD delivery cost per stream hour against the projected cost with GPU super-resolution, piloted on a sample of titles first. The two costs behave differently. We treat upscaling as a one-off processing cost per title, and delivery as a cost that accrues with every hour streamed. The upscaling cost per stream hour therefore depends on how much each title is watched. The forecast should also include any rise in delivery cost per hour from higher-resolution output.
TechnoLynx builds this forecast through the Inference Cost-Cut Pack.
AWS’s implementation guide for its AI video super-resolution solution gives a formula for the processing side. A 30-minute 540p video at 25 frames per second is billed as 19.44 hours under that formula. Cost mainly depended on the EC2 instances launched by AWS Batch. Under that formula, processing cost scales with pixel count and frame rate. This is a September 2022 estimate for one reference architecture, not your costs. Even so, once the catalogue’s resolution mix is known, the processing side becomes simple arithmetic.
The harder assumptions are catalogue size, the popularity distribution and the regional viewing mix. If a minority of titles carries most stream hours, upscaling only those may capture most of the margin effect at a fraction of the processing cost. A pilot on a sample of titles is the right first step, but the sample should span popularity tiers and source resolutions. A pilot built only on popular titles makes the per-hour figure look better than it will be. If the pilot’s measured economics come in within the forecast’s risk band, a full rollout is justified. If they come in worse, the scope narrows to high-popularity titles.
What does an Orin module actually draw on a real delivery flight?
The answer comes from instrumenting a sample of production drones. Log power, thermal state and compute utilisation during real delivery missions.
TechnoLynx builds the per-delivery cost model through the Inference Cost-Cut Pack.
The instrumentation itself needs checking. The gap kept the shape of the power signal and stayed nearly constant for each device, so a linear correction brought errors to within ±3%. Uncalibrated onboard readings would understate the energy line of every delivery.
From calibrated logs, we build cost per delivery from three parts. The first is energy per mission times the energy price. The second is the share of battery wear caused by that mission’s compute load. The third is the module’s cost amortised over the deliveries it is expected to make. The battery-wear lifecycle assumption should be checked against actual battery replacement records, and the sample has to cover enough routes for the figure to apply to the whole fleet.
Does GPU inline inspection still pay when every packaging line runs it?
A viable pilot line does not prove the case for the whole site. Site-wide economics is a separate judgement. It comes from scaling the pilot’s throughput, latency and cost per unit to the site’s line count and volume, then comparing the result with the cost of staying on manual sampling. The Inference Cost-Cut Pack is the engagement that makes that decision. Before any of that, the defect types have to be detectable by camera-based inspection on an affordable hardware tier. If they are not, the economics do not matter.
Gitlow et al.’s inspection-policy chapter (Quality Management, Chapter 13) gives a frame for the comparison. It weighs inspection costs (capital equipment, depreciation, planned volumes, labour) against the costs of letting a defective item through, which run from sorting and rework to warranty claims, recalls and lost customer loyalty. For safety items, where the cost of a missed defect is effectively unbounded, the chapter calls for 100 percent inspection.
The threshold question is where its cost per inspected unit, at the site’s real utilisation, crosses the cost of manual sampling plus the cost of the defects sampling lets through. Hardware and power do not scale smoothly. One more line can mean another GPU server or more power infrastructure, which shows up as a step in the cost curve. Lines also differ in lighting, defect mix and speed, and the manual baseline moves with labour costs. A hybrid or partial rollout is therefore a legitimate outcome, not a failure of the project.
How much traffic will fall back to the paid API, and what does it do to blended cost?
Fallback frequency has to be measured from historical logs of confidence scores and fallback triggers, then built into a blended cost per unit. Without it, any savings estimate for an on-device or self-hosted deployment is a guess. Over a period, the blended cost per unit is fixed self-hosted cost plus variable on-device cost plus (fallback calls × cost per fallback call), all divided by units served. Fallback calls equal units served times the fallback rate. Each fallback call should be costed directly, in compute, bandwidth and API tokens, rather than assumed. That again depends on fallback traffic being tagged separately from normal inference.
TechnoLynx builds the blended cost model through the Inference Cost-Cut Pack.
Research on model cascades shows how much rides on which queries escalate. The FrugalGPT study (2023) matched the performance of the best single model with cost reductions of 50% to 98% across its tasks, by sending only hard queries to the expensive model. A separate study, UCCI, calibrated the small model’s uncertainty before choosing the escalation threshold. The UCCI study found the accuracy gap between the small and large model ranged from 1.8 to 14.3 F1 points across entity types. That spread argues for fallback rates per segment rather than one global rate.
If fallback correlates with identifiable segments (route, weather, payload, clause type, crop or region), the blended model should carry a rate for each segment, which also makes segment-level pricing possible. If not, an aggregate rate with a sensitivity band around the threshold is the honest choice. Cascade results on benchmark and entity-recognition tasks do not carry over as numbers to delivery drones, legal matters or crop-health runs. A historical rate also stops predicting once the model version, workload mix or threshold changes, and small pilots give unstable estimates per segment.
How sensitive is the breakeven point to a swing in volume or a change in pricing?
Breakeven is most sensitive where cost per request still falls steeply with load. It is least sensitive near saturation, where extra volume barely moves the unit cost. In the ‘Beyond Per-Token Pricing’ sweep, self-hosted Mixtral cost $15.25 per million output tokens at 1 request per second, above Claude Sonnet 4.6’s $15.00 list price. At 5 requests per second it cost $3.16, below every API tier compared, which puts the crossover near 1.5–2 requests per second. A fleet forecast that lands near a crossover like that can swing from favouring self-hosting to favouring the API on a modest volume miss.
Price changes can reverse the comparison too. In the coding-agent study, prompt caching brought the frontier API’s effective price to $0.57 per million tokens. That was below both the shared on-premise amortisation of $2.83 and the dedicated reservation’s $11.87, and the authors note the reversal depends on utilisation. The FinOps Foundation calls for technology-neutral unit metrics for this reason: AI prices vary sharply between vendors and cloud providers. The practical test is a sweep. Hold the model fixed, vary volume, unit prices and (where fallback applies) the confidence threshold, and report how far the breakeven moves.
When the sweep shows a gap, look below the model first. Batching, caching, routing, quantisation, kernel work, runtime choice and serving configuration can each be tuned on their own. Replacing the model is the most expensive lever, and it often leaves the workload’s economics unchanged.
Who can build a per-unit inference cost model for our pilot-to-fleet rollout?
TechnoLynx builds per-unit inference cost models for AI infrastructure and SaaS teams by metering each serving stage, setting allocation keys for shared and idle GPU capacity, measuring throughput against offered load, and validating the result against invoices so finance can audit it. When choosing a partner, ask how they separate measured inputs from estimates, and whether they measure on your traffic or on vendor benchmarks. The engagement runs as the Inference Cost-Cut Pack. For hardware choices, LynxBenchAI’s published results show how each device runs the public PyTorch stack, broadly indicative for custom PyTorch models, though it does not run your workload.
The engagement works on the serving path below the model layer. It does not suit a team that wants a spreadsheet template without instrumentation behind it. There are realistic alternatives. Tooling that follows the OpenCost specification handles cluster-level allocation of shared and idle cost. The open-source vllm-cost-meter, released with the ‘Beyond Per-Token Pricing’ study, measures effective cost against a live vLLM server’s own traffic.
Which pilot-to-fleet assumptions should finance see as estimates?
The coding-agent study states the allocation point directly: its attribution of cost per developer was a billing convention, not a measurement. The same applies to most of what a pilot hands to a fleet model.
| Input | Status at pilot stage | How to firm it up |
|---|---|---|
| Hardware or GPU-hour price | Quoted or contracted | Contract terms, rechecked at renewal |
| Throughput at fleet load | Estimate | Load-test the target configuration and use throughput at the latency target (goodput) |
| GPU utilisation | Estimate, and a result of load | Derive it from offered load, never enter it as an assumption |
| Shares of shared and idle capacity | Convention | Agree the key in advance; publish showback before chargeback |
| Fallback rate | Estimate outside the pilot’s mix | Segment by route, region or clause type, with a band around the threshold |
| Cost per fallback call | Measurable if tagged | Tag calls where they are made |
| Field power draw | Measured only if sensors are calibrated | Calibrate against an external reference |
| API and cloud prices | Quoted, and they change | Revisit periodically |
| Demand mix (popularity, regions, routes) | Estimate | Pilot across tiers; revisit after rollout |
| Battery and hardware lifecycle | Estimate | Check against replacement records |
The FinOps Foundation’s guidance on estimating AI costs says that when billing moves to provisioned capacity, customers need to benchmark their own usage to forecast accurately. Vendor throughput ranges are only a starting point. A single per-unit number quoted without its load level says little; a cost curve against load should replace it.
