A defensible cost-per-request figure is the hourly cost of the serving hardware divided by the request rate that hardware sustains while still meeting its latency SLA. It is expressed per million requests, then checked against each model’s measured latency and throughput when models share a server. None of the terms should be assumed to scale in a straight line when lines, tenants or adapters are added or when traffic peaks. That is why a number extrapolated from today’s small deployment cannot be carried to scale without re-measuring. When the figure comes in too high, the first thing to look at is the serving path (batching, caching, routing, quantisation, runtime configuration), not a replacement model.
This matters most to teams running AI infrastructure and production SaaS, where the same figure has to support admission control, pricing and sourcing decisions. The method below builds the number up in layers: the base formula, how sharing a GPU changes it, how shared cost is attributed to services and tenants, how the figure behaves at peak, and how multi-adapter serving shifts the economics.
The cost-per-request formula, term by term
The Perseus measurement study of CNN serving on NVIDIA P4 and T4 GPUs uses cost per one million inference requests, a standardized metric that accounts for each device’s relative performance and so promotes price comparisons across hardware. The throughput that goes into it is the peak or steady-state request rate. Latency is checked at the 95th or 99th percentile, and latency determines whether a configuration is viable at all.
cost per 1M requests = (hourly cost of the serving node)
÷ (requests per hour sustained at the latency SLA)
× 1,000,000
The hourly cost has two versions, and the choice between them is the first decision a model has to make explicit. A 2026 scenario analysis of inference economics defines full cost as four-year straight-line amortisation of the accelerator price times a 1.5× system multiplier, plus power at $0.08/kWh with a PUE of 1.3, plus $0.25 per GPU-hour of allocated operations. In the same analysis, marginal cost omits the amortisation and is the economically relevant floor for a fleet whose capital is already sunk.
The denominator is where utilisation comes in. A node that is paid for by the hour but runs at a fraction of its sustainable rate divides the same cost over fewer requests. Defensibility therefore depends on measuring the sustained rate under the real request mix, not reading it off a datasheet. Our guide to finding and fixing GPU bottlenecks through profiling covers the instrumentation side, and the principle is to profile first and optimise second, in context.
Does cost per inspected sheet fall as more lines share one GPU server?
For a surface-inspection system, adding lines to the same GPU server spreads the fixed server cost over more inspected sheets. That holds only until the combined throughput saturates the GPU or breaks the latency budget. The eight-line figure therefore has to come from a TCO model with lines-per-GPU as a parameter, not from multiplying the two-line cost by four.
TechnoLynx would approach this by first establishing each line’s inspection throughput requirement and the current two-line GPU utilisation as the baseline. TechnoLynx would then set the sharing configuration from the measured headroom, build the cost-per-sheet model parameterised by line count, and check its projections against two-line actuals before the full rollout.
The question of one GPU per line or several lines per GPU should be settled by arithmetic on measured numbers. If the combined sheet rate of several lines fits inside what the GPU sustains for the defect-detection model at the required latency, those lines share. If one line’s rate alone saturates a GPU, that line gets its own. The shape of the cost-per-sheet curve therefore has to be measured on the line itself, with general multi-tenant results as the nearest analogue.
Perseus is a general analogue from CNN serving on P4 and T4 GPUs, not a measurement of inspection lines. When two models shared GPU servers at roughly equal request rates, serving was on average 9.5% cheaper across the tested configurations, with savings of up to 12% on two P4 GPUs. That saving had a latency cost: steady-state latency rose by 55% for ResNet50 and 26% for Inception-V3. Perseus also found that serving several models on the P4 risked over-filling GPU memory because each model is large relative to the card. The sharing gain is modest and bounded, and it is paid for partly in latency headroom and partly in shared GPU memory.
Three caveats belong in the model rather than in a footnote. Throughput headroom has to be re-measured whenever the defect-detection model or the sheet rate changes. A model validated only at two lines carries extrapolation risk. Validating contention effects at an intermediate scale such as four lines before committing to eight reduces that risk. Energy and maintenance also do not necessarily scale in proportion to line count, so each should be modelled explicitly.
Who on a shared GPU is paying less than they cost?
A tenant is priced below its true cost to serve when the revenue it brings is lower than the cost allocated to its share of the shared GPU, CPU and memory. Identifying such tenants requires metering each tenant’s consumption and correlating it with infrastructure cost, which AWS describes as the basis for calculating cost per tenant and setting a pricing model.
The vendor-neutral OpenCost specification assigns a workload the greater of its requested and its used GPU (and CPU) resources, written max(request, usage). A tenant that reserves a whole GPU and uses a fraction of it still pays for the whole reservation. AWS Split Cost Allocation Data for EKS applies the same greater-of rule to pod-level accelerator, CPU and memory reservations.
Idle capacity is the second decision. OpenCost defines cluster idle cost as cluster asset costs minus workload costs. It allows idle, shared and overhead costs to be distributed uniformly across tenants, in proportion to each tenant’s consumption, or by a custom metric such as bytes of network egress. AWS distributes unused instance cost to each pod in proportion to its unused ratio. AWS also weights a unit of GPU at nine times a unit of CPU and memory when it normalises cost per resource.
Each of these is a policy choice that moves cost between tenants. A defensible model records which rule it uses and why. The tenants that look cheap under uniform idle distribution and expensive under proportional distribution are the ones worth examining before any price change.
Can we break out exactly how much of our storage bill belongs to one service?
A service’s storage spend can be tracked as its own line item through cost allocation tags. The method is to isolate the storage-specific spend attributable to that service (capacity, retention copies, backups, requests, retrieval and egress) and divide it by the number of requests or documents the service processed in the same period. Reported separately from compute, that line can be audited on its own.
Microsoft’s Azure Blob Storage documentation states that each meter (data storage, operations, data transfer, data retrieval and others) appears as a separate line item on the invoice. Finout puts real-world cloud storage spend at often two to five times the base per-GB rate. A per-request storage line built only from the per-GB price will therefore understate cost.
Attributing that spend to one service depends on tagging. In Amazon S3, buckets are tagged with a key such as a project identifier, and the tags are then activated for cost allocation in the Billing console. After that, the Cost and Usage Report breaks costs down by tag. The resulting line is only as accurate as the tagging. A bucket shared by several services without a usage-based split cannot be attributed cleanly. AWS’s SaaS guidance makes the matching point that separate unit costs per resource, such as database and stream, are needed for that granularity.
Retention for audit needs its own treatment. Archive tiers carry minimum storage durations: Finout lists 30 days for S3 Standard-IA, 90 for Glacier Flexible and 180 for Deep Archive, with early deletion charged as if the data had stayed. Those durations make retention a cost committed over months, so amortising it over a single month’s request volume misstates the per-request figure. One full cost accounting model for storage includes hidden and environmental costs alongside regular costs.
What does one vision inference call cost in GPU-seconds, and what happens at 3x peak?
Nobody can state the cost of an authentication-and-grading call with confidence until GPU-seconds per request are instrumented by model stage and image complexity. Without that breakdown, every savings estimate is a guess. Once the breakdown exists, the per-call cost at three times normal volume is a modelling question: price structure combined with measured utilisation under each scaling strategy.
TechnoLynx’s Inference Cost-Cut Pack addresses this need for a vision pipeline. The engagement starts by defining the cost-per-inference metric and the breakdown it needs. It then instruments per-request GPU-seconds at that granularity, tunes batching on the target GPU, and validates the cost model against measured normal and 3x volume before the chosen scaling strategy goes live.
The shape of the curve at peak is the part that most needs measuring. In Perseus’s CNN serving tests on P4 and T4 GPUs, cost rose polynomially as throughput decreased when peak throughput was not met. In under-utilised conditions, keeping hardware cost under control depends on estimating a model’s throughput accurately. The danger runs both ways. Capacity sized for back-to-school peak sits under-utilised for the rest of the year, while capacity sized for the average fails the latency SLA at peak. A per-call figure taken from either regime alone misrepresents the year.
Utilisation and offered load can move the figure by more than an order of magnitude. A 2026 serving-cost methodology study, cited in the scenario analysis above, found effective cost on identical H100 hardware ranging from $0.21 to $15.25 per million output tokens depending on utilisation and offered load. The same analysis notes that realised cost per token also depends on batchability, latency objectives, traffic burstiness and other workload factors, so a hardware-level unit cost is not a deployment figure.
Batching changes that relationship. Batching increases GPU utilisation, and OpenAI’s batch endpoint offers bulk inference at roughly 50% of real-time token costs. That is an API price for work that is not latency-sensitive, not a measurement of a self-hosted vision pipeline. Hardware mix matters as well. The Mélange study of LLM serving found that request size, request rate and SLO determine which GPU type is most cost-efficient, and that mixing GPU types cut deployment cost by up to 77% in conversational settings compared with a single type.
The 3x curve for a vision pipeline under batching and autoscaling, compared with fixed added capacity, therefore has to be measured on the pipeline itself. The decision rule is the following. Batching or autoscaling wins if it lowers cost per call at peak without breaching the latency SLA. Fixed added capacity wins only where peak latency cannot tolerate the variability that batching introduces. Three caveats apply. The projection assumes today’s image-complexity mix holds. Spot or autoscaled pricing adds variance that a single per-unit number understates. The estimate is never more precise than the GPU-second instrumentation beneath it. Where the peak plan involves buying or renting a different accelerator, LynxBenchAI publishes how each device runs the public PyTorch stack. Those results are broadly indicative for custom PyTorch models but are not a measurement of the reader’s own pipeline.
Capacity planning and autoscaling policy as such are covered in our companion piece on cost-per-request modelling under bursty and scaling traffic, which also covers whether a pipeline will hold as tenant count grows tenfold without re-architecture.
At what adapter count does a dedicated fine-tuned replica become cheaper than S-LoRA?
A dedicated replica is cheaper below roughly 15 to 20 adapters on one base model. That breakeven comes from one worked comparison, a 2026 ClusterBid blog. It describes multi-LoRA as definitively cheaper above 50 and as the only economically viable approach above 200. The crossover for a given workload has to be measured on its own traffic mix.
TechnoLynx’s Inference Cost-Cut Pack covers this comparison. It frames the problem with the reader’s actual adapter count, traffic mix and GPU inventory, sets up S-LoRA and dedicated-replica configurations side by side, measures the throughput and latency penalties of adapter swapping under real traffic, and delivers the validated crossover as deployment guidance.
The mechanism explains why the crossover exists. According to the S-LoRA paper, the system keeps all adapters in main memory and fetches those needed by running queries into GPU memory. A unified memory pool (“Unified Paging”) manages adapter weights and KV cache together. Against HuggingFace PEFT and vLLM with naive LoRA support, S-LoRA reports up to 4× higher throughput and several orders of magnitude more served adapters. Punica takes a related approach, holding a single copy of the base model and batching operations across different LoRA models. It reports 12× higher throughput than other LLM serving systems while adding 2 ms of latency per token.
The ClusterBid comparison gives one cost example. Serving 150 adapters on a 70B base model with dedicated replicas needed at least 19 H100 GPUs, roughly $15,732 a month at that marketplace’s mid-2026 rate. Multi-LoRA served the same adapters on four H100s for about $3,312 a month, which the blog calls roughly 4.8× cheaper.
GPU memory is what limits co-resident adapters. In the same source, a rank-64 adapter for Llama 3.1 70B is about 640 MB at FP16, and an 80 GB H100 holds about 80 resident adapters with no active requests. That falls to about 12 to 16 at a typical concurrency of four to eight requests, because KV cache takes the space. Forty adapters exceed the 12 to 16 resident at that concurrency, so some are swapped in from host memory on demand. Growth in pressure can be nonlinear, so throughput measured at one adapter count does not extrapolate safely to a higher one.
The swap penalty also has to be measured on real traffic. ClusterBid puts a CPU-to-GPU swap of a rank-64 adapter at roughly 15 to 25 ms over PCIe 5.0, often hidden behind compute by prefetching. The dLoRA paper (OSDI 2024) adds a less obvious result: batching with adapters kept unmerged is not always beneficial when request types are skewed. Its system dynamically merges and unmerges adapters and migrates requests between replicas, reporting up to 57.9× higher throughput than vLLM. A skewed traffic mix, where a few tenants dominate, is exactly the case where an average swap cost misleads.
This also informs pricing. Flat per-token billing does not distinguish a tenant whose adapter must be swapped in from one whose adapter stays resident. Adapter-aware or usage-tiered pricing needs the per-tenant metering described earlier. Its floor is the marginal cost of serving, which the scenario analysis calls the relevant floor for fleets with sunk capital. The caveats are that the crossover shifts with adapter size and request pattern, and that any model built on current GPU prices ages as prices change.
Shared versus dedicated: the checks that settle it
| Check | Points to sharing | Points to dedicating |
|---|---|---|
| Combined sustained throughput vs GPU capacity at SLA | Fits with headroom | One workload alone saturates the GPU |
| Latency headroom | Can absorb a measured rise under co-location (Perseus saw 26-55%) | p95 already near the SLA |
| Memory budget | Models, adapters and KV cache fit at real concurrency | Over-filled memory under co-location |
| Adapter count | Above the measured crossover | Below it |
| Traffic mix | Balanced rates across tenants (Perseus’s best case) | Heavily skewed rates |
| Attribution | Tagging and max(request, usage) metering in place | Cannot be attributed cleanly, so pricing cannot be defended |
Who can build a cost-per-request model for our inference platform?
TechnoLynx builds cost-per-request models for inference platforms from the measured serving path. That covers instrumenting GPU-seconds per request, deciding where workloads, lines or adapters should share a GPU, and validating the figure at normal and peak load. On the build-or-bring-in question, a team with profiling skills and time can do the instrumentation. A specialist is worth bringing in when the figure has to stand up to finance or customers and the serving path also needs tuning. When choosing a partner, look for one that measures before recommending a model change. The catalogue offer is the Inference Cost-Cut Pack. For the hardware side, LynxBenchAI is the measuring tool: its published results show how devices run the public PyTorch stack.
The engagement’s unit of work is the serving path, not the invoice. Teams that only need tag-based showback can get it from tools that implement the OpenCost specification or from AWS split cost allocation data.
A figure that names its own scope
The cost-per-request number most worth defending states its conditions: which hardware cost (full or marginal), which sustained rate, which latency percentile, which allocation rule for idle capacity, and which traffic mix it was measured under. A model that carries those conditions as parameters can be re-run when a line, a tenant or a peak arrives. A single number without them cannot.
FAQ
Is the cost of compute going down?
At constant capability, yes. Stanford HAI found GPT-3.5-class inference roughly 280× cheaper over two years, and a16z estimates about 10× per year. The 2026 analysis that cites these figures notes that they combine hardware efficiency, model right-sizing, routing and margin compression.
What do you mean by shared GPU memory?
In inference serving, one GPU’s memory pool is shared between base model weights, adapters and KV cache.
