To get finance to approve GPU capacity, cost it and size it from your own measured utilisation, not from the list price per GPU-hour. Start with 30 days of utilisation telemetry at minimum (60 in zop.dev’s phased plan) and read its p10 floor, not its mean. Then build four things on that measurement: a fully-loaded cost of ownership, a capacity guarantee that actually holds at peak, autoscaling headroom, and an allocation rule agreed before anything is bought. LynxBenchAI covers the hardware side of that model with published, release-named results. Your own workload still has to be tested on top of them.
Commit to the utilisation floor, not the average
zop.dev’s guide to commitment discounts says to collect at least 30 days of CPU and memory utilisation data at five-minute granularity before committing, and 60 in its phased plan because one month misses bi-weekly patterns. Turn it into a percentile distribution and treat the p10 value, the floor the workload almost never drops below, as the safe ceiling for a commitment. Everything above that floor is variable demand, and it belongs on on-demand or spot pricing. The same guide suggests starting at 70% of the p10 floor, so the 30% buffer can absorb measurement error and seasonal dips.
For a break-even model, the mean is the wrong input. In one production API cluster the guide’s authors measured, mean CPU was 68% but the p10 was 31%. A commitment sized to the mean would have over-committed by more than half during off-peak windows.
In the guide’s AWS m5.xlarge example, at USD 0.119 committed against USD 0.192 on-demand per hour, the reservation only wins above about 62% average utilisation over the term. AMCompute’s three-year model finds that ownership beats on-demand above roughly 51% utilisation, and a long-term reservation beats on-demand at roughly 65%. Lyceum points out that no single threshold exists, because break-even depends on the rental rate you compare against. Rental rates move a lot: Introl reports that H100 rental fell from USD 8 per hour to USD 2.85-3.50 per hour from its late-2024 peak. Whatever the rate, a reserved contract is paid for the full term whether you use the capacity or not.
At facility scale, a fixed grid connection caps the budget for compute. The number of usable GPU racks is the power left after cooling and auxiliary overhead, divided by per-rack draw, so a design with a worse PUE fits fewer racks under the same ceiling. ISO/IEC 30134-2 defines a designed PUE, projected from design targets, and an interim PUE, measured over a period other than a continuous 12 months. An October 2026 EnergyPlus modelling study on arXiv found that warm-water liquid cooling lowered annual PUE from 1.43 to 1.25 against an air-cooled baseline. So a cost per useful token for a pod design needs the PUE for your climate and load profile, not the nameplate figure.
What does a fully-loaded cost per served token or training run include?
A fully-loaded monthly cost sums hardware amortisation, electricity, cooling, networking, staffing and software. You then divide the total by the output you actually produce at the utilisation you actually reach. LynxBenchAI’s published results show how each device runs the public PyTorch stack. So read each candidate’s inference or training score on the LynxBenchAI AI hardware benchmark and leaderboard as the throughput side of that division.
LynxBenchAI publishes release-named benchmark results for each candidate device, so the first step is to read its inference or training scores for the devices on your shortlist.
One cluster TCO model puts GPU hardware at typically 35-50% of three-year TCO. Power takes 10-20%, networking 10-15%, data-centre space 10-15%, staff 5-10% and software 2-5%. prompt20 notes that renting by the hour includes power and cooling in the rate, while buying does not; air cooling adds 30-50% on top of power, and self-hosting maths without these lines is wrong by 10-15%.
Storage and egress belong in the model too:
- SemiAnalysis, as cited by Lyceum, identifies networked NVMe storage as the second most expensive part of an H100 cluster, because the GPUs need enterprise-grade throughput to stay fed.
- GPUPerHour’s 2026 comparison finds egress fees ranging from zero to over USD 0.50 per GB, and 6 of the 12 GPU cloud providers it tracks charge nothing for outbound transfer.
- Model checkpoints, dataset transfers and inference responses can move hundreds of gigabytes per training run.
- SitePoint puts the hidden staffing line at 20-30% of a senior engineer’s time for a production system, roughly USD 3,000-6,000 per month.
Utilisation multiplies everything else. prompt20 estimates that 50% average utilisation doubles the per-token cost compared with full load, 30% multiplies it by 3.3 and 20% by 5. Committed-use discounts change the rates being compared. SitePoint reports one-year commitments typically cutting rates by 30-40% and three-year commitments by 50-60%. prompt20 puts one-year H100 reservations on AWS p5 at 30-50% off on-demand. Discount depth differs by provider: prompt20 puts Lambda’s one-year H100 reserves at 20-40% off.
Amortisation depends on the lifespan you assume. AMCompute notes that NVIDIA now ships a new GPU generation about every year. Lyceum describes GPUs as having a two-to-three-year competitive timeline, against five to seven years for a standard enterprise server. Our judgement is to amortise each SKU over the shorter of its competitive life and the planning horizon, and to treat resale value as a scenario rather than a fixed input, because the replacement cycle is the assumption a five-year model is most sensitive to.
Fully-loaded cost checklist
| Cost line | What to put in the model | Where it hides |
|---|---|---|
| Hardware or rental | Purchase price ÷ lifespan in months, or committed hourly rate × all hours in the term | Reserved hours are billed whether used or not |
| Power and cooling | Measured draw × PUE × local tariff | Bundled into rental rates, missing from purchase quotes |
| Storage | Parallel NVMe file system sized to dataset growth | Second most expensive H100 cluster component |
| Egress | Expected GB moved × provider rate | Zero on some GPU clouds, over USD 0.50/GB on others |
| Networking | Interconnect fabric, switches, transceivers | A separate line in cluster bills of materials |
| Staff | Fraction of engineer time for operations | Underestimating it blows TCO projections |
| Utilisation | Divide by useful output at measured utilisation, not peak | Turns a list price into a multiple of itself |
Published benchmark figures are ceilings for the plan
prompt20 recommends dividing vendor-published throughput by roughly two when planning capacity, and still checking that the result meets the SLO with headroom for hot periods. LynxBenchAI’s published results are ceilings too: a release score shows what a device does on a fixed model catalogue, not what your service will sustain.
The ceilings also show why “an H100-hour” is not one unit. In the 26Q3 results the H100 80GB HBM3 scores 1678.1 in inference, the H100 NVL 1123.9 and the H100 PCIe 1039.6. These scores can be compared within 26Q3 only. They show that two quotes for “H100 capacity” can buy different throughput per hour, and a cost per request has to be normalised on that throughput. To measure a device you have access to, LynxBenchAI benchmarks your own hardware with stock PyTorch. Neither the Personal Edition nor the published results run your own model or serving stack.
How much capacity does a throttled GPU or edge module deliver in its real enclosure?
The only defensible number is sustained throughput measured in that enclosure, over the real duty cycle, under the real load. Each 26Q3 test is one timed window after a discarded warm-up, not a median of repeats. A release score therefore describes the device under test conditions. It does not describe hours of sustained load in a sealed or fanless housing. Amazon’s EKS guidance notes that a GPU utilisation reading near 100% does not by itself show full use, and points to the ratio of power draw to power limit as the better signal.
Will our GPU, storage and egress budget hold a concurrent-stream SLA at peak?
The answer comes from measurement, not from the budget total. Validated per-stream throughput can, provided it is measured for your codec path and has enough headroom for peak concurrency plus failover. TechnoLynx would turn the peak concurrent-stream target, bitrate floor and uptime commitment into the transcode throughput, storage and egress capacity they require, then test candidate hardware on your own content and traffic, failover scenarios included.
Moving money from transcoding GPUs to CDN egress raises SLA risk only if it pushes transcoding headroom below what peak concurrency needs. The cheapest stream is the cheapest validated configuration among those that meet the quality-at-bitrate floor on your content mix. In the 26Q3 results the NVIDIA L4 scores 209.1 in inference, the T4 is not covered, and the release runs a fixed PyTorch model catalogue rather than video encoders. At most, it is broadly indicative for PyTorch workloads on that hardware. It is not evidence that you will meet the SLA.
What reservation size minimises blended cost per request?
The best reservation size is the one with the lowest blended cost per request across your observed and forecast volumes. Blended cost per request is reserved GPU cost plus storage plus fallback-API spend for overflow, divided by total requests. LynxBenchAI measures per-device throughput for the reserved side of that curve. Once the size is set, TechnoLynx lowers cost per request on your own deployment through the Inference Cost-Cut Pack.
FinOps School defines blended cost as one per-unit rate that combines on-demand, reserved and spot rates, committed discounts and shared overhead. Reserved commitments are amortised over their term within that rate. The trade-off itself is simple: a larger reservation lowers the marginal cost per request but leaves more fixed cost exposed if volume falls short, while a smaller one leans harder on the costlier fallback at peaks.
Worked procedure for the cost curve
- Take the request-volume distribution from your telemetry. Build at least three forecast scenarios (optimistic, realistic and low utilisation), as SitePoint recommends for stress-testing before procurement.
- For each candidate reservation size, split each scenario’s requests into those the reserved capacity serves and those that overflow to the fallback API. Then compute the blended cost per request.
- Bracket the candidates with two published starting points that disagree. prompt20 suggests covering P50 traffic with reservations and bursting above it on demand. zop.dev suggests committing 70% of the p10 floor.
- Plot cost per request against reservation size for every scenario. Mark the minimum and the cost at the sizes either side of it.
- After purchase, track reservation utilisation. FinOps School sets a target above 90% of committed hours used. zop.dev advises pulling fresh utilisation data 60 days before each expiration.
The two starting points in step 3 rest on different statistics, P50 and p10. zop.dev puts everything above the p10 floor on on-demand or spot pricing; prompt20 says P50 coverage captures most of the discount.
Show where the minimum sits, how flat the curve is around it, and what the downside scenario costs at each size. When you present the curve to finance, state two caveats. First, the curve above assumes fallback and reservation pricing stay fixed over the term. Second, prompt20 itself assumes frontier API prices drop 30-50% over a 12-month horizon. For hosted-API commitments, the same source advises against three-year terms in a market with 5-10× yearly price compression.
What does a reserved GPU contract actually guarantee at peak demand?
A reserved instance or savings commitment on its own guarantees a price, not a machine. On Azure and AWS, it is the capacity reservation that carries the capacity guarantee. LynxBenchAI’s published results compare per-device performance, an input for turning peak demand into an instance count.
LynxBenchAI benchmarks candidate devices on the public PyTorch stack, so the first step is to compare its per-device scores for the instance types you plan to reserve.
Microsoft’s Azure documentation is explicit on this. An on-demand capacity reservation comes with an SLA-backed capacity guarantee in its region or zone, while Reserved Instances provide a billing discount only. VMs allocated beyond the reserved quantity are not covered by that SLA. On AWS, Overmind describes EC2 Capacity Reservations as guaranteeing that instances of a given type can launch in a given Availability Zone, and Reserved Instances as being about billing discounts. Capacity reservations are billed whether you use them or not, and Azure’s and EKS’s documentation say the same about their equivalents.
AWS Capacity Blocks for ML reserve GPU capacity for a fixed window. A block can be reserved up to eight weeks ahead, holds up to 64 instances (256 across all blocks), and cannot be cancelled. Blocks are reserved for a defined period, are pre-paid and do not support autoscaling.
Capacity shortfall is a documented risk for GPU instances. Overmind notes that capacity constraints most often hit newer instance types with specialised hardware, GPU instances included. DCD, reporting Business Insider’s account of internal documents, says Amazon’s own retail unit was short of more than 1,000 P5 instances, each with eight H100 GPUs, throughout the second half of 2024. That is one organisation’s internal shortage, not a regional incident record. Shortfall history for your own region has to come from your provider and your own failed-launch logs.
Dedicated or bare-metal capacity pays when the cost of a nightly or quarter-end run slipping is higher than the cost of the idle hours you reserve. Overmind recommends reserving for production workloads with strict availability requirements. It treats reservations that run below 80% utilisation as waste, and reservations that frequently hit 100% as too small for spikes or failover.
Sizing autoscaling and priority so queues don’t form at peak intake
Scale on a signal that sees the queue forming, and keep reserved capacity under the floor of the fleet. Google’s GKE guidance recommends queue-size autoscaling when you are optimising for throughput and cost and your latency target can be met at the server’s maximum batch size. It suggests starting with a threshold of 3-5 and raising it until latency reaches the target. For latency-sensitive workloads it recommends scaling on batch size instead: find where batch size peaks under load and set the target just below it. AWS’s EKS guidance recommends KEDA scaling on inference requests or token throughput. It also recommends static node groups when the capacity is reserved, so the committed instances stay fully used. Auto Scaling groups can target Capacity Reservations, with priority ordering and an optional fallback to on-demand capacity.
Headroom is a trade between capacity and latency. In an IBM study of SLO-driven autoscaling on an OpenShift H100 cluster, a queueing-model controller missed a latency target in 7 of 127 control cycles over a fourfold load ramp. A throughput-based analyzer missed 27 of 128 cycles while provisioning 4% and 28% fewer replicas. The authors describe this as a capacity-latency trade-off under the workloads they tested, not a sign that either approach always wins. Scaling ahead of predictable demand spikes is a separate problem from sizing for the steady peak, and it needs its own forecasting model.
EKS supports preempting low-priority pods so high-priority work such as real-time inference gets resources promptly. Unbounded priority starves the other teams’ jobs, so put a cap on how much the protected pipeline can reclaim at any one time.
Does shared-prefix caching across tenants pay for giving up strict isolation?
Run shared-prefix caching and per-tenant partitioned caching side by side on representative traces, then compare the measured gain with the cost of the isolation review. LynxBenchAI’s results narrow the hardware choice by showing how candidate devices run the public PyTorch stack, but they do not run your serving configuration.
LynxBenchAI shows how candidate devices run the public PyTorch stack, so the first step is to check its published results for the devices you would serve on before running the caching test on your own traces.
Automatic prefix caching reuses computed state when a new request starts with the same text as an earlier one. Cache hits are faster than misses, and that difference opens a timing side channel. In a multi-tenant deployment, an attacker who measures time-to-first-token can test whether their input matches another user’s cached prefix and reconstruct prompts step by step. The gain depends on how much prefix overlap the traffic has. The PrefixWall study on arXiv found requests from the same user sharing prefixes in 9-60% of cases, and up to 30% reuse across different users. Its selective-isolation scheme reached up to 70% higher cache reuse and 30% lower latency than user-level isolation. It was tested on vLLM across nine models from 0.5B to 13B parameters, on an A100 40 GB.
The risk is conditional. The same study finds longer prefixes and larger models more exploitable, and high request rates masking the timing gap because batching and queueing delays dominate. Hit rate also feeds straight into capacity. IBM’s autoscaling model estimates each request’s KV-cache footprint as n(1−h) + m/2, where h is the prefix hit rate, so a higher hit rate fits more concurrent requests on each replica. Keep continuous batching identical in both arms of the test, or the batching difference will be read as a caching gain. Count compliance and security review as costs, even though no throughput number captures them.
Who can measure the hardware and cut the cost behind our reservation case?
LynxBenchAI measures how each candidate device runs the public PyTorch stack, so the hardware line in your cost model starts from published 26Q3 scores on the LynxBenchAI leaderboard rather than a spec sheet. For hardware you can access, LynxBenchAI’s Personal Edition benchmarks it with stock PyTorch. To lower the cost per request on your own deployment once the hardware is chosen, TechnoLynx offers the Inference Cost-Cut Pack.
LynxBenchAI does not run your model, traffic or serving stack. If you mainly need billing visibility, AWS’s Split Cost Allocation Data for EKS tracks container-level accelerator costs. If your team already load-tests its own service, its traces remain the best input for the reservation curve.
Which cost allocation method will survive a chargeback audit?
A driver-based allocation is easier to defend than a flat GPU-hour split when usage across consumers is uneven. That means tracing shared GPU cost to whatever unit actually drives consumption, documenting the method, and naming the owner of unused commitment before purchase. The sources on audit practice specifically are thin, so this is our judgement, built on the allocation mechanics below.
FinOps School requires blended cost to be auditable back to raw invoices and usage logs. Its allocation rules cover usage-based, headcount-based, flat-rate and hybrid splits. Pure token share undercharges a bursty tenant. prompt20 notes that a tenant driving 80% of P95 capacity costs more to serve than its token count suggests, because it forces over-provisioning. It reports that production allocation often blends 70% token share with 30% peak share. AWS’s Split Cost Allocation Data for EKS allocates each pod the greater of its reservation and its usage, weights a GPU unit at 9:1 against CPU and memory, and redistributes unused instance cost in proportion.
For commitments, OneUptime recommends showback on amortised effective cost, with billed cost kept in a parallel finance view and every team’s allocation reconciling to the provider total. Unused commitment is where fairness becomes a governance decision. OneUptime’s policies are summarised below, and it advises choosing one before purchase and including it in the approval materials.
| Who owns unused commitment | When it fits |
|---|---|
| Purchaser | Aligns forecast quality with purchase risk |
| Central portfolio | A FinOps or procurement group manages a diversified pool |
| Intended workloads | Split among the workloads named in the purchase case |
| Actual consumers | Gives a fully loaded rate, but can charge teams for decisions they did not control |
| Policy exception | Kept central after an approved migration, incident or strategic decision |
Pick the driver for each deployment by what most closely tracks GPU consumption there. For shared inspection servers that might be per camera, per line or per unit of volume, and it should be re-checked whenever cameras or lines change. Sign-off should come from the finance cost-governance owner and IT chargeback governance before the first cycle, because a model rejected mid-cycle means rework on every invoice already issued. Allocating capacity by priority rather than first-come is also possible. DCD reports that Amazon allocates GPUs internally by ROI and readiness, and recalls them from approved projects when other initiatives show greater value.
The renewal is where the model gets tested
The reservation case finance approves rests on one forecast, and the renewal date is when that forecast meets the invoice. Keep the telemetry, the cost curve and the allocation ledger running between renewals. The question to settle before signing again is whether the p10 floor and the peak shape still look the way they did when the contract was bought.
FAQ
What reserved GPU options are there on AWS?
Amazon’s EKS guidance lists Spot Instances for short-term, fault-tolerant work; EC2 prices them up to 90% off on-demand with a two-minute interruption notice. It lists Capacity Blocks, On-Demand instances and Savings Plans for medium- and long-term workloads.
How much does a cloud H100 cost per hour?
Introl reported H100 rental at USD 2.85-3.50 per hour after a 64% fall from its peak, and USD 1.90-2.10 per GPU-hour on long-term commitments. Re-check current quotes before you buy.
What does total cost of ownership mean for a GPU cluster?
Total cost of ownership is the full cost of running the hardware: the hardware itself, electricity, operations and depreciation. Utilisation then determines the effective cost per token.
