In a text-generation service, the request boundary is obvious: one prompt in, one completion out. In a perception pipeline it is not. A “request” can be a decoded frame, a camera-second, a ten-second clip, or a single detected event — and the number you report changes by an order of magnitude depending on which one you pick. That ambiguity is why cost-per-request is harder to apply to vision workloads than to LLM features, and why so many vision cost models quietly stop being useful the moment someone tries to act on them.
The common shortcut is to divide the monthly GPU bill by the number of frames processed and call the result cost-per-frame. It produces a number. It rarely produces a decision.
Pick the billable unit before you pick the denominator
The unit should match what the product charges for, not what the pipeline happens to count. If you sell a camera subscription, the unit is a stream-hour. If you sell inspection on a line, it is an inspected part. If you sell a clip-analysis API, it is an analysed clip. If you sell alerting, it is a detected event — and that one is dangerous, because event cost is decoupled from compute: a quiet camera produces almost no events while consuming exactly the same GPU reservation as a busy one.
Frames make a poor billable unit in almost every case, because frame count is an engineering choice you control (sampling rate, resolution, keyframe-only decode) rather than a commercial quantity the customer recognises. Frames belong inside the model as an intermediate term, not as the headline denominator.
| Billable unit | Fits when | Main distortion to watch |
|---|---|---|
| Stream-hour | Camera subscriptions, continuous monitoring | Idle GPU reservation dominates on low-activity streams |
| Inspected part / item | Manufacturing and quality inspection | Line-rate variation makes cost swing between shifts |
| Analysed clip | Clip- or batch-priced APIs | Clip length variance; per-call warm-up amortised badly |
| Detected event | Alerting and per-incident pricing | Cost is decoupled from event count; scene activity sets margin |
| Frame | Internal engineering diagnostics only | Not a commercial quantity; invites sampling-rate gaming |
Once the unit is chosen, the arithmetic is a chain, not a division. Cost per stream-hour equals GPU-hour cost, divided by the number of concurrent streams that hardware actually sustains at your p95 latency target, multiplied by the reservation factor for capacity you hold but do not use. Every term in that chain is measurable, and every term is separately attackable.
Why does the detector rarely dominate the bill?
Because a perception pipeline is mostly not the detector. H.264 or H.265 decode, resize and colour-space conversion, letterboxing, tensor layout transforms, the tracker that associates detections across frames, and the post-process that turns boxes into product events all consume real time — some on the GPU’s video engines, some on CUDA cores, some on the CPU while the GPU waits. Add the reserved-but-idle share of an instance you provisioned for peak concurrency, and the model’s own forward pass can end up a minority of the cost per unit.
In a perception pipeline, non-model stages plus reserved-but-idle capacity frequently account for more of the per-request cost than the detector’s forward pass — which is why a cost model that counts only inference time reports a number that does not move when the bill moves. That is the practical failure: the reported figure and the invoice become uncorrelated, and the optimisation programme spends its effort on the one stage that had the least headroom.
This is also where a per-request figure earns or loses its right to exist. If you cannot decompose the number into stage-level contributions, it is a reporting artefact, not an optimisation target. Decomposition is what makes it actionable, because each stage has a different lever: decode moves to NVDEC, pre-process moves off the CPU into CUDA or DALI, the detector moves through TensorRT or ONNX Runtime with a precision change, the tracker moves to a cheaper association scheme or a lower cadence, and idle reservation moves through consolidation and autoscaling policy.
A worked decomposition
Take one perception feature: sixteen 1080p RTSP streams, a detector running on every third frame, a tracker filling the gaps, on a single GPU instance sized for peak concurrency. The measurement you want is a stage-level trace at p95 concurrency — not at idle, and not on a synthetic loop over a cached frame.
The checklist below is the minimum needed for a defensible baseline. It is the same evidence set our [inference-cost audit](Inference Cost-Cut Pack) collects against a deployed serving path, and it is deliberately unglamorous:
- Billable unit declared and written down, with the product’s price per unit next to it.
- Wall-clock time per stage per unit — decode, resize/colour conversion, detector forward, tracker update, post-process — captured with Nsight Systems or a PyTorch profiler trace on the deployed path, not a bench script.
- GPU occupancy and memory-bandwidth utilisation at p95 concurrency, so you know whether the device is saturated or merely reserved.
- Reservation factor: provisioned GPU-hours divided by GPU-hours doing useful work over the same window.
- p95 end-to-end latency and the threshold above which the product breaks, since every cost lever below trades against it.
- Frame-sampling and resolution settings recorded as configuration, because they silently change the denominator.
- Cost per unit before and after, with the stage shares summing to the total.
Three levers change cost-per-unit without touching the model at all. Batching across streams raises detector efficiency but adds queueing latency, so it is bounded by your p95 threshold rather than by throughput. Resolution reduction cuts decode, pre-process, and detector cost together — often the largest single win — but it has to be validated against detection recall at the smallest object size the product promises. Frame sampling with a tracker carrying the gaps reduces detector invocations directly, and its ceiling is set by object speed and the tracker’s tolerance for drift, not by cost appetite.
The stage-level attribution is only as good as the profiling behind it; we treat GPU profiling as the evidence layer under the cost model, because an estimated stage share will not survive a margin review. And the wider question of when per-request economics is the right target at all — versus a monthly spend ceiling, or unit economics per business action — belongs to the parent discussion of cost-per-request as the optimisation target for production AI. Teams running camera- or clip-priced features under margin pressure usually find the vision-specific part is the unit definition, not the arithmetic; the platform patterns we see across AI infrastructure and SaaS deployments are broadly the same once the denominator is honest.
Frequently Asked Questions
What does cost-per-request applied to a perception inference workload mean in practice?
Perception inference workloads demand different cost accounting than text-based AI because each image or video frame multiplies token equivalents by orders of magnitude. Asked about Cost Per Request Applied, most engineers point here. Looked at closely, Cost Per Request Applied is this. Asked about Cost Per Request Applied, most engineers point here. In the context of CostPerRequest Applied to a Perception Inference, it means expressing the recurring compute cost of a vision feature in the same unit the product charges for — a stream-hour, an inspected part, an analysed clip — and decomposing that figure so each pipeline stage’s contribution is visible. In practice the output is a single number with a stage-level breakdown behind it, measured at realistic concurrency rather than on an idle device., choose the unit that matches the commercial quantity: stream-hours for camera subscriptions, inspected parts for line inspection, clips for clip-priced APIs. Avoid frames as the headline unit, because frame count is an engineering setting you control and can therefore be gamed; event-based units are legitimate but leave cost decoupled from scene activity, which needs to be modelled explicitly.
How much of per-request cost comes from reserved-but-idle GPU capacity, and how is that measured?
Measure it as a reservation factor: provisioned GPU-hours divided by GPU-hours doing useful work over the same window, corroborated by GPU occupancy at p95 concurrency. On instances sized for peak stream counts this term is frequently one of the largest single contributors, and it is invisible to any cost model that only counts detector inference time.
What does a defensible cost-per-request baseline for a vision feature look like before optimisation starts?
A declared billable unit, per-stage wall-clock timings from a profiler trace on the deployed serving path, occupancy and reservation figures at p95 concurrency, the current p95 end-to-end latency, and the sampling and resolution settings in force. Stage shares must sum to the reported total; if they do not, the baseline is not yet a baseline.
If the decomposition holds, the remaining question is not which model to swap but which stage the product can afford to make cheaper — and how much recall or latency headroom you are willing to spend to get there.
Measuring success in perception workloads
Three numbers matter: latency at p99, cost per thousand inferences, and error rate under production load. Everything else is detail.
