A vendor throughput figure — streams per GPU at a reference resolution — is not a sizing input. It is a measurement taken on a workload that is not yours, and dividing your stream count by it is arithmetic, not engineering. Sizing a GPU fleet for video analytics means measuring achieved throughput per inference stage on the candidate instance, under your real workload mix, and composing those stage profiles into a stream ceiling you can defend at peak.
That distinction is not academic. The headline-figure route fails in one of two directions, and both are expensive: over-provision, and you pay for GPU capacity that sits underutilised for the life of the reservation; under-provision, and you discover the gap during a live event and re-tier under pressure.
Why don’t vendor stream counts transfer to your workload?
Because a published streams-per-GPU number is a composite of decisions the publisher made and you did not. Resolution, frame rate, batch size, model set, the number of concurrent streams sharing the device, whether decode runs on the hardware decoder block or in software, whether tracking and post-processing were in the graph at all — every one of those moves the number, and most published figures optimise all of them simultaneously.
A stage-level throughput profile predicts an instance’s stream ceiling within a usable planning margin; a headline throughput figure predicts nothing about your pipeline except its upper bound. That is the whole methodology in one sentence, and it is why we treat sizing as a profiling exercise rather than a spec-sheet exercise.
The second reason is structural. Detection is rarely the stage that runs out first. Decode consumes fixed-function hardware capacity that is separate from the general compute cores; tracking is often latency-bound rather than throughput-bound; metadata assembly and post-processing frequently sit on the CPU side of the PCIe boundary and stall the GPU stages waiting on them. A single composite figure hides which of those binds first, so it cannot tell you what to buy.
The sizing method, stage by stage
Work in this order. Skipping a step does not save time; it moves the discovery later, when it costs more.
- Fix the workload mix before you measure anything. Resolution, source frame rate, the frame rate the analytics actually needs (often lower than the source), model set and precision, batch depth, and the sustained concurrent stream count. If you cannot state these, you are not ready to size — you are still designing.
- Profile each stage in isolation on the candidate instance. Decode, detection, tracking, classification, post-processing, metadata write. Record achieved frames or streams per second per stage, plus device utilisation and memory footprint. One instance type, one workload, measured — not extrapolated.
- Profile the composed pipeline on the same instance. The composed number is always lower than the weakest stage’s isolated number, because of memory transfers, format conversions, and scheduling contention. The gap between composed and weakest-isolated is your integration overhead, and it is the figure most estimates omit entirely.
- Identify the binding stage. The stage that saturates first sets the ceiling. Everything above it in the profile is spare capacity you are already paying for.
- Test the CPU alternative for every non-binding stage. Where a stage profiles poorly on GPU — low occupancy, small kernels, heavy host-device copies — moving it to CPU releases GPU capacity for the stages that use it well. That released capacity raises the stream ceiling without changing the instance tier.
- Compose the ceiling, then apply headroom. Streams per instance at steady state, then the peak multiple your workload actually experiences. Headroom is a measured property of your traffic pattern, not a round number.
- Price the tiers. Cost-per-analytics-hour at each viable instance tier, at the composed ceiling — not at the isolated best-case throughput.
What each workload variable does to the ceiling
| Variable | Direction of effect | Where it usually binds first |
|---|---|---|
| Resolution | Higher resolution reduces streams per instance super-linearly once decode or memory bandwidth saturates | Decode block, device memory |
| Frame rate | Analytics frame rate, not source frame rate, sets inference load; decimating input is often the cheapest capacity gain available | Inference stage |
| Batch size | Larger batches raise inference throughput but raise per-frame latency; capped by the real-time deadline | Inference throughput vs latency budget |
| Concurrent stream count | Adds scheduling contention and per-stream memory overhead; throughput per stream falls before the device is nominally full | Scheduler, device memory |
| Model set | Each additional model in the graph is an additional saturation candidate, not a proportional cost | Whichever model has the worst occupancy |
| Post-processing placement | GPU-side post-processing competes with inference; CPU-side adds transfer cost | PCIe boundary |
Use the table as a diagnostic order, not a formula. The point is that each variable has a different binding stage, which is why one composite number cannot represent all of them.
Fewer large instances, or more small ones?
Both configurations can serve the same aggregate stream load at similar list cost, so the deciding factors are elsewhere. Larger instances win when stages share decoded surfaces in device memory and when batch depth needs headroom — keeping a stream’s stages co-resident avoids transfers entirely. Smaller instances win when the workload is bursty, when failure-domain size matters (losing one node should not take out a quarter of your channel capacity), and when scaling granularity needs to match how channels are actually added.
In our experience the memory footprint of the composed pipeline settles this more often than throughput does. If one stream’s working set plus model weights leaves little room for concurrency, the larger tier is not optional. Profile the footprint before you argue about price.
The related question of running analytics alongside an existing transcode workload on the same device is its own decision — decoding once and fanning the surface out to both paths changes the sizing arithmetic materially.
What correct sizing looks like afterwards
Sustained GPU utilisation in a defensible band, a per-instance stream ceiling that holds at peak without emergency re-provisioning, and a stated cost-per-analytics-hour at the chosen tier. Where a stage moved to CPU, the measurable outcome is the CPU-serviced share of the pipeline and the GPU capacity that share released. Chronic low GPU utilisation across a video-analytics fleet is usually a sizing artefact rather than a tuning problem — it is the direct signature of provisioning against a headline figure instead of a stage profile.
For the broader economics of when GPU acceleration is the right call for a broadcast pipeline at all, see our GPU-accelerated video processing for media and telecom work, and the GPU engineering practice page for how stage-level profiling is run. The prior question — whether GPU earns its cost against a CPU pipeline for a given workload — is worked through separately in when GPU video analytics earns its cost over CPU pipelines.
The uncertainty we would name honestly: stage profiles are stable against workload drift only as far as the workload mix is stable. Add a model, raise the analytics frame rate, or change the resolution mix, and the binding stage can move. That is an argument for re-profiling on change, not for going back to arithmetic.
Frequently Asked Questions
What does sizing GPU instances for a video-analytics workload mean in practice?
With Size GPU Instances Video, the detail that matters is this. It means measuring achieved throughput for each pipeline stage — decode, detection, tracking, classification, post-processing — on a specific candidate instance under your real workload mix, then composing those measurements into a stream ceiling per instance and an instance count. The output is a tier plus a count you can defend with numbers, not an estimate derived from a published figure.
What do you measure to get a defensible throughput-per-GPU figure, and why don’t vendor stream counts transfer?
Measure per-stage frames or streams per second, device utilisation, and memory footprint, both in isolation and in the composed pipeline. Vendor stream counts do not transfer because they encode a resolution, frame rate, batch size, model set, and stage list chosen to maximise the number — usually with post-processing and tracking excluded.
How do resolution, frame rate, batch size, and concurrent stream count change the per-instance stream ceiling?
Each binds at a different stage: resolution pressures the decode block and device memory, analytics frame rate drives inference load, batch size trades throughput against per-frame latency, and concurrent stream count adds scheduling contention and per-stream memory overhead. Because the binding stage differs, no single scaling factor covers them all.
How do you compose stage-level profiles into an instance tier and instance count?
Find the stage that saturates first under the composed pipeline — that stage sets the ceiling — then subtract the integration overhead visible between the composed and isolated measurements. Apply the peak multiple your traffic pattern actually shows, and divide the total stream load by the resulting per-instance ceiling.
Which stages should be sized for CPU instead of GPU, and how does that change the GPU tier you buy?
Stages with low GPU occupancy, small kernels, or heavy host-device copies — commonly metadata assembly and parts of post-processing — often return better economics on CPU. Moving them releases GPU capacity for the binding stage, which can raise the stream ceiling enough to keep you on a cheaper tier than the composite estimate would have selected.
Putting instance sizing into practice
Start with stream count and frame budget, then benchmark two adjacent instance families under production traffic patterns. Everything else is detail.
