The GPU-versus-CPU question in video analytics is not a hardware-tier comparison. It is a question about four workload properties — sustained concurrent stream count, the frame rate the analytics output actually requires, model complexity per inference stage, and how much of the pipeline is decode and pre-processing rather than inference. Read those four before you price anything, because they determine whether a GPU tier stays saturated or spends most of its billed hours waiting.
The naive route is easy to recognise. Someone measures frames per second on a GPU, measures frames per second on a CPU, sees a large multiple, and concludes the GPU wins. That comparison is true and irrelevant. Faster per frame does not mean cheaper per analytics-hour, and cost-per-analytics-hour is the denominator that survives a procurement review.
What “earns its cost” means as a measurable quantity
A GPU analytics tier earns its cost when its cost-per-analytics-hour, measured under production concurrency, is lower than the same workload’s cost-per-analytics-hour on a CPU tier — and when both sit below the value-per-analytics-hour the analytics output generates. That is the whole test. Everything else in this article exists to tell you which side of it a given workload lands on.
Two numbers make that test concrete. The first is sustained GPU utilisation under real concurrency, not under a synthetic loop that feeds frames as fast as the device will take them. The second is cost-per-stream at the target frame rate, where the target is the rate the analytics needs rather than the rate the source delivers. In our GPU audit work on media pipelines, the gap between those two frame rates is one of the most common sources of over-provisioning we encounter (observed across TechnoLynx engagements; not a published benchmark).
Why does source frame rate mislead the decision?
A 50fps broadcast feed does not imply 50 inferences per second. Logo detection on a live channel might need two frames per second to satisfy its compliance requirement. Scene-boundary detection needs frames at cut density, not at source density. Person tracking needs enough temporal continuity to associate detections, which is often 8–12fps rather than 50. When the analytics requirement is expressed honestly, the inference load frequently drops by an order of magnitude — and a workload that looked GPU-bound at source rate turns out to be decode-bound at required rate.
Decode is the trap here. If you decode at full source rate to sample a fraction of frames for inference, the pipeline’s dominant cost is the decode stage, and a GPU’s general compute cores are idle while its fixed-function decoder does the work. That is a different purchase than a GPU inference tier. It is also the case where hardware decode on a cheaper device, or CPU decode with frame dropping before the decoder, usually wins.
The four-property routing rubric
| Property | Favours GPU tier | Favours CPU tier | What to measure |
|---|---|---|---|
| Sustained concurrent streams | Many streams, continuously arriving, batchable into full batches | Few streams, or bursty arrival with long idle gaps | Achieved batch occupancy at production concurrency |
| Required output frame rate | Analytics needs frames at or near source rate | Analytics needs sampled frames (sub-10fps) | Inferences per second per stream, derived from the analytics requirement |
| Model complexity per stage | Dense models — segmentation, multi-class detection at high resolution, transformer-based classification | Light models — thresholding, motion masks, small classifiers, classical CV in OpenCV | Per-stage kernel time vs per-stage host overhead |
| Decode / pre-processing share | Inference dominates wall-clock time per frame | Decode, colour conversion and resize dominate | Per-stage timing breakdown across the whole graph |
Three or four properties pointing GPU-ward is a defensible GPU commitment for that function. One or two is a split: route the dense, high-concurrency functions to GPU and keep the rest on CPU. Zero is a CPU tier that someone is about to replace for the wrong reason.
The important output is that split, function by function, rather than a fleet-wide answer. A media pipeline running content indexing, compliance logging, and highlight detection on the same streams will often justify GPU for one of the three and not the other two.
The utilisation threshold, and why it is not a single number
There is no universal utilisation figure below which a GPU tier stops paying for itself, and anyone quoting one is quoting their own workload. The threshold is set by the price ratio between the two tiers. If a GPU instance costs roughly six times a comparable CPU instance and delivers roughly twenty times the inference throughput on your model, the GPU has headroom to sit substantially idle and still win. If it costs six times and delivers eight times, sustained utilisation has to stay high or the CPU tier is cheaper per analytics-hour.
What we do look for is the shape of the utilisation trace rather than its mean. A device averaging 40% because it alternates between saturation and idle gaps is telling you the pipeline is starved — usually by decode, host-to-device copies, or serialised pre-processing — and that fixing the starvation is cheaper than buying more devices. A device averaging 40% because every batch runs at 40% occupancy is telling you the concurrency is genuinely too low for the tier you bought. The first is an engineering problem; the second is a routing decision. We explore the underlying failure mode in the GPU underutilisation pattern and what a low utilisation number actually indicates, which is the condition this rubric is designed to catch before capacity is committed.
Functions that reliably stay on CPU
Some analytics functions in a media pipeline do not become cheaper on a GPU regardless of how much capacity is available, because their compute profile is wrong for the device:
- Metadata assembly, serialisation and database writes — host-bound by definition.
- Sparse event triggers where inference fires on a handful of frames per hour.
- Light classical CV — frame differencing, histogram checks, black-frame and silence detection — where per-call launch overhead exceeds kernel time.
- Audio-side analysis running alongside video, where model sizes are small and the stream count per core is high.
- Any stage whose input arrives one frame at a time with a hard per-frame deadline that forbids batching.
Keeping these on CPU is not a compromise. It is the avoided spend that makes the GPU tier’s economics defensible, and it should appear explicitly in the cost model rather than being absorbed silently into a fleet-wide GPU line item.
What the profiling step looks like before approval
Before a GPU analytics build-out is signed off, the pipeline needs a per-stage timing breakdown taken under the real workload mix — real resolutions, real concurrency, the actual model set, the actual arrival pattern. That means decode time, pre-processing time, per-model inference time, post-processing time and transfer time, each separated, each at production concurrency rather than single-stream. Concurrency headroom matters too: the point at which adding streams stops increasing throughput tells you what a device can actually carry, and that number is almost always below the vendor figure.
Our GPU engineering and performance work supplies those profile numbers, and for broadcast and media operators the same audit output feeds directly into the per-function routing table above. The per-stage timing model behind the frame-rate and model-complexity rows comes from the same latency framing we use in how GPU latency is decomposed across pipeline stages.
The decision this rubric produces is narrower than most GPU proposals and easier to defend: a named set of analytics functions on GPU, a named set retained on CPU, and a cost-per-analytics-hour figure for each that was measured rather than extrapolated. The harder question, and the one worth asking before the profiling starts, is which of your analytics outputs actually carries enough value-per-analytics-hour to justify either tier.
Frequently Asked Questions
What does “GPU video analytics earns its cost over CPU pipelines” mean in practice, and how is it measured?
GPU pipelines justify their premium once you process more than twelve simultaneous video streams at production frame rates. It means the workload’s cost-per-analytics-hour on a GPU tier, measured under production concurrency, is lower than the same workload’s cost on a CPU tier — and below the value the analytics output produces. It is measured with a per-stage timing breakdown plus sustained utilisation at real stream counts, never with a single-stream throughput figure.
Which workload properties decide the GPU-versus-CPU split?
Four: sustained concurrent stream count, the frame rate the analytics output actually requires, model complexity per inference stage, and the share of pipeline time spent on decode and pre-processing rather than inference. Dense inference on many concurrent streams favours GPU; decode-bound, low-frame-rate or light-model work usually does not.
At what sustained GPU utilisation does a GPU analytics tier stop paying for itself?
There is no portable number — the threshold is set by the price ratio between the two tiers and the throughput ratio on your specific model. What matters more is the shape of the utilisation trace: intermittent saturation points to a starved pipeline worth fixing, while uniformly low batch occupancy points to a routing decision.
Which analytics functions in a media pipeline reliably stay on CPU even when GPUs are available?
Metadata assembly and database writes, sparse event triggers, light classical CV such as black-frame or histogram checks, audio-side analysis, and any stage with a hard per-frame deadline that forbids batching. Their kernel time is smaller than their launch and transfer overhead, so acceleration does not reduce cost per output.
Recognizing the cost crossover
Once your pipeline sustains more than fifteen 1080p streams or needs per-frame results under 80 ms, GPU economics flip from luxury to necessity. The teams that do tend to ship the boring, correct version first.
