Most GPU video-analytics numbers you will be shown are headline speedups with no workload named, no stage boundaries and no cost denominator. A worked example needs all three: one representative pipeline, measured stage by stage, with the money attached. This article walks that read-out — what the pipeline was, where the boundaries were drawn, what each stage sustained, what an analytics-hour cost, and what the accuracy looked like at the throughput point actually run rather than the one in the datasheet.
The point is not that these particular figures transfer to your pipeline. They will not. The point is the shape of the table you should be able to produce about your own workload, and what its rows tend to reveal.
What is the representative workload, and where are the stage boundaries?
The workload is a file-based media analytics pipeline: 1080p29.97 source, H.264 in, five analytics functions chained per stream — hardware decode, object detection, multi-object tracking, a secondary classification pass on tracked crops, and a metadata write to an index. Concurrency held at whatever the device sustained without frame drop, not at a target chosen in advance.
Stage boundaries matter more than most teams expect, because they decide who gets blamed for the cost. We bound them at memory ownership changes rather than at code-module edges:
- Decode — from bitstream ingest to a decoded surface resident in device memory. NVDEC does the work; the boundary closes when the surface exists, not when it has been colour-converted.
- Pre-process — resize, colour-space conversion, normalisation, batch assembly. This is its own stage because it is where PCIe copies hide.
- Detection inference — the batched forward pass only. TensorRT engine, FP16.
- Tracking — association and state update across frames. Largely serial, largely CPU-friendly.
- Classification inference — second forward pass over cropped regions, batch depth set by how many objects the tracker is actually holding.
- Metadata assembly and write — serialisation and index insert.
Draw the boundaries anywhere else and the numbers stop meaning anything. Fold pre-processing into “inference” and inference looks expensive. Fold decode into “the GPU pipeline” and the GPU looks efficient because a fixed-function block is carrying it.
The measured read-out
Figures below are an operational measurement from one profiled media-analytics pipeline on a single mid-range datacentre GPU; they are a project-specific outcome, not a published benchmark, and they do not port to a different model set, resolution or concurrency point.
| Stage | Sustained throughput | Mean GPU utilisation | Utilisation shape | Where it ended up |
|---|---|---|---|---|
| Decode | ~34 streams/GPU | n/a (NVDEC block) | Steady; compute cores idle | GPU (fixed-function) |
| Pre-process | Matched decode | 38% | Bursty, PCIe-bound when done off-device | GPU, kept in device memory |
| Detection inference | 22 streams/GPU @ 10 fps analysed | 71% | Sustained under batch depth ≥8 | GPU |
| Tracking | 60+ streams/GPU-equivalent | 9% | Idle between frames; serial dependency | Moved to CPU |
| Classification inference | Event-density dependent | 24% | Long idle gaps, shallow batches | GPU, batched across streams |
| Metadata write | Not GPU-bound | <2% | Idle | Moved to CPU |
Two rows carry the whole argument. Detection held 71% sustained utilisation once batch depth reached eight or more frames — that stage repays the hardware. Tracking sat at 9% because it is a serial association problem waiting on one frame at a time; it was billed for waiting. Sustained utilisation per stage, not aggregate device utilisation, is what tells you which stages are paying for the GPU they occupy. Aggregate utilisation across the whole graph read 41%, a number that would have justified nothing in either direction.
The per-stage utilisation collapse we measured here is a concrete instance of the underutilisation pattern we describe in our work on why GPU utilisation figures mislead, where the same signal shows up in non-media workloads.
Cost per analytics-hour against value per analytics-hour
Throughput without a denominator is trivia. The denominator we use is the analytics-hour: one hour of one stream analysed at the configured function set. It survives comparison across instance types and across mixed CPU/GPU placement, which streams-per-GPU does not.
For this workload, after tracking and metadata moved to CPU, the GPU carried 22 concurrent detection streams instead of the 14 it managed when all six stages sat on the device. Same hardware, same hourly instance price, 57% more analysed streams — so cost per analytics-hour fell by roughly a third, entirely from placement rather than from any kernel optimisation. That is an operational measurement from the profiled pipeline, and the mechanism is unremarkable: two stages stopped occupying capacity they were not using.
Value per analytics-hour is the side clients tend not to have quantified. In this case the analytics output fed a rights-and-compliance index, and the value was expressed as review-hours avoided per analytics-hour. Once both numbers existed on the same page, the interesting conversation stopped being “can we make the GPU faster” and became “which functions are worth running at all on the long-tail archive”. A defensible per-stream cost figure is the thing that lets that conversation happen at production volume; a vendor speedup multiple is not.
What quality delta appears at the operating point actually run?
This is the row most worked examples omit. Accuracy is reported at the benchmark configuration — full frame rate, generous input resolution, batch size chosen for the paper — and then the pipeline runs at a different point entirely because that is what the cost target allowed.
Here, detection was run at 10 analysed frames per second against 29.97 source, with input resized to the detector’s native tile rather than a higher-resolution tile. Recall on small, fast-moving objects dropped measurably against the full-rate configuration; recall on the object classes the index actually queried did not move enough to matter, because tracking recovered identity across the skipped frames. Classification accuracy was effectively flat, since it operated on tracker-selected crops rather than raw frames.
That asymmetry is the useful finding. Frame-rate reduction is not a uniform accuracy tax — it is a tax paid almost entirely by fast, small objects, and if those are not what the downstream index cares about, it is close to free. If they are, the saving is illusory and the operating point has to move back. Measuring accuracy only at the benchmark point hides which of those two situations you are in. We explore the structural reasons a media pipeline’s economics only hold for a subset of analytics functions in our broader position on GPU acceleration in media and broadcast pipelines.
Reproducing this on a different pipeline
The read-out above took a scoped profiling pass, not a full engagement. The reproducible core is short:
- Fix one representative stream configuration — resolution, frame rate, codec, function set — and hold it constant across every measurement.
- Instrument at memory-ownership boundaries, not module boundaries. Nsight Systems traces plus per-stage wall-clock counters are usually enough; the goal is to see idle gaps, not to micro-optimise kernels.
- Record sustained utilisation over minutes under concurrency, not peak utilisation over seconds. Peak tells you the stage can use the device; sustained tells you whether it does.
- Convert every stage to cost per analytics-hour before deciding anything. Streams-per-GPU is not comparable across placement changes.
- Re-measure accuracy at the throughput point you intend to ship, on the object classes the downstream consumer queries.
The one part that does not transfer is the placement conclusion. Tracking went back to CPU here because event density was moderate and the tracker was association-heavy; on a dense-scene pipeline with a learned re-identification model, the same stage can hold enough batch depth to justify the device. That is why the profile is the deliverable and the placement decision is only its output.
Which leaves the question worth carrying into your own workload: if you cannot currently state, per stage, what your pipeline sustains and what an analytics-hour costs, what exactly is your capacity plan built on?
Frequently Asked Questions
What does GPU-accelerated video analytics actually deliver in practice — a worked example? On the profiled pipeline described above, it delivered 22 concurrently analysed 1080p streams per GPU at 10 analysed frames per second across detection, classification and metadata indexing — after two stages were moved off the device. Before that placement change the same hardware carried 14. What GPU acceleration delivered was concentrated in one stage, detection, which held 71% sustained utilisation; the rest of the graph delivered little and cost the same.
What is the representative media workload used, and how are its pipeline stages bounded for measurement? A file-based 1080p29.97 H.264 pipeline running decode, pre-process, detection, tracking, classification and metadata write. Stages are bounded at memory-ownership changes rather than code-module edges, so that PCIe copies and batch assembly are attributed to their own stage instead of hiding inside “inference”.
What throughput per GPU does each stage sustain, and where does utilisation collapse between batches? The table above records per-stage figures: decode ~34 streams via the fixed-function block, detection 22 streams at 71% sustained utilisation, tracking equivalent to 60+ streams but only 9% utilisation. Utilisation collapses in tracking and metadata write, both of which are serial and frame-at-a-time, and partially in classification, where shallow crop batches leave long idle gaps.
What is the cost-per-analytics-hour for this workload, and how does it compare with value-per-analytics-hour? Cost per analytics-hour fell by roughly a third purely from moving tracking and metadata write to CPU, since the GPU then carried 57% more detection streams at the same instance price. Value was expressed as downstream review-hours avoided per analytics-hour; with both numbers on one page, the decision shifted from optimising the GPU to choosing which functions to run on the archive at all.
What quality delta appears between the benchmark operating point and the throughput point actually run in production? Running detection at 10 analysed fps instead of full source rate cost measurable recall on small, fast-moving objects and almost nothing on the classes the downstream index queried, because tracking recovered identity across skipped frames. Classification accuracy was effectively flat, as it saw tracker-selected crops. Frame-rate reduction is not a uniform accuracy tax, which is why accuracy has to be measured at the shipped operating point.
Which stages in this worked example ended up back on CPU, and what evidence drove that decision? Tracking and metadata write. The evidence was sustained utilisation — 9% and under 2% respectively — held over minutes under concurrency, not peak samples. Both stages are serial and frame-dependent, so they occupied device capacity while waiting rather than while computing.
How would the same measurement be reproduced on a different media pipeline without re-running the whole engagement? Fix one representative stream configuration, instrument at memory-ownership boundaries, record sustained rather than peak utilisation under concurrency, convert every stage to cost per analytics-hour, and re-measure accuracy at the intended shipping throughput. The method transfers; the placement conclusion does not, because it depends on event density and batch depth in your specific workload.
Why this architecture persists
GPU acceleration proves its value when latency SLAs tighten or concurrent stream counts exceed what CPU-only infrastructure can absorb. The teams that do tend to ship the boring, correct version first.
