A GPU that spends most of its time waiting for frames is still billed for the waiting. That single fact explains most of the disappointing video-analytics economics we are asked to diagnose: the model runs fine, the pipeline works, and the cost per analytics-hour is indefensible because the workload never had the shape a GPU needs. Frame rate, batch depth, event density and arrival pattern decide whether GPU acceleration pays long before the choice of detector architecture does.
This is the failure side of a profile-first position. GPU acceleration is not wrong for video analytics — but the profile has to look at the shape of the work, not only at the model.
What “wrong workload shape” means on a real channel
Take two analytics deployments that both use the same detector, the same runtime, the same instance type.
The first is a set of 25 fps channels feeding continuous object detection. Frames arrive on a metronome, they batch cleanly, and the device stays busy. Throughput measured in isolation — say the detector clears a few hundred frames per second at the working resolution — actually translates into channels served, because there is always another batch queued.
The second is a fleet of 2 fps channels, or a set of cameras that only fire analytics when a motion trigger opens a five-second clip. The same detector on the same GPU now sees a trickle. Between arrivals the device does nothing measurable. Kernel launches are short, batches are shallow or single-frame, and a meaningful share of each inference is fixed overhead — memory transfer across PCIe, pre-processing, launch latency — amortised over almost nothing.
Sparse-event and low-frame-rate workloads are the cases where a workload profile most often sends analytics back to CPU. Not because the CPU is faster per frame; it is not. Because the CPU is not being paid to idle.
Why does the model benchmark stay good while the economics collapse?
Because a model benchmark measures the model, under saturation, with frames already resident. It answers “how fast can this detector go when nothing starves it?” — a question no sparse channel ever asks. The channel asks “what does one analytics-hour cost on this device given my arrival pattern?”, and the answer depends on occupancy, which the benchmark deliberately holds at 100%.
In our media and broadcast engagements the gap between those two numbers is the recurring surprise. A team provisions from vendor throughput figures, then finds device occupancy under real arrival patterns well below what the cost assumption implied (observed pattern across TechnoLynx GPU audits; not a published benchmark rate). The hardware is fine. The placement was decided against the wrong measurement.
The four shape variables, and what each one breaks
| Shape variable | Healthy for GPU | Breaks GPU economics when | What it damages |
|---|---|---|---|
| Frame rate | Analytics genuinely needs 15–30 fps per channel | Analytics output only needs 1–5 fps (periodic inspection, occupancy counts, logo checks) | Occupancy — the device idles between frames it was sized to consume |
| Batch depth | Many concurrent channels batch into deep tensors | Few channels, or per-channel latency deadlines force batch size 1 | Per-frame overhead dominates; effective throughput falls far below the benchmark |
| Event density | Events are near-continuous, so the pipeline runs steady-state | Triggers fire minutes apart; clips are short | Cost per analytics-hour — you pay for a reserved device to serve seconds of work |
| Arrival pattern | Smooth, predictable, schedulable | Bursty and correlated (all cameras trigger on the same real-world event) | Both ends: idle in the gaps, queued and over-deadline in the bursts |
The variables interact, which is why reading them one at a time misleads. Bursty arrival plus a tight latency deadline is the worst combination available: the burst is exactly when batching would help, and the deadline is exactly what forbids waiting to form a batch.
The batching-versus-latency conflict on event-driven analytics
GPU economics improve with batch depth. Event-driven analytics improve with immediacy. These pull against each other and there is no configuration that satisfies both.
If a trigger-to-alert budget is 200 ms and a batch window of 150 ms is what it takes to assemble a useful batch, the batching that would make the GPU pay consumes the budget that justified the alert. Teams usually resolve this by dropping batch size to 1 and accepting the overhead — which is a defensible engineering call and an indefensible provisioning call, because the device is now being paid GPU rates to run at CPU-like effective utilisation.
The alternative resolutions are worth naming plainly:
- Split the deadline. Run the trigger-side classification on CPU inside the latency budget, and batch the heavier enrichment (re-identification, attribute classification, indexing) on GPU as best-effort behind it. Deep-batch work goes where deep batches exist.
- Consolidate triggers. If the bursts are correlated, a shared batch window across channels can reach useful depth without extending any single channel’s wait. This works only when arrivals really are correlated.
- Stop reserving. Where events are rare enough, a shared or on-demand device beats a dedicated one, and the accounting changes more than the engineering does.
This is where our work on GPU acceleration and profiling tends to start: not with the model, with the arrival trace.
What an audit actually measures
A GPU Performance Audit scoped to video analytics does not begin by benchmarking the detector. It instruments the workload:
- Achieved frames per second per channel, separated from the source frame rate — analytics frequently needs far less than the stream provides, and the difference is free money.
- Realised batch depth distribution at the inference stage under production traffic, not the configured maximum.
- Device occupancy over time, sampled across a full duty cycle including quiet hours, so the idle fraction is visible rather than averaged away.
- Inter-arrival intervals for event-triggered channels, with burst correlation across the fleet.
- Cost per analytics-hour on the sparse subset, compared against a CPU-serviced equivalent doing the same job.
The counter-metric matters as much as the headline one. The batchable, high-frame-rate subset where the GPU genuinely pays should not be starved of capacity to fund a fleet-wide rollout onto channels that will never saturate a device. We see that trade made backwards often enough to check for it deliberately: the dense channels get squeezed so the sparse ones can be “accelerated”.
When CPU is the right answer, and how to mix without fragmenting
A video-analytics stage belongs on CPU when the arrival pattern cannot fill a batch, when the per-frame work is small relative to transfer and launch overhead, or when the required output rate is low enough that a CPU core clears it inside the deadline. Periodic compliance checks, low-rate occupancy counting, and metadata post-processing are common examples.
Mixing devices in one pipeline is a real cost, not a free option — every GPU↔CPU boundary is a copy and often a format conversion. The way to keep it manageable is to make device placement a property of the stage, declared once, rather than a per-deployment improvisation: one container image carrying both paths, one orchestration layer (Docker and Kubernetes do this adequately), and ONNX Runtime or a similar runtime with both execution providers available so the same graph can be scheduled either way. The deployment stays single-shaped; only the placement decision varies per channel class.
The structural causes of this — why the profile, rather than the framework default, should decide what runs where across a media pipeline — are developed in the parent analysis of GPU acceleration economics in media and broadcast pipelines.
The number that tells you the placement is wrong
Two figures, read together, are usually enough. Sustained device occupancy under real arrival patterns below roughly a third, combined with a cost per analytics-hour above the CPU-serviced equivalent for the same output, means the workload is on the wrong device. Neither figure alone is conclusive — low occupancy with a hard latency justification can still be the right call, and a high cost per analytics-hour can be worth paying when the analytics output is what the business sells.
What remains genuinely uncertain is the boundary case: workloads whose event density is rising. A channel set that fires sparsely today and continuously in eighteen months has a shape that will move, and provisioning against today’s trace can be as expensive as provisioning against a vendor benchmark. How often is the arrival trace worth re-measuring before the shape argument stops holding?
Frequently Asked Questions
Which workload-shape characteristics actually determine whether a GPU pays on a video-analytics stream?
Four: the frame rate the analytics output genuinely requires (not the source rate), the batch depth realised in production, event density, and arrival pattern. Each independently caps device occupancy, and they interact — bursty arrival combined with a tight latency deadline is the least GPU-friendly shape there is.
Why do low-frame-rate channels and sparse event-triggered clips break GPU economics even when the model runs well on GPU?
Because a model benchmark measures the detector under saturation, while the channel’s economics depend on occupancy. At 2 fps or with triggers minutes apart, the device is idle most of the duty cycle and fixed per-inference overhead — transfer, launch, pre-processing — is amortised over almost no work.
How do batching and latency targets conflict on event-driven analytics?
Batch depth is what makes a GPU cheap per frame; immediacy is what event-driven alerting is for. If assembling a useful batch costs more than the trigger-to-alert budget allows, the batch window has to go — leaving batch size 1, high per-frame overhead, and a device paid at GPU rates for CPU-like effective utilisation.
When is CPU the correct answer for a video-analytics stage?
When the arrival pattern cannot fill a batch, when per-frame compute is small relative to transfer and launch overhead, or when the required output rate clears comfortably on a CPU core inside the deadline. Periodic inspection, low-rate counting, and metadata post-processing frequently fall here.
What utilisation and cost figures indicate a workload has been placed on the wrong device?
Sustained occupancy under real arrival patterns below roughly a third, together with a cost per analytics-hour higher than the CPU-serviced equivalent for the same output. Read them together — low occupancy alone can be justified by a hard latency requirement.
Spotting incompatible workload geometry early
Sparse frame sampling, low concurrency, or bursty arrival patterns all waste GPU occupancy and push total cost above CPU equivalents. Everything else is detail.
