In practice, AI media means machine learning running on broadcast and streaming video: captioning, metadata tagging, automated quality control, highlight clipping, content-aware encoding and live analytics. Each of these jobs has to decode compressed video before the model sees a single pixel, so the codec runs ahead of the model on every frame. That makes codec and compression engineering a first-order lever. H.264 and H.265 were designed for people watching screens. They were not designed to feed inference or to close a low-latency control loop. The compression layer is engineering surface to design for the workload, not something to inherit from transport defaults.
Compression is already a quality lever on the distribution side. PlayBox Technology describes Netflix’s Dynamic Optimizer, which analyses the complexity of each frame to decide how hard it can be compressed, and reports up to a 20% bitrate reduction with no noticeable quality loss. The same thinking applies when the consumer of the video is a model rather than a viewer.
On the sports side, AI in Olympic and sports broadcasting covers what AI could change for live events and AI engineering for sports analytics covers the pipelines behind it, while where computer vision pays in media and entertainment covers vision models in production and post.
AI media work starts at the decoder
GPU compute keeps getting faster, and the stages that feed it (storage I/O, host-to-device PCIe transfers, CPU decoding and resizing) do not always keep up. The result is an accelerator that sits waiting for data. NVIDIA’s developer blog calls this GPU starvation. The same NVIDIA post puts the point plainly: AI pipelines do not only need faster models, they need data arriving at the rate the model consumes it.
Is it the codec or the model that is slowing our pipeline down?
You can only answer this by measuring. The CoVA paper (2022) studied Tahoma, an existing cascade video analytics system, on an NVIDIA RTX 3090. The cascade improved performance by up to 327× over a DNN-only approach. Even with the NVDEC hardware decoder, though, decoding throughput was far below what the cascade could process, and that cancelled most of the gain. In CoVA’s measurements, the decode-bound cascade ran at NVDEC’s H.264 throughput of 1,431 FPS. The same authors found decoding throughput falling almost linearly as resolution rose.
After the authors moved most analysis into the compressed domain, some datasets were still limited by the decoder and others by the DNN object detector. The same pipeline design was limited by different stages depending on the dataset. Profile before deciding whether the fix belongs in the codec or the model, because best practice only makes sense in the context profiling gives.
A case study of video analytics pipelines on farm footage reached a similar conclusion. The bottlenecks came from scheduling, data flow and the way components interact, not only from neural network inference. The fixes gave up to a 2× speedup with no loss of model accuracy. The pipeline sampled 2 FPS from 30 FPS video, but cv2.read() decoded every frame, so 14 of every 15 decoded frames were thrown away. OpenCV’s grab() and retrieve() split avoids decoding frames that are never processed. Moving detection from one process to four cut detection time by 47% and raised GPU utilisation from around 20% to over 50%.
One instinctive response to slow detection is to add GPUs. Wowza, a streaming vendor, makes the counter-argument: when receiving, decoding and protocol translation upstream are saturated, extra GPUs just wait for frames, and GPU utilisation can stay low as the feed count grows. Idle accelerators are a cost of their own, covered in the hidden cost of GPU underutilisation.
A diagnostic checklist for decode-bound pipelines
- Run the inference stage alone (source, streammux, nvinfer, fakesink) and sweep the batch size to find the model’s plateau FPS. NVIDIA’s DeepStream profiling workflow uses this to separate the model’s true peak from everything else.
- Compare decoded frames per second per stream against that plateau. If decode delivers fewer frames than the model can take, the model is not your bottleneck.
- Read the microbenchmark shape alongside
nvidia-smi dmonfor decode-bound behaviour, andcuda_gpu_mem_time_sumfor time spent in memory copies. - Profile the end-to-end pipeline under Nsight Systems and pull per-plugin timings with
nsys stats. - Check whether frames are being decoded and then discarded by a sampling stage.
- Check how many decode sessions run in parallel. A single decoding session cannot exceed the performance of one NVDEC engine, however many engines the GPU has.
What VC-6 and NVDEC change about per-GPU throughput
SMPTE VC-6 (ST 2117) is built around repeated s-tree structures rather than block-based DCT or wavelet transforms. It can carry up to 255 data planes, each compressed as a hierarchy of resolutions with the residuals between them encoded separately. Because of that hierarchy, a decoder can extract a specific resolution, region of interest or colour plane without processing the whole image. If a lower resolution is all the model needs, the higher echelons are never decoded, which saves time and memory.
NVIDIA reports that the CUDA implementation of VC-6 decodes single images up to 13× faster than the CPU version, and 1.2× to 1.6× faster than the OpenCL GPU version, on an RTX PRO 6000 Blackwell Server Edition. On the DIV2K dataset, fetching only the bytes needed for a target quality level cut data transfer by about 37% at medium resolution and 72% at low resolution. A follow-up NVIDIA post on batch mode reports about 36% lower per-image decode time at batch size 1, 70–80% lower at batch sizes 16–32 for the lower quality levels, and up to about 85% lower at batch size 256. In that mode, roughly 4K decode drops below 1 ms per image at large batch sizes, with identical output quality.
Three caveats matter for a video feed. First, NVIDIA’s own profiling shows the latency-optimised single-image path under-uses the GPU, and the gains come in throughput mode. Second, the implementation was described as an alpha version. Third, the published figures are for image decodes, so we would confirm them on your own content and batch shape before planning capacity.
The fixed-function route has improved too. NVIDIA states that on Blackwell GPUs, NVDEC doubles H.264 decoding throughput and adds 4:2:2 decoding for H.264 and HEVC. Some GPUs carry two or three NVDEC engines, and the driver balances load across them. The gain only appears when several decode sessions run at once, though. A pipeline that pushes one high-resolution stream through one session leaves that extra capacity unused.
Why do H.264 and H.265 defaults add latency to a control loop?
Encoder features that improve quality per bit, such as lookahead, work by queuing frames. A practitioner guide from Itechguides describes H.264 delay as the sum of several queues and processing stages rather than one setting. The main sources it lists:
- B-frames refer to future pictures, so both encoder and decoder have to reorder frames.
- Lookahead deliberately holds future frames back to improve rate control and scene-change decisions.
- A large VBV buffer lets compressed data pile up before transmission.
NVIDIA’s NVENC guide confirms the lookahead behaviour on the GPU path. With lookahead enabled, the encoder returns NV_ENC_ERR_NEED_MORE_INPUT until it has buffered enough frames.
There is a second penalty outside the codec settings. The GPU path pays off when capture, decode, processing, encode and rendering all stay on the GPU, and repeated GPU-to-CPU copies can wipe out its latency advantage. NVENC can leave its bitstream output in video memory to avoid unnecessary PCIe transfers when CUDA works on it next. NVENC also exposes four tuning modes (high quality, low latency, ultra-low latency and lossless), each with presets P1 to P7 trading performance against quality. Choosing among them is a design decision. Inheriting one by default is a decision too, just an unexamined one. Even -tune zerolatency in libx264 only reduces encoder-side delay. It does not control the camera, transport, player or display.
These milliseconds matter because closed-loop lane keeping fails sharply, not gradually. A study of vision-based teleoperation streamed RTP/H.264 over UDP. Over WiFi 5 it measured a baseline median video latency of about 62 ms (p95 about 86 ms). Closed-loop lane keeping stayed stable at 75 ms of one-way delay, became clearly unstable by 150 ms and collapsed widely by 225 ms. In a study of teleoperation over commercial 5G, with a 100 ms end-to-end target, H.265 over RTSP left only 0.487% of frames over that deadline. WebRTC cut median per-frame total delay by 30.47% against RTSP. The transport protocol mattered there, not only the codec.
The OASIS work (2026) argues that H.264 suits real-time vision poorly because of its latency, its energy cost and its fixed compression settings. OASIS reports a 2.5× cut in link latency and 6× compression, with at most 1.5% task degradation. It gets there with a custom image sensor that does analogue compression, though, so it is not a drop-in codec swap. It also makes a useful general point: updating compression every frame only works if the whole loop finishes inside one frame time, 33.3 ms at 30 fps. Measuring that budget on your own hardware is covered in benchmarking GPUs against your own workload.
When GPU transcoding beats CPU transcoding on an ingest path
Transcoding time is decode time plus re-encode time. NVIDIA’s NVENC and NVDEC are hardware engines separate from the CUDA cores, so offloaded decoding frees the graphics engine and CPU. A 2018 study of cloud transcoding found that GPU VMs generally gave better execution times than other VM types. With few frames per GOP, however, the GPU performed about the same as cheaper VMs, and the gap only widened as GOPs grew.
A high-volume CDN case shows where the real ceiling can sit. A CDN’s queue kept growing while CPU load stayed around 40%, because the transcoding card’s decoder could not sustain 4K60 H.264 decode. With six NETINT T2 VPUs, each server produced 270 streams. The server count fell from 500 (CPU-only) to 38, and cost per stream dropped to $141, less than half the cost of the NVIDIA T4 system. Those units are dedicated video processing units. The lesson that carries over is that codec hardware capacity set the limit, and it only showed up under measurement.
| Situation on the ingest path | Lean towards | Reason |
|---|---|---|
| Many concurrent live streams and ABR ladders | Dedicated codec hardware such as a VPU | One CDN example: servers fell from 500 (CPU-only) to 38 |
| Short GOPs with few frames each | Cost-efficient CPU instances | The GPU advantage is negligible for small GOPs |
| Decoded frames go straight into GPU inference | NVDEC with frames kept in device memory | Keeping decode, compute and encode on the GPU reduces PCIe transfers |
| Format NVDEC does not support | A device-side decoder such as nvJPEG 2000 | NVIDIA recommends running the decoder on the device when NVDEC cannot |
| Encoder parameters need fine control | NVDECODE/NVENCODE APIs rather than FFmpeg | FFmpeg may not expose every encoder parameter |
Learned codecs against GPU-native codecs as an inference feed
Learned (neural) video codecs now compress better than the standards, at least on some benchmarks. A 2025 survey reports that learned codecs beat the H.266/VVC reference software on specific test datasets. DCVC-RT (2025) saves an average 21% in bitrate compared with H.266/VTM, and encodes and decodes 1080p at 125.2 and 112.8 fps on an NVIDIA A100. On an RTX 2080 Ti, the same model manages 40 fps encode and 34 fps decode.
Bitrate is not the whole story when you are feeding a model. A fully neural codec needs no special-purpose hardware beyond a general AI accelerator. In a pipeline, that accelerator is the same GPU your model is waiting for. The DCVC-RT authors also found that operational costs such as memory I/O and function calls, more than raw computation, are the main obstacle to speed.
Determinism adds another cost. Floating-point nondeterminism across devices can produce visible artefacts after around 30 frames, so DCVC-RT uses 16-bit integerisation. On the A100, however, fp16 runs more than four times faster than int16. The MPAI-EEV standardisation work (2023) reports a 15.05% average BD-rate saving over HEVC for EEV-0.3. Its EEV-0.4 model took about 185 seconds per 100 frames at 1920×1024, with most of that time spent moving data between CPU and GPU. The survey also notes that learned codecs suffer severe error propagation in low-delay inter-predictive coding.
| Aspect | NVDEC | VC-6 on CUDA | Learned codec |
|---|---|---|---|
| Where decoding runs | Fixed-function engine separate from CUDA cores | CUDA kernels, with bitstream parsing on the CPU | Neural network on a general AI accelerator |
| Selective decode for the model | Full frames only; CoVA had to modify a software codec to get partial decoding | Resolution, region of interest or plane | Not reported in these sources |
| Compression relative to standards | Not reported in these sources | Not reported in these sources | About 21% bitrate saving against H.266/VTM |
| Competes with the model for GPU compute | No | Yes, on CUDA | Yes |
| Maturity | Broad codec support across GPU generations | Alpha implementation at publication | Research models; determinism needs integerisation |
Our judgement: for feeding inference today, hardware decode is the safer route, because it leaves GPU compute free for the model. A learned codec earns its place where bandwidth or storage is the binding constraint and there is GPU headroom to spare.
Bitrate, quality and cost per stream without retraining the model
The balance can be tested at the compression layer, but the score has to come from the downstream task, not from picture quality. In the farm-video study, MJPEG reached over 3,000 FPS at 480p, yet its intra-frame artefacts in low-light or occluded regions slightly hurt detection. Dropping H.264 from 1080p to 720p kept accuracy and more than halved decoding latency, so the authors chose H.264 at 720p.
Lower bitrate cost accuracy in the 5G teleoperation study. H.265 cut the bitrate from 87.97 Mbps to 4.946 Mbps, but with a significant penalty in downstream object detection and recognition. It degraded detection far more than H.264 did. The effect is not always negative: in OASIS, mild compression removed high-frequency noise and improved task performance. The bitrate that best serves a detector cannot be read off a viewing-quality curve.
Device class enters through the same mechanisms: resolution per rung, codec per decoder generation, and how many sessions each GPU can carry. NVENC can change bitrate, frame rate and resolution within a live session without tearing it down and recreating it. That avoids the latency of tearing down and re-creating the encode session. We would not trust a generic bitrate ladder for an inference feed. We would score each rung on detection or tracking metrics for each device class before fixing cost per stream.
What a compression-layer audit measures and hands back
A compression-layer audit answers one question before any model work begins: which stage limits the pipeline, and by how much. It measures:
- per-plugin timings under Nsight Systems
- the number of parallel decode sessions against NVDEC engines
- glass-to-glass latency measured under real operating conditions
NVIDIA’s DeepStream profiling skill derives the configuration from two measured numbers, the inference plateau batch and the hardware ceiling.
A broadcast engineering lead should get back the data-path bottleneck isolated before any model work is considered. An encoding change that costs detection accuracy is not a free win. Itechguides makes the same point about latency figures: “H.264 adds 10 ms” means nothing without resolution, preset, GOP structure, VBV, transport, decoder and measurement method.
Who can audit the compression layer of our AI video pipeline?
TechnoLynx runs a GPU Performance Audit narrowed to the compression layer, isolating the data-path bottleneck before any model work is considered. The work covers profiling the data path on the target hardware. It also covers the fix: kernel, memory and parallelism work that makes the pipeline faster on the target hardware. Our walkthrough of how codec choice becomes the bottleneck in AI video pipelines shows the reasoning in a pipeline-first framing.
The audit is the wrong engagement if profiling already shows the model is the binding stage, or if the problem is accuracy rather than throughput or latency. A transport or CDN problem may be better served by the streaming vendor, as the WebRTC result suggests.
Designing the codec for whoever consumes the frames
The deciding question is not which codec is best. It is who consumes each stream: a viewer, a model or a controller. A frame that feeds a detector, or a teleoperation loop that falls apart somewhere between 150 and 225 ms, needs a codec, GOP structure and buffering chosen for that consumer. Defaults chosen for a living-room screen will not do. Profile which consumer your current settings serve before spending on models or GPUs.
FAQ
Can you provide some examples of video codecs?
NVIDIA’s NVDEC decodes MPEG-2, VC-1, H.264, H.265 (HEVC), VP8, VP9 and AV1. SMPTE VC-6 is a newer standard designed to work directly with GPUs.
Does H.265 have better quality than H.264?
At the same bitrate, H.265 usually gives better picture quality than H.264, which is why it is used to cut bandwidth. For AI inference that saving can be costly: one 5G teleoperation study found H.265 degraded object detection far more than H.264.
Why does AI rely on GPU and not CPU?
Neural network inference is made of large numbers of independent arithmetic operations, and GPUs execute those in parallel far better than CPUs do. The same applies to transcoding, where one cloud study found GPU VMs generally faster than other VM types.
