A YOLO detector that clocks a fast per-image forward pass in a notebook and then delivers a fraction of the expected frames-per-second in production is almost never limited by the model kernels. The GPU is idle most of the time. It sits waiting for one image to be decoded, resized, and normalised on the CPU, runs a forward pass on a batch of one, then stalls again while non-maximum suppression (NMS) filters boxes back on the host. The convolutions are fast. The pipeline around them starves the device. This is the trap in “slow YOLO inference.” The instinct is to reach for kernel-level tuning — better occupancy, a faster NMS implementation, lower precision. Those help at the margin. But when a pipeline feeds the GPU one frame at a time, no kernel tuning recovers the throughput lost to an under-fed device. The largest gains live in the shape of the pipeline, not the shape of the kernels. How does YOLO inference work in practice? A YOLO (You Only Look Once) model is a single-stage object detector: one forward pass produces a dense grid of candidate boxes with class scores, and a post-processing step prunes them down to final detections. That “single-stage” property is exactly why people underestimate the pipeline. Because the model itself is one pass, teams assume inference is one thing. It is three. Every YOLO inference request moves through three distinct stages, and each runs on different hardware unless you force otherwise: Pre-processing — decode the image (often JPEG), resize to the model’s input resolution (commonly 640×640), convert layout (HWC to CHW), normalise to float, and move the tensor to the GPU. By default much of this runs on the CPU with libraries like OpenCV or Pillow. Model forward pass — the convolutional backbone and detection head run on the GPU. This is the part everyone profiles, and it is usually the fastest of the three per image once the data has arrived. Post-processing (NMS) — the model emits thousands of raw boxes per image; NMS removes overlapping duplicates and applies a confidence threshold. In many stock implementations this happens on the CPU after copying results back from the device. The forward pass is GPU work. Pre-processing and NMS, in a naive setup, are CPU work with a device round-trip on each side. That structure means the GPU is genuinely busy for only a slice of wall-clock time per request — which is why single-image serving so often shows single-digit or low-double-digit GPU utilisation while the CPU pegs a core. Why does batching change YOLO throughput more than tuning kernels? A modern data-centre GPU is a wide parallel machine — thousands of CUDA cores fed by high-bandwidth memory. A single 640×640 image does not contain enough parallel work to saturate it. The convolution kernels launch, finish quickly, and the scheduler has nothing else queued. You are paying for a wide machine and running it narrow. Batching fills that width. When you assemble, say, 16 or 32 frames into one tensor and run a single forward pass, the same kernels now operate on far more data per launch. Kernel-launch overhead amortises across the batch, memory access patterns become more regular, and occupancy rises. In configurations we have worked through, moving a stock YOLO pipeline from batch-of-one to dynamic batching lifts GPU utilisation from a single-digit or low-double-digit percentage to well above 70%, which translates into several-fold throughput gains at a fixed batch latency budget (observed across TechnoLynx GPU pipeline engagements; not a published benchmark). The reason kernel tuning cannot match this is arithmetic. If the GPU is idle 85% of the time because it is starved, making the busy 15% run 20% faster buys you a 3% end-to-end improvement. Feeding it properly turns the 85% idle into useful work. This is the practical face of the claim that the unit of performance is the workload feeding the device, not the device in isolation — the same principle that separates peak-spec throughput from what you actually measure in a serving loop. The catch is latency. Batching trades per-request latency for throughput: a frame that arrives just after a batch closes waits for the next one to fill. Dynamic batching — where the server collects requests up to either a max batch size or a max wait window, whichever comes first — is how production servers like NVIDIA Triton Inference Server resolve this. You cap the wait (say, a few milliseconds) so tail latency stays bounded while still amortising kernel launches across whatever arrived in that window. Where do the bottlenecks actually sit? Once you accept that inference is three stages on two devices, the bottleneck question becomes concrete. Here is where the time goes in a typical unoptimised versus reworked YOLO serving pipeline, and what moves the needle in each stage. YOLO pipeline stage diagnostic Stage Naive setup Where it stalls The rework Pre-processing CPU decode/resize/normalise, then H2D copy CPU-bound; blocks the GPU per frame On-GPU decode (NVIDIA DALI, nvJPEG) + batched resize; overlap copy with compute via CUDA streams Forward pass Batch-of-one, FP32 GPU under-occupied Dynamic batching; FP16/INT8 via TensorRT if accuracy allows Post-processing (NMS) CPU NMS after D2H copy Device round-trip + serial host work On-GPU / fused NMS (TensorRT EfficientNMS plugin, torchvision GPU ops) Orchestration Synchronous, one request at a time No overlap between stages Pipeline overlap so stage N+1 of batch k runs while stage N of batch k+1 is in flight The pattern is consistent: the biggest single win is usually eliminating the device round-trips and the batch-of-one, not optimising any individual kernel. When pre-processing runs on the GPU with a library like NVIDIA DALI and NMS runs on-device via a TensorRT plugin, the data stops bouncing across PCIe on every request, and the CPU stops being the pacing item. How do batch size and GPU occupancy interact for real-time detection? Occupancy — how many of the GPU’s execution slots are active — climbs with batch size until the model saturates the device, then plateaus. Past that plateau, larger batches add latency without adding throughput because you are already memory- or compute-bound. The useful batch size is the smallest one that reaches the plateau, because anything larger just inflates latency. Finding it is empirical, not theoretical. A worked example makes the trade-off concrete: Illustrative sizing exercise. Suppose a YOLO model runs a forward pass in roughly 4 ms at batch-of-one on a given GPU, and 12 ms at batch-of-32. Per-image cost has dropped from 4 ms to under 0.4 ms — roughly a 10× throughput gain — while the per-batch latency rose only 3×. If your latency budget is 20 ms, batch-32 fits comfortably. If the model reaches its occupancy plateau at batch-16 (say, 9 ms), batch-32 buys little extra throughput and costs latency, so batch-16 is the better operating point. These are illustrative numbers; the actual curve must be measured on your executor. Real-time workloads (live video, on-camera detection) add a wrinkle: frames arrive at a fixed rate, so you cannot always wait to fill a large batch. Here the dynamic-batching wait window is the lever — it bounds how long a frame waits before dispatch. Streaming detection also benefits from where the model runs; for constrained or edge targets, a lighter model or a more efficient segmentation backbone like MobileSAM on constrained GPUs changes the batching maths entirely because the per-image cost is lower to begin with. When should NMS and pre-processing run on the GPU? The default answer for a throughput-bound serving workload is: both should run on the GPU. The device round-trip — copying decoded images up and detections back down over PCIe on every request — is pure overhead that dynamic batching alone does not remove. Keeping the whole pipeline resident on the device removes it. The exceptions are real, though: Pre-processing stays on CPU when the CPU is idle anyway (batch offline jobs where the GPU is the scarce resource and CPU decode overlaps cleanly), or when input decoding needs codecs that lack a GPU path. NMS stays on CPU when detection counts per image are tiny and the host NMS is genuinely negligible, or when a custom NMS variant has no GPU implementation and the port cost outweighs the gain. Both move to GPU for any sustained real-time or high-QPS serving workload, which is the common case that motivates the question in the first place. For decode specifically, on-GPU JPEG decoding via nvJPEG and batched augmentation via NVIDIA DALI keep the tensor on the device from the start. For NMS, TensorRT’s fused NMS plugin runs suppression as part of the engine, so the model output never leaves the GPU before the final boxes are ready. How do I tell whether my pipeline is limited by structure or by kernels? This is the diagnostic that should precede any optimisation work, because it determines whether you touch the pipeline or the kernels. Run this before writing a single CUDA kernel. Structure-vs-kernel diagnostic checklist Check GPU utilisation under load. If it sits below ~40% while requests queue, the limit is structural (starvation), not kernel efficiency. This is the single most informative reading, and it connects directly to how you read GPU utilisation across the three pillars of observability. Check the batch size in production. If it is effectively one, you have not started the real optimisation yet. Profile the wall-clock split across the three stages. If pre-processing plus NMS dwarf the forward pass, the bottleneck is the pipeline, not the model. Watch a CPU core. A pegged single core during inference is the classic signature of CPU-bound pre/post-processing. Measure PCIe traffic. Heavy per-request host-device transfer means you are paying the round-trip tax that on-GPU pre/post-processing removes. Only if utilisation is already high and the forward pass dominates does kernel-level work — occupancy tuning, kernel fusion, precision changes — become the right lever. The metrics worth tracking to know an optimisation paid off are narrow and operational: frames-per-second per GPU, p99 inference latency, and cost per million detections. These are exactly the levers a GPU Performance Audit quantifies before recommending any kernel-level work, and they are the same throughput-and-tail-latency framing that separates ML metrics that explain GPU bottlenecks from metrics that mislead. A rise in mean throughput that comes with a p99 latency blowout is not a win; the metrics have to be read together. FAQ How should you think about yolo inference in practice? A YOLO model is a single-stage detector: one forward pass produces a dense grid of candidate boxes, and post-processing prunes them to final detections. In practice, inference is three stages — pre-processing (decode, resize, normalise), the GPU forward pass, and NMS post-processing — that by default run across CPU and GPU with a device round-trip on each side. The forward pass is fast; the surrounding pipeline is usually what limits throughput. Why does batching change YOLO throughput more than tuning individual kernels? A single 640×640 image does not contain enough parallel work to saturate a wide GPU, so the device sits idle most of the time. Batching fills that width, amortises kernel-launch overhead, and raises occupancy. If the GPU is idle 85% of the time from starvation, speeding up the busy 15% barely helps end to end, whereas feeding it properly turns idle time into useful work — which is why batching dominates kernel tuning. What are the main stages of a YOLO inference pipeline and where do the bottlenecks sit? The three stages are pre-processing, the model forward pass, and NMS post-processing. In a naive setup pre-processing and NMS run on the CPU with a device round-trip each, so the bottleneck usually sits there rather than in the model kernels. The biggest wins come from eliminating the round-trips and the batch-of-one, not from optimising individual kernels. How do batch size and GPU occupancy interact for real-time detection workloads? Occupancy rises with batch size until the model saturates the GPU, then plateaus; past the plateau, larger batches add latency without adding throughput. The useful batch size is the smallest one that reaches the plateau. For real-time streams, a dynamic-batching wait window bounds how long a frame waits before dispatch, keeping tail latency in check while still amortising kernel launches. When should NMS and pre-processing run on the GPU instead of the CPU? For any sustained real-time or high-QPS serving workload, both should run on the GPU to remove the per-request PCIe round-trip. Pre-processing can stay on CPU for offline batch jobs where the CPU is idle and decode overlaps cleanly, and NMS can stay on CPU when detection counts are tiny or no GPU implementation exists. The common serving case that motivates the question almost always wants both on-device. How do I tell whether my YOLO pipeline is limited by algorithmic structure or by kernel-level inefficiency? Check GPU utilisation under load first: below roughly 40% while requests queue means the limit is structural starvation, not kernel efficiency. Corroborate by checking the production batch size, profiling the wall-clock split across the three stages, watching for a pegged CPU core, and measuring PCIe traffic. Only when utilisation is already high and the forward pass dominates does kernel-level work become the right lever. What throughput and latency metrics should I track to know a YOLO inference optimisation actually paid off? Track frames-per-second per GPU, p99 inference latency, and cost per million detections. Read them together — a rise in mean throughput that comes with a p99 latency blowout is not a win. These are the same operational levers a GPU Performance Audit quantifies before recommending any kernel-level work. The question worth asking first Before you optimise a YOLO deployment, the useful question is not “which kernel is slow” but “what is actually feeding the GPU, and how much of the time is it genuinely busy.” Object detection sits inside a larger computer vision deployment stack, where detection accuracy and serving throughput have to hold together at once — and the accuracy target you commit to constrains whether you can drop to FP16 or INT8 to buy more headroom. Classify the bottleneck as algorithmic (batching, layout, pipeline overlap) or micro-level before you touch a kernel; that classification is exactly what a GPU Performance Audit produces, and getting it wrong is how teams spend weeks tuning the fast part of a starved pipeline.
