A 2D convolutional neural network is not an accuracy engine you can grow at will. In a robot, it is a latency-budgeted subsystem: the feature extractor whose per-frame cost has to fit inside one control cycle, or the whole perception stack misses its deadline. Stack more layers and mAP goes up on your validation set — but if the deepest backbone can’t finish a frame before the next control tick arrives, you have shipped an accuracy demo, not working autonomy. That is the reframe worth internalising before touching any of the mechanics below. The convolution is a way of extracting visual features cheaply and with strong spatial priors. Everything that follows — kernel size, stride, channel width, input resolution — is a knob on a compute budget, not a free dial on accuracy. Experts pick those knobs against the compute envelope first, then recover accuracy inside what’s left. How does a 2D convolutional neural network work in practice? At the core is one operation repeated over the whole image: a small weight matrix — the kernel — slides across the input, and at each position it computes a weighted sum of the pixels it covers. That output becomes one value in a feature map. A 3×3 kernel applied across a full image produces a new map where each cell summarises a 3×3 neighbourhood of the layer below. Two properties make this cheap and effective. First, weight sharing: the same kernel is reused at every spatial position, so a layer learns a feature detector (an edge, a texture, a corner) once and applies it everywhere. Second, locality: each output depends only on a small patch of input, which matches how visual structure actually behaves — nearby pixels are correlated, distant ones usually aren’t. A single convolutional layer learns many kernels in parallel, each producing its own feature map. Stack the maps along a channel axis and you get the layer’s output: a tensor of shape (height × width × channels). The next layer convolves over that tensor, building features of features — edges become contours, contours become object parts, parts become objects. In a framework like PyTorch this is a Conv2d layer; the same operation lowers to cuDNN or a fused kernel through TensorRT at inference time. The mechanics don’t change; only the cost accounting does. What are kernels, feature maps, stride, and pooling, and how do they shape the receptive field? Four terms carry most of the weight. Getting them straight is what lets you reason about cost instead of guessing. Kernel — the learned weight window (3×3 and 5×5 are the common sizes). Bigger kernels see more context per step but cost more multiply-accumulates. Feature map — the output grid produced by convolving one kernel across the input. One kernel, one map; N kernels, N channels. Stride — how far the kernel jumps between positions. Stride 1 keeps resolution; stride 2 halves each spatial dimension and quarters the number of output positions, which is the single fastest way to cut cost. Pooling — a fixed (unlearned) downsampling, usually max or average over a 2×2 window, that shrinks the map without adding parameters. These interact to set the receptive field — the region of the original image that a given deep-layer neuron actually “sees”. Each convolution widens the receptive field by roughly the kernel size; each stride-2 step or pooling operation multiplies how much image ground a single step now covers. A robot that needs to reason about a whole pedestrian, not just an edge, needs a receptive field large enough to contain one at working distance — and that requirement, not raw depth, is often what forces a deeper backbone. The reason this matters operationally: receptive field and compute cost are coupled. You can widen the receptive field cheaply with stride and pooling (downsample early, convolve on smaller maps) or expensively with more layers at full resolution. Atrous, or dilated, convolution is a third route — it widens the receptive field without downsampling and without extra parameters, which is why it shows up so often in segmentation backbones under a latency constraint. Why is a 2D CNN the default feature extractor rather than a fully connected network? A fully connected layer connects every input pixel to every output unit. For a modest 224×224×3 image that is over 150,000 inputs; a single dense layer mapping to even a few thousand units carries hundreds of millions of parameters. That model has no built-in notion that a cat in the top-left corner is the same object as a cat in the bottom-right — it would have to learn the pattern separately at every location, from data. The convolution bakes in two priors the dense layer lacks: translation equivariance (a feature detected in one place is detected identically elsewhere, because the kernel is shared) and locality (structure is built from local neighbourhoods). Those priors are correct for natural images, so a CNN learns useful features from far less data and with orders of magnitude fewer parameters. That parameter economy is exactly what makes the model small enough — and fast enough — to run on an embedded GPU inside a robot’s control loop. This is claim territory worth stating plainly: for image-based perception, the 2D CNN’s advantage over a dense network is not primarily accuracy, it is the compute-and-data efficiency that comes from encoding spatial structure into the architecture. That efficiency is what buys you a latency budget at all. The same reasoning underpins why detection and segmentation stacks are built on shared convolutional backbones — a point we develop for the vision stack broadly on the computer vision practice page. How do backbone depth, channel width, and input resolution trade accuracy against latency? Three axes dominate the cost of a 2D CNN, and they scale differently. The naive team turns exactly one dial — depth — because that is what the papers report. The dials that actually move latency hardest are often the other two. The dominant cost term of a convolutional layer is proportional to output-height × output-width × input-channels × output-channels × kernel-height × kernel-width. Read that formula and the trade-offs fall out: Axis Effect on accuracy Effect on per-frame latency Notes Backbone depth (more layers) Diminishing returns after a point Roughly linear in layer count Deeper widens receptive field; watch memory too Channel width Strong, but saturates Quadratic — cost scales with in-channels × out-channels The most expensive dial to grow blindly Input resolution Strong, especially for small objects Quadratic in each spatial dimension Halving resolution ≈ quarters the conv cost Stride / early downsampling Mild loss if done too early Large saving Cheapest way to recover budget The practical implication: input resolution and channel width are quadratic levers, so they dominate the latency bill. A team chasing small-distant-object accuracy by doubling input resolution is buying a roughly 4× convolution cost, which is usually where the control-cycle deadline breaks. This is an observed-pattern across the robotics perception scoping work we do — the blown-deadline moment almost always traces back to a resolution or width decision made for accuracy reasons without a cost model in front of it. It is not a benchmarked constant; the exact multiplier depends on your backbone and runtime. The disciplined move is to fix the compute envelope first — how many milliseconds the CNN gets — then choose resolution and width to fill it, and only then add depth if the budget survives. Recovering accuracy inside a fixed budget (better augmentation, a stronger head, quantisation to INT8 via TensorRT) is a different and more productive exercise than growing the backbone until the deadline breaks. How do you estimate whether a 2D CNN will fit inside a control-cycle budget? You can get a defensible estimate before writing training code. The point is to catch a blown deadline on paper, not at integration. Worked example — assumptions stated explicitly: Control loop runs at 30 Hz → one cycle is ~33 ms. Perception must return inside that, and it shares the cycle with planning and actuation, so a realistic CNN budget might be ~15–20 ms, not the full 33. The candidate backbone reports, say, ~4 GFLOPs per frame at your chosen input resolution (read this off the model’s published profile or measure it with a profiler). Your embedded accelerator delivers some fraction of its peak throughput in practice — call it a sustained rate, well below the datasheet peak, because memory bandwidth and kernel launch overhead eat into it. Divide the per-frame FLOPs by the sustained (not peak) throughput to get a first-order latency figure, then treat that as a floor and profile the real thing. The single most common estimation error is dividing by datasheet peak throughput — the spec number that never survives contact with a real workload. What matters is measured, sustained per-frame latency (ms) under realistic input, which is the theme we develop in real-time object detection and what throughput really costs. A quick estimation checklist: Fix the control-cycle period and subtract everything that isn’t the CNN. That remainder is the budget. Get per-frame FLOPs at your target input resolution, not at the paper’s resolution. Divide by sustained accelerator throughput, not datasheet peak. Add overhead you can’t skip: preprocessing, memory transfers over PCIe, NMS on the detection head. Profile the candidate end-to-end (torch.compile / TensorRT engine, the real image size) and compare to the estimate. If the measured number blows the budget, change resolution or width before you change anything else. Because sustained throughput is where estimates go wrong, this is exactly the boundary where our GPU real-time inference and latency-budget work sits — the cost model here only makes sense against how an accelerator actually behaves under load, which is a distinct discipline from the CNN mechanics themselves. Where does the 2D CNN sit in a robotics perception stack? The CNN is the shared front end, not the whole system. A typical image-based stack runs the backbone once per frame to produce feature maps, then branches into task-specific heads that reuse those features: a detection head emits bounding boxes, a segmentation head emits per-pixel masks, a pose head emits keypoints. Sharing the backbone across heads is a deliberate cost decision — you pay for feature extraction once and amortise it across every downstream task. Downstream, detections feed tracking and sensor fusion, and — where 3D geometry matters — the 2D features often feed lifting stages that reason about depth and space. If your robot needs metric 3D understanding rather than image-plane detection, the backbone choice interacts with the geometry stage; we walk that boundary in 3D object detection in practice and where it feeds tracking. The backbone’s latency budget has to account for every head it feeds, because they all wait on it. FAQ What’s worth understanding about a 2D convolutional neural network first? A small learned kernel slides across the image computing weighted sums, producing feature maps; stacked layers turn edges into parts into objects. In practice it means the CNN is a feature extractor with a fixed per-frame compute cost — in a robot, that cost is what must fit inside the control cycle, so every architecture choice is a budget decision, not a free accuracy dial. What are kernels, feature maps, stride, and pooling, and how do they shape the receptive field? A kernel is the learned weight window; a feature map is the grid it produces; stride is how far it jumps (stride 2 quarters the output positions); pooling is fixed downsampling. Together they set the receptive field — the region of the original image a deep neuron sees. You can widen it cheaply via stride/pooling/dilation or expensively via more full-resolution layers. Why is a 2D CNN the default feature extractor for robotics perception rather than a fully connected network? A dense network connects every pixel to every unit, giving hundreds of millions of parameters and no notion that an object is the same wherever it appears. The convolution bakes in translation equivariance and locality — priors that are correct for images — so it learns from far less data with far fewer parameters. That efficiency is what makes the model small and fast enough to run inside a robot’s control loop. How do backbone depth, channel width, and input resolution trade accuracy against per-frame latency? Depth scales cost roughly linearly with diminishing accuracy returns; channel width and input resolution both scale cost quadratically. The quadratic levers dominate the latency bill, so doubling resolution for small-object accuracy buys roughly a 4× convolution cost — usually where the deadline breaks. Fix the compute envelope first, fill it with resolution and width, add depth only if the budget survives. How do you estimate whether a given 2D CNN will fit inside a robotics control-cycle latency budget? Fix the cycle period, subtract non-CNN work to get the CNN budget, read per-frame FLOPs at your target resolution, and divide by sustained (not datasheet peak) accelerator throughput. Add preprocessing, memory transfer, and head/NMS overhead, then profile the real engine end-to-end. The common error is dividing by peak throughput, which never survives a real workload. Where does a 2D CNN sit in a robotics perception stack relative to detection, segmentation, and pose-estimation heads? The CNN is the shared front end: it runs once per frame to produce feature maps, which task-specific heads reuse — detection for boxes, segmentation for masks, pose for keypoints. Downstream, detections feed tracking, sensor fusion, and 3D lifting. The backbone’s latency budget must cover every head that waits on it. The knob you reach for first tells you whether a team has a cost model. If the answer to “the CNN is 8 ms over budget” is “cut a few layers,” the reasoning is still stuck on depth. The productive question — the one that scopes a robotics perception subsystem correctly — is which precision, resolution, and channel width let the feature extractor land inside one control cycle while holding the detection accuracy the task actually requires.
