In computer vision and robotics, perception was the dominant source of system failures in one robot study cited here. Planning and control can only act on what the vision stack reports, and only once the report arrives. One study, whose 105 trials mixed hardware and simulation scenarios, shows the pattern clearly: an autonomous doorway-traversal and payload-delivery robot completed 97 of them, and its failures were concentrated in perception rather than in planning or manipulation. A programme that treats computer vision as a latency-budgeted subsystem, with a deadline it has to meet on every control cycle, is building towards working autonomy. A programme that tunes accuracy without that deadline is building a demo.

How tight is the latency budget for each CV component in a typical robotics control loop, and how do you measure whether perception is meeting its deadline versus blowing it?

The budget is tight, and it is set by several rates at once. A 30 Hz control loop gives 33.3 milliseconds per tick, and inside that window the robot has to read its joints, pull a camera frame, turn both into an action and put that action on the servo bus. In a model-predictive manipulation system published on arXiv, the 6D pose localiser runs at 5 Hz, the camera and tracker at 30 Hz, the optimal-control solver at 100 Hz and the robot controller at 1 kHz.

The table below gathers figures reported for the cited systems. Read each row as what that one setup achieved, not as a general benchmark.

Component	Rate in the cited system	Latency or period	Source system
RGB camera stream	30 Hz at 640×480	33.3 ms between frames	Panda torque-control rig
6D pose localiser	5 Hz	200 ms period (1 / 5 Hz)	Same rig
Frame-to-frame pose tracker	30 Hz	about 5 ms per estimate	Same rig
Optimal-control solver	100 Hz	10 ms period	Same rig
Torque controller	1 kHz	1 ms period	Same rig
Stereo disparity graph, 1080p	up to 99.4 fps	16 ms and 28 ms on the two listed platforms	ros2_benchmark results
YOLOv11n detection offloaded to Jetson Orin NX	10 Hz behaviour-tree tick	about 200 ms end to end	Doorway-traversal robot
U-Net segmentation on GPU	30 Hz target	10 ms per frame	Surgical segmentation study
SAM2 segmentation	30 Hz target	about 600 ms per frame	Same study
Learned policy, one action step	30 Hz tick	20 ms (ACT) to 485 ms (Pi0.5)	AY-Robots platform catalogue

Two things in that table matter more than any single row. First, a fast component can sit beside a slow one when the architecture keeps the slow one off the critical path. The 200 ms localiser coexists with a 1 ms control period because a 5 ms tracker answers every frame. Second, a figure that looks fatal at one loop rate can be acceptable at another. The 200 ms detection latency on the doorway robot costs two missed ticks of its 10 Hz behaviour tree in the worst case, which the authors judge acceptable for human walking speeds.

For the vehicle version of a perception stack, see our article on the self-driving car machine learning pipeline.

Why averages hide the deadline misses that matter

Whether perception meets its deadline has to be judged on worst case, not on median latency. A loop with a 33 ms budget is wrecked by a 400 ms p99 even when the median is a comfortable 40 ms. Promwad’s guidance on industrial control loops makes the same point from the controls side: on a 1 kHz loop, the 1% of cycles beyond p99 comes to roughly 10 cycles per second, about 600 a minute, so the pass criterion for a hard real-time path is a defensible upper bound rather than a percentile.

Some of the worst-case terms rarely appear on a network diagram. An input that arrives just after a task boundary waits nearly a full period before the controller uses it, and on a 1 ms motion task that wait alone can take close to 1 ms of budget. Sensor filtering adds phase lag at the loop’s crossover frequency, so “more filtering to reduce noise” is a decision against the stability budget. The operating system matters too: stock Linux can have a latency tail reaching into milliseconds under load, while PREEMPT_RT reduces scheduling latency by an amount that depends on the SoC, drivers, IRQ configuration and workload.

A procedure for checking perception against its deadline

1. Fix the two numbers the plant dictates: the required update period and the maximum tolerable end-to-end response time, derived from the process’s dominant time constant and the stability margin the control law needs.
2. Measure the perception graph as deployed, transport included. ros2_benchmark measures the throughput, latency and compute utilisation of ROS 2 graphs without altering the code under test, and it counts frames dropped in rclcpp transport between the playback and monitor nodes.
3. Run the latency test at the sensor’s real rate, since the harness can tie playback to the input rate (many LiDARs run at 10 Hz, cameras at 30 or 60 fps).
4. Trace inside the graph when a stage overruns. ros2_tracing adds a mean end-to-end message latency overhead of 0.0033 ms with all instrumentation enabled, and pairing its traces with LTTng kernel traces shows OS-level causes: in one example, a timer callback that ran late had been blocked for 9.9 ms waiting for CPU.
5. Break each tick down by section. A cadence report that gives mean and worst times for observe, inference and send shows where the time goes: if observe dominates, the problem is cameras or the servo bus, and if inference dominates, it is the model.
6. Add a jitter reserve on the timing side, and treat any stage whose worst case does not fit as the layer to work on.

Node-level numbers alone are misleading, and the ros2_benchmark results show why: a stereo disparity node measured at 6.4 ms becomes 16 ms once it runs as a graph.

What is the architectural pattern that connects CV outputs to robot motion and control loops?

The pattern in the cited systems is multi-rate and asynchronous. In one cited pose-tracking system, the 1 kHz controller reads an extrapolated state instead of waiting for the 5 Hz pose estimator. The 6D manipulation system cited above shows the pattern. A slow, accurate pose detector re-initialises several fast trackers running in separate processes. The optimal-control problem is solved at 100 Hz, and 1 kHz torque feedback comes from Riccati linearisation. The 1 kHz loop runs on a separate real-time-preempted computer, which talks to the perception computer over ROS. An older bin-picking system split the work the same way, with one computer for robot control and a second, GPU-equipped one for vision.

The interface between the two halves needs more than raw detections. Off-the-shelf single-view RGB pose estimators lack the temporal consistency that stable feedback control needs. A pose estimate that suddenly puts an object 10 cm away, or flips its orientation by 180 degrees because of a symmetry, can make the controller command dangerously large torques. A factor-graph filter that uses motion models and explicit measurement uncertainty stays stable when the object is partly occluded, in frames where the per-frame estimator gets the orientation wrong. Its horizon and smoothing parameters let engineers tune the precision-recall trade-off for control robustness. Our view is that the filter between detection and control deserves as much engineering attention as the detector itself, because it decides what the controller is allowed to believe.

Above the geometric loop, most current robotic systems add a hierarchy. A high-capacity task planner breaks a long-horizon goal into subtasks, and a low-level policy handles speed and precision. Reactive executives such as behaviour trees re-evaluate priorities on every tick, and that reactivity is why perception latency translates directly into missed ticks.

Why is perception the bottleneck in robotics, and what changed between 2024 and 2026?

In the cited studies, perception is the bottleneck because the downstream modules depend on what it reports, and the evidence points the same way. In the doorway-traversal study, the failures in both the assisted-traversal and post-traversal manipulation scenarios came entirely from detection errors. When detections were stable and confident, the manipulation pipeline performed reliably. The detection failures came from partial occlusion by clutter and from lighting outside the training distribution. The door-state classification also kept treating a partly opened door as closed, and the YOLO models had been trained on a controlled set of 600 images. Agricultural manipulation shows the same dependency: modular systems are only as good as the target localisation and scene representation passed between components.

The evidence for change in field perception is thin. A 2026 Patsnap patent review counts 517 industrial robot perception patents from 2017 to 2026, reports a surge in filings in 2025, and attributes to OECD research the view that perception is the main bottleneck for autonomous manufacturing. Patent counts measure investment interest, not deployed perception accuracy. The same review argues that setup complexity, not accuracy, has historically been the main barrier to deployment. At the architecture level, vision-language-action models now serve as control policies that generate low-level robot actions. Humanoid robots also moved into factory trials, BMW’s August 2024 test at Plant Spartanburg among them. None of the supplied sources measures whether field perception failure rates fell over the period, so we would not claim they did.

What changes when you move from a single fixed camera to RGB-D and multi-sensor fusion?

Single-camera RGB leaves object distance uncertain, and adding depth brings its own processing costs. For RGB-only pose estimation, pinhole projection leaves high uncertainty in the camera-to-object distance. That uncertainty is largest along the ray from the camera to the object, because large changes in depth barely change how the object looks. In a cluttered-manipulation study by Schwarz, Milan, Periyasamy and Behnke, adding depth measurements improved detection, and training a depth CNN with cross-modal distillation gave the best results of the fusion methods compared.

The same study shows where the cost lands. The team saw systematic, angle-dependent depth artefacts on shelf walls, which pushed them to a dual-sensor setup. Three depth streams then had to be projected into one frame and fused with fixed weights, and guided upsampling had to be written in CUDA to reach under 100 ms per image. Resolution is the expensive dial: the full-resolution detector took about 1 s per image, while an intermediate resolution reached 94.7% mAP at 550 ms. Accuracy scores like that one only mean something next to the latency they cost, which our explainer on mAP for YOLO detectors covers from the scoring side. Small and shiny objects such as toothbrushes and scissors were hardest for both methods, pushing semantic segmentation below an F1 of 0.5. Transparent and reflective surfaces remain hard because standard depth sensors rely on light-return behaviour those materials break.

One multi-camera mobile stack, NVIDIA’s Isaac Perceptor, runs its processing on GPU-accelerated modules. Isaac Perceptor rectifies and undistorts the camera streams on the GPU and throttles the side-camera pairs to reduce GPU load. It runs learned stereo depth estimation and visual SLAM concurrently, and feeds both into nvblox for a voxelised 3D map. Transport adds a measurable cost. Earlier work cited in the ros2_tracing paper measured ROS 2 communication overhead at up to 50% compared with using DDS directly.

Where do classical computer vision, deep learning and world models each fit?

In one surgical-segmentation study, a 30 Hz target ruled out the slower segmenters for deployment. That study, for an automated da Vinci surgical arm, measured classical, learned and foundation-model methods against one 30 Hz target. Scored against SAM2-generated masks (so SAM2 scores 100 by construction), a U-Net trained on those masks reached Dice 95.6 at 10 ms on GPU and 80 ms on CPU, CLIPSeg 81.3 at 150 ms, and classical computer vision 63.2 at 5 ms. The classical methods were fast but failed under occlusion, lighting variation and tool-background similarity. The pattern that worked was to use the foundation model as a labeller and deploy the small trained network.

Classical computer vision still has a defined role. Surveyed robotics domains that cannot rely on network connectivity for time-critical perception, such as autonomous driving and emergency response, can, as one option the survey gives, keep a safe fallback mode built on classical tools that run on local compute. Deep-learning detection and segmentation (the kind our article on 2D convolutional neural networks explained walks through) does the semantic work in the clutter-picking and doorway robots cited here. In the cluttered-manipulation study, fine-tuning pretrained networks beat a classical CNN-plus-SVM combination despite limited training data. Large foundation segmenters are not yet loop components: SAM cannot run in real time, though FastSAM and MobileSAM reach comparable performance at faster inference speeds.

Meta’s V-JEPA 2-AC, trained on under 62 hours of unlabelled robot video, ran pick-and-place zero-shot on Franka arms in two labs, planning towards given image sub-goals. It needed 16 seconds to compute each action, against 4 minutes per action for Cosmos. Its predictions reach roughly 16 seconds ahead. It is also sensitive to camera position, and the authors tried several placements by hand before settling on one. Surveys place world models next to low-level control policies, because they predict the next state given an action, and text-based reasoning at the task-planning level.

Which robotics use cases (pick-and-place, mobile robots, surgical, agricultural) push CV hardest?

The hardest cases are the ones where the target moves or deforms on contact, its appearance varies, and the loop still has to close at video rate. On that reading, surgical and agricultural manipulation sit at the top. Pick-and-place in clutter and mobile robots are hard in narrower ways.

• Surgical robotics. The da Vinci system currently operates under full teleoperation. Automating a third arm needs real-time segmentation of robot arms and manipulated objects, at or above 30 Hz, in a scene where false positives could cause incorrect manipulation.
• Agricultural harvesting. Crops vary in geometry, orientation and height, and leaves and branches occlude targets and block manipulator access. Selective harvesting robots use computer vision to find ripe fruit and guide the arm. On a physical robot, branches and fruit move when touched, so the target pose changes during the final approach.
• Pick-and-place in clutter. Objects are partially occluded among distractors in restricted spaces. In one disaster-response dataset the main difficulty was small object size rather than clutter, with some targets taking up as little as 4% of the image width.
• Mobile robots. Isaac Perceptor fuses depth images and visual SLAM poses into a voxelised map, and the doorway robot misclassified a partly open door as closed.

Where will embodied AI and LLM-planner stacks change the architecture next?

The direction is a split between a slow semantic channel and a fast geometric one, with perception feeding both. A survey of vision-language-action models describes a dual-system design in which a VLM-based System 2 processes images and instructions at 10 Hz while a diffusion-based System 1 generates closed-loop motor actions at 120 Hz. Figure AI describes its Helix model the same way: a 7B-parameter onboard VLM at 7 to 9 Hz and an 80M-parameter visuomotor policy at 200 Hz. Those figures come from the vendor’s own description of Helix.

The split does not remove the latency problem on edge hardware. On edge hardware, NVIDIA’s published GR00T N1.7 timings, as reported by AY-Robots, are 144.9 ms end to end on AGX Thor in PyTorch eager, 342.8 ms on Orin, and 93.8 ms on AGX Thor with TensorRT. All of these are usable for slow manipulation with action chunking, and none of them is 30 Hz closed-loop control. Chunking keeps the action queue full without making the actions fresher. In the SmolVLA ablation on LIBERO, average success fell from 82.8% with a new observation every 10 steps to 51.8% with one every 50. Hierarchical planners add their own cost, because frequent execution failures trigger replanning, and replanning can add significant latency.

Two constraints will shape what comes next. Cloud-hosted models make latency depend on networks and data centres outside the robot’s control, so models will need distilling onto onboard hardware, which trades away some capability for size and speed. The sim-to-real gap in images also limits how far simulation-trained policies transfer, and in a recent agricultural VLA study the real-robot scores were lower than in simulation. We expect the fast loop to stay with small, local, often classical or compact learned perception for some time, while foundation models take over the semantic side.

How do humans and AI-powered robots actually work together in production today?

Today’s collaboration is narrower and more supervised than the demo reels show. BMW described its 2024 Figure 02 test at Spartanburg as an early trial for identifying the requirements of integrating humanoid robots into an existing production system. A 2026 industry analysis by The Tech Trends, a secondary source, reports that humanoid deployments it reviewed are scoped to a specific task, zone or shift. It reports that demo videos often use teleoperation and leave out faults and charging cycles. Where people share the workspace, the same analysis describes Apptronik’s Apollo at Mercedes-Benz working inside zones bounded by light curtains and pausing when a person crosses the boundary. Collaboration levels, safety limits and cobots versus humanoid pilots get their full treatment in our article on human-robot collaboration in production.

Who can scope the perception subsystem of our robot against its control loop?

TechnoLynx builds computer vision systems for robotics, scoped against the latency contract of the control loop they feed. That covers object detection, tracking and multi-view vision, plus GPU profiling and performance engineering so that perception runs faster on the target hardware. TechnoLynx would start with a robotics computer vision subsystem scoping engagement against the loop’s latency budget, within TechnoLynx Computer Vision R&D.

That engagement has limits. Functional-safety certification and regulatory sign-off are outside the offer catalogue and stay with your safety team and assessors. A team that already has a robotics group fluent in ROS 2 tracing and real-time Linux may only need the measurement steps above.

FAQ

What is perception in robotics?

Perception is the part of the robot stack that turns raw sensor data (camera images, depth, LiDAR) into the object positions, poses and scene state that planning and control act on. Detecting affordances, meaning what an object lets the robot do with it, is the step that bridges perception and action.

What are VLA models?

Vision-language-action models are multimodal models that take visual observations and language instructions and generate robot actions. Many share a structure of modality encoders, a Transformer backbone that fuses the tokens, and an action head that generates action chunks.

What are some examples of robotic surgery?

The da Vinci Surgical System is one example, and it currently runs under full surgeon teleoperation. Research on automating one of its arms depends on real-time segmentation of instruments and objects.