We would add four things to the video feed of a teleoperation interface built for fast escalation decisions. It shows the autonomy stack’s confidence, the specific reason the vehicle asked for help (with that cause marked in the scene), who controls the vehicle right now, and how old the picture is, with a separate warning when the link degrades. Raw video on its own is not enough. Research on teleoperation GUIs at TUM (Wolf et al., 2025) treats information shown alongside the camera feed as crucial for both remote driving and remote assistance.
TechnoLynx designs and builds this kind of console through its Generative & Agentic AI R&D engagement, including the confidence-scoring and staleness-monitoring telemetry behind it. The first step is a session with your operations team to define the decision the operator has to make, how much time they have for it, and which autonomy-stack signals actually exist to show.
How long does a remote operator have to decide?
The takeover studies cited below measure drivers inside the vehicle, not remote operators deciding on an escalation. Any “five-second” figure for a remote console is therefore a design target, not a measured finding. Related measured evidence comes from takeover time for drivers inside the vehicle. A 2025 systematic review of takeover studies found mean takeover times between 0.69 s and 19.79 s, with an average of 2.72 s. The same review lists suggested time budgets of 5 to 15 seconds in the literature. It also warns that takeover times are right-skewed, so a budget built around the mean leaves slower responders without enough time. Budgets that are too tight cause over-reactions and poor takeover performance.
Two things change when the human is remote. First, the operator arrives cold: someone alerted to drop in has to build situation awareness of an unfamiliar scene from a monitor and a video feed. Second, in driverless Level 4 or 5 operation the vehicle must reach a minimal risk condition, a stopped state, before it disengages.
That changes where the time pressure falls. For a stopped vehicle, the few seconds are spent on the first read: is the escalation genuine, and what kind of help does it need? That decision should come before the operator commits attention or sends a command. For a vehicle still moving under a request to intervene, the driver-takeover figures are the better guide, and everything needed for the decision has to be readable in one glance.
How do you show the stack’s confidence and why it escalated, not just raw video?
Show the escalation reason as a named cause, mark that cause in the scene, and attach a calibrated confidence to it. A 2025 study of a command-based tele-assistance interface, tested with 14 expert teleoperators, assumes that a vehicle which cannot resolve a situation will usually still recognise why, for example an obstacle blocking the road. Each recognised reason then maps to a set of high-level commands the operator can choose from. In a 2024 expert workshop reported by Brecht et al., the failure causes judged most likely to lead to teleoperation were object detection, lane detection, trajectory planning and path planning. That list makes a sensible first set of reason codes.
Marking the cause on the video also matters. The same 2025 study notes that marking the reason for intervention (a red circle around police vehicles blocking the road, for instance) increases the operator’s situation awareness and reduces reaction time.
The three-layer transparency model from Chen et al. (2018), as summarised in a review of explainable autonomous vehicles, gives a useful structure for the reason panel. The first layer is the agent’s current status and action, the second is the reasoning behind it, and the third covers predicted outcomes, uncertainty and limitations. On a console that might read: stopped; object detection cannot classify the object blocking the lane; classification confidence low, and the planner has no valid trajectory.
Confidence is the element most likely to mislead
A 2024 study on explanation specificity and perception errors concluded that interfaces should communicate the system’s confidence levels and limitations, not only accurate information. The study also found that detailed explanations which exposed frequent perception errors lowered perceived safety. For a passenger that is a cost. For an operator deciding whether to take over, it is the point: a perception stack that is often wrong should look unreliable on the console.
A raw classifier score is not the probability of being right unless it has been calibrated against held-out data. A 2026 review of teleoperation in hazardous environments describes confidence as an authority signal and notes that miscalibrated confidence can prolong incorrect assistance when feedback is delayed.
A modular autonomy stack pays off here. We prefer modular, observable systems because when something breaks you need to know where and why. When perception, prediction and planning are separate stages, the escalation reason can name the stage that gave up, and the confidence can belong to that stage. An end-to-end model makes that attribution harder.
Control ownership has to be on screen at all times
The operator should never have to wonder who is driving. The 2025 command-based interface uses a status bar to show the current or upcoming road situation, the vehicle’s state and the current control owner. It distinguishes three control states: Vehicle, Remote Assistant and Remote Driver. Several expert operators still said the control owner status should be more visible. One suggested “Auto” instead of “Vehicle” and “Manual” instead of “Remote Driving”. In the TUM study, experts grouped control owner and mode with vehicle speed and the video stream as elements needed throughout remote assistance. That GUI also switches from a green to a blue tint when human input is required.
The failure this prevents has a name. The 2026 hazardous-teleoperation review lists mode confusion, where operators struggle to predict the system’s current level of autonomy, among the reported failure modes of shared autonomy. The same review argues that interventions should be legible at the interface, so the operator can predict what was changed, why, and when assistance will hand back control.
How should the console flag stale video or a degraded link?
Show the age of the frame as a number, flag freezes and degraded image quality separately, and set a frame-age threshold for each mode. In remote driving, the console should block or clearly warn against continuous commands once the frame age crosses that threshold.
The numbers explain why this needs to be explicit. Glass-to-glass latency, the delay from a vehicle sensor capturing an event to that event appearing on the operator’s display, is critical for real-time situation awareness and safe control. A 2025 test of teleoperated driving over a commercial 5G network in Luxembourg measured an average glass-to-glass latency of 202 ms, with peaks above 300 ms. At 30 km/h a vehicle travels about 1.7 metres in that time, and the authors judged the setup safe at low speed, below 20 km/h. In a 2026 simulator study by Trautmannsheimer, Azab and Diermeyer, the best-case pipeline alone already had a glass-to-glass latency of about 413 ms.
A 2026 study of a deterministic lane-keeping controller, not a human operator, found that stability collapsed between 150 ms and 225 ms of one-way perception latency, with route completion falling from 100% to below 50%. The authors’ broader conclusion is that perception-driven teleoperation has sharp stability limits rather than smooth decay. For humans, the 2026 review cites round-trip delays of around 300 ms as already causing measurable deterioration. A TeleSim robot-arm study (2025) also reported a nonlinear relationship between network quality and task success across three network tiers: success fell from 92% to 28% between its best and worst tiers, and the worst tier showed macro-blocking and intermittent freezes. Taken together, these results suggest the indicator should switch state at thresholds measured for your mode and speed, rather than fade smoothly from green to red.
Latency is only half of the readout. The Trautmannsheimer study found that 300 ms of added delay at 2000 kbit/s matched the best case on speed, while 300 ms at 500 kbit/s did not. The authors recommend treating latency and video quality as largely independent levers. A console that shows only a latency figure can therefore look healthy while the image is too degraded to judge. Experts in the TUM study rated latency and network-quality indicators as important. The predictive-display control station built by Prakash et al. (2022) shows observed network latency next to speed and the estimated path.
Computing frame age depends on embedded timestamps and synchronised clocks. The 2026 vision-teleoperation testbed embeds a send-time stamp in each frame and synchronises the vehicle and operator machines with Chrony, which lets it compute one-way latency per frame. That controller also includes frame staleness detection as a safety mechanism. Where latency cannot be avoided, predictive video transformations that overlay estimated future states have cut task completion time by up to 26% under high latency, according to the 2026 review of robotic teleoperation in hazardous environments.
The staleness flag can lag too
A staleness indicator is itself an estimate, and it can be late. In the Prakash et al. setup, the uplink delay for driving commands is unknown at the control station when the image is transformed, so it is estimated statistically from recent samples. They base their hold-and-apply strategy on the 95th percentile of that estimate, refitted every second. We would apply the same idea to the console: key the warning to a high percentile of recent latency rather than the last sample, and treat a clock-sync failure as a degradation in its own right. Before go-live, validate the flags against injected, known network conditions.
What to leave off the screen during a time-critical escalation
Leave out anything that does not help with the decision in front of the operator. Graf and Hussmann (2020), as cited by the TUM team, identified 80 general information requirements for teleoperation and reduced them to 20 essential elements. For remote assistance, the TUM experts rated engine RPM, mileage, range, energy consumption and level, oil level and temperature, and a compass as irrelevant at every stage of the process. The 2024 review of explainable autonomous vehicles warns that explaining every action leads to information overload, and recommends prioritising by event criticality.
Placement on screen also matters. An expert operator in the 2025 study said the video is so dominant that peripheral elements such as a warning sign can be missed. The study suggests reducing peripheral elements to the minimum and placing them over the video where appropriate. Pop-ups along the top of the screen were not visible enough to all users, and one operator suggested a prominent centred pop-up that collapses once seen. Icons need testing: eleven participants read a “Microphone” icon as “Mute”.
In the TUM study, a display that changed with the phase of the process was faster than a fixed one: guiding the vehicle through a scenario took an average of 76.7 s with a static GUI, and a dynamic GUI showing fewer elements cut that by 16%. Usability rose from 68.5 to 76.6 on the System Usability Scale. Our rule of thumb follows from this. Show only the elements each phase of the escalation needs, and let the display change as the process moves on.
Ordering the screen so an unfamiliar scene reads at a glance
Order the elements by what the operator needs to understand first. Mutzenich et al. describe a remote operator’s situation awareness as perception, comprehension and prediction. An operator dropping in cold first needs to grasp why the vehicle stopped, then whether the picture can be trusted, then what to do.
| Read order | Element | Question it answers | Remote assistance | Remote driving |
|---|---|---|---|---|
| 1 | Escalation reason, centred and over the video | Why is the vehicle asking? | Required | Required |
| 2 | Cause marked in the scene, with calibrated confidence | What exactly is uncertain, and how uncertain? | Required | Required |
| 3 | Frame age, freeze flag, link quality | Can I trust what I am seeing? | Warn | Warn and gate continuous control |
| 4 | Control owner and mode | Who is in control right now? | Always shown | Always shown |
| 5 | Camera video stream and vehicle speed | What is happening around the vehicle? | Mostly necessary | Mostly necessary |
| 6 | Map, route and trip information | Where should the vehicle go next? | More critical | Less critical |
| 7 | Steering angle, turn signal, predicted ego trajectory, outside sound | Rated necessary | ||
| 8 | Traffic signs, lights and object highlighting | What must I react to while driving? | Less critical | More relevant |
A wider “World View” camera perspective helped operators feel in control and improved remote situation awareness in the 2025 study. Mutzenich et al. expect interface designers to end up with a data overlay that combines several information sources about the car and the road.
Remote assistance and remote driving need different consoles
The two modes differ in who performs the driving task. SAE J3016 (2021) defines remote assistance as event-driven information or advice from a remote human, with the automated driving system still performing the complete driving task. It defines remote driving as real-time performance of part or all of the driving task, including braking, steering and acceleration, by a remote driver. In tele-assistance the operator still watches the video but delegates low-level manoeuvres to the vehicle, issuing high-level directives through the interface. A static obstacle might need only guidance, while merging into a busy intersection might call for full remote control.
The modes also tolerate network problems differently. Experts in the Brecht et al. workshop attributed remote assistance’s greater robustness to network instability to its discrete inputs, whereas remote driving’s continuous inputs are more exposed. That workshop ranked remote assistance concepts above remote driving on every KPI except controllability. Controllability suffers in assistance because a decision generally cannot be changed once it has been sent to the vehicle. In the TUM study, remote driving needed more elements rated as necessary, including steering angle, turn signals with hazard lights, outside sound and the predicted ego trajectory.
In practice the staleness gate should be strict in remote driving, where a late frame feeds straight into a steering correction. In assistance it can be a warning, provided the console keeps monitoring how the command is carried out. The 2025 study notes that after issuing a high-level command the operator must still monitor execution until the scenario is resolved. Its authors propose offering both modes in the same console and argue for interfaces that suggest the most appropriate mode for the situation.
Who can build the escalation console and the telemetry behind it?
TechnoLynx builds the escalation console and the telemetry feeding it through its Generative & Agentic AI R&D engagement: confidence scoring, reason codes and frame-age and link monitoring, running on your compute and network stack with as little added latency as possible. The engagement covers choosing which indicators to show and how to rank them, implementing the pipeline, validating it in timed operator trials that measure decision accuracy and time to decide, and checking staleness flags against ground-truth network conditions. Deployment follows, with monitoring of indicator accuracy and operator outcomes so thresholds can be refined. When choosing any partner, ask how they will prove the flags fire on time under injected delay.
A team that already has an HMI human-factors group and mature vehicle telemetry may reasonably build in-house, and university human-factors labs are a good route for formal operator studies. For the wider automotive context, see our AI engineering for manufacturing and automotive work.
What still has to be proven on your own fleet
The studies cited here test speed, usability and situation awareness, not whether such a console reduces wrong takeover decisions compared with video alone. The supporting evidence covers faster task completion and better usability with dynamic displays, not decision accuracy. A remote-assistance workplace evaluated by Schrank et al. (2024) with 34 participants meeting Technical Supervisor criteria let them reach sufficient situation awareness and resolve scenarios quickly, even under added cognitive load. That is encouraging, but it was tested in simulation. The open question for any fleet is whether its own confidence scores are calibrated well enough to show, and whether its staleness flags fire before the operator acts on an old frame. We would log what the operator was shown, including frame age and the reason code, as input to defensible session recording for teleoperated driving.
