Autonomy stall logging has to capture four things on one clock: what the sensors delivered, what fusion made of it, what the planner believed and decided (its confidence values, timers and decision trace), and the health and mode of every module. Published practice for road automated driving sets the floor. The AVSC best practice recommends recording at least 5 seconds before the trigger, at 2 samples per second for salient-object data and 5 per second for ADS actions and vehicle motion control, with data recorded by multiple ECUs aligned in time. With that record in place, a stall is something you replay through the same stack instead of reconstructing from sparse text logs.
Those documents were written for automated road vehicles. The same layering suits mobile robots, agricultural and mining machines and warehouse fleets, where a planner that stops and refuses to replan is exactly the kind of incident that is expensive to reproduce in the field.
What should an incident log for an autonomy stall include?
An incident log for an autonomy stall needs synchronised raw sensor data, intermediate fusion outputs and the planner’s internal state, all captured at the time of the stall, because none of them can be rebuilt afterwards from sparse logs. Reconstruction needs structured, timestamped logging that correlates across the sensor, fusion and planner boundaries, plus replay tooling.
TechnoLynx builds this logging and replay tooling for fleet autonomy stalls. TechnoLynx starts by working out which sensor, fusion and planner signals have to be captured to tell the fault layers apart. TechnoLynx then implements the synchronised instrumentation and builds a replay harness that re-runs recorded data deterministically through the planner stack, as part of its AI engineering for robots, drones and autonomous machines.
The research framing of the problem is useful here. Berkeley DeepDrive separates autonomous-system logs into input event logs (cameras, GPS, lidar, radar), prediction event logs (the decisions the system makes about direction, speed and lane) and system event logs from the operating system and controllers. Its design goal is that the logged information should be enough both to detect an anomaly and to diagnose its root cause, which are two different bars. The AVSC practice organises the same ground into four categories: what the ADS did, what it thought was important, what the sensors saw, and general parameters such as time of day and trigger type. It also ranks data elements into three priority tiers, with lower tiers important to recover only as long as higher-tier recording is not negatively impacted.
Minimum signals per layer, with published rates
| Layer | What to record | Published minimum |
|---|---|---|
| Saliency (fusion output) | Salient object classification, ID, relative position, velocity; traffic control device state | 2 samples/s over the 5 s before time zero, with each element ranked Priority 1 or 2 |
| Planner actions | Requested lateral and longitudinal motion, requested gear, reference speed, ADS mode, failure mitigation strategy activated | 5 samples/s for ADS actions, 4 samples/s for requested gear |
| Control and vehicle response | Brake delivered effort, indicated speed, actual steering position | 5 samples/s |
| Mode transitions | Time-stamped activation, deactivation, transition demands and overrides | Time-stamped and stored continuously, in the DSSAD model |
| Recording window | Time before and after the trigger | AVSC: 5 s before time zero, with recording past time zero recommended; AHEAD proposal: 30 s before to 10 s after |
The SAE recommended practice for ADS data loggers does the same job. It defines a minimum data element set and a common record format, and requires the logger to capture information leading up to the triggered event.
These minimums were set for collision analysis. The table records what the planner requested. Its element list does not include why the planner held: the confidence values behind that choice, the state of its timers and timeouts, or the decision trace that ended in “wait”. The AVSC definition of an event already includes failures that push the system into a fallback manoeuvre to reach a minimal risk condition. For a robot fleet we would add a stall trigger of our own, such as the planner holding without progress past a set time or any module reporting a stale status, so that the event buffer is dumped when it matters.
Stamp every message and keep the stack in one recording
Time is the most important logging item for reconstruction. The ETRI log analysis framework records every logging item against time precisely so that problem analysis can start from time. AVSC asks for data from multiple ECUs to align within the minimum recording frequency, and to be correlatable through elements such as odometer and ignition cycle. On distributed ROS 2 hosts, the ros2_tracing message-flow work synchronised the clocks of the two machines with NTP before merging traces.
The TUM autonomous driving team (2025) found that logging signals separately inside each of many nodes produces scattered log files that make full-stack problems hard to debug. They chose rosbag so that ROS 2 messages and internal debug signals sit in one file that can be replayed for simulation. They also enforced message stamping on every publisher, which made latency analysis from recorded bags efficient, and had every node monitor its input topics for timeouts. rosbag2 playback can order messages by reception or publication timestamp, with reception as the default. For stall work we would record both, because the gap between when a message was produced and when the planner got it is evidence in its own right.
Lidar fault, fusion bug or planner timeout: how do we tell them apart?
You tell them apart by isolating each layer’s output at the moment of the stall and checking it against known-good reference behaviour for that layer, working downstream from the sensors to the planner. Session video or evidence recording on its own cannot do this. It shows what happened, but without fusion and planner state kept separate by layer it cannot say which layer caused it.
The replay step has public precedent. Autoware’s driving_log_replayer plays a recorded rosbag into the stack, lets Autoware publish its /diagnostics topic, and has an evaluation node score the result. In the NTSB investigation of the 2018 Uber ATG crash in Tempe, the developer supplied a playback of sensor and vehicle dynamics data, and investigators used it to build a timeline. ATG later re-ran the crash sensor data through an updated software version to test the fix. Feeding recorded data back through modified software is the same move a stall investigation needs.
A layer-by-layer localisation procedure
- Replay the recorded event through the same stack version and confirm that the stall reproduces; a replay that does not reproduce it localises nothing.
- Test the sensor layer for admissibility. Antonante, Spivak and Carlone (MIT) describe consistency checks between a module and its predecessor, for example flagging the lidar module when its point cloud contains many coincident points. Autoware’s blockage evaluation checks that ERROR is raised above a set rate when a lidar is deliberately covered, and not raised when it is uncovered.
- Compare the system’s behaviour with and without the suspect fault, as the FADE study’s differential testing does with injected sensor faults. FADE found sensor-fault effects reliable and deterministic in physical tests, mirroring their behaviour in simulation.
- If the sensor data passes, inspect the fusion output over the window. In the Tempe case the system kept changing an object’s classification between vehicle, bicycle and other, and could not predict its path as a result. Unstable tracks fed into a planner that is behaving correctly look like a planner problem until you look one layer up.
- If the fusion output is stable, examine the planner’s own state: its input-topic timeouts, its timers and the module status. In the TUM stack a module that can no longer produce meaningful output reports STALE and stops publishing, and the controller falls back to an emergency trajectory when valid trajectories stop arriving.
| Suspected layer | What the log shows at the stall | Replay result that confirms it |
|---|---|---|
| Sensor fault | Raw stream fails admissibility checks (blockage, coincident points, dropouts); sensor diagnostics at ERROR | Stall reproduces, then clears when the suspect stream is replaced by known-good data |
| Fusion bug | Raw sensors pass; tracks flip class, lose history or disagree with the raw data | Stall reproduces with clean sensor input; injecting reference tracks clears it |
| Planner timeout | Fusion output stable; planner input topics time out, module reports STALE, no valid trajectory published | Stall persists with reference tracks; timer and callback timing explain the hold |
Two cautions keep the procedure honest. First, the layers interact. FADE found that fusion compensates when one sensor loses partial data, but strong light degrades camera and lidar together and removes that fallback, so a sensor co-fault can surface as a downstream error. The same study found lidar faults had much stronger effects than camera faults, with lidar deflection and displacement skewing distance measurements. Second, bug base rates are only a prior. Garcia et al. (ICSE 2020) found that planning held 27.05% of 499 bugs studied in Apollo and Autoware, against 16.63% for perception. However, the bug reports did not say whether a bug made the vehicle stall. Planning is a reasonable first suspect, but the replay has to decide.
Race conditions need timing traces, not only message logs
A deadlock caused by a timing or race condition may not reproduce in replay if the original logs did not capture precise inter-process timing. Bédard et al. (2022) point out that standard middleware-based data recording does not give enough information on internal computation and performance bottlenecks. Their ros2_tracing framework, built on the LTTng tracer, added a mean end-to-end message latency overhead of 0.0033 ms with all ROS 2 instrumentation enabled. Its message-flow extension infers causal links between input and output messages and visualises the executor’s state over time, and the ROS 2 executor is itself a known performance bottleneck.
Replay fidelity also depends on how much of the original timing and sensor noise was captured, and coarse logging can make a replay actively misleading. Physical runs bring in non-deterministic factors such as CPU load and packet drops. rosbag2 itself notes that recording at high rates can produce spurious packet drops under low bandwidth, high CPU load or slow storage. A gap in the recording can therefore look like a sensor dropout, so check the recorder’s own health before blaming the lidar.
Sizing autonomy stall logging to a fleet storage budget
The minimum logging rate and window follow from the resolution you need to reconstruct planner and sensor state around an event. Multiply that by event frequency, fleet size and retention period, then check the result against storage and offload bandwidth. With intermittent connectivity, that calculation points to selective, prioritised capture rather than continuous high-bandwidth offload.
TechnoLynx sizes stall logging so stalls stay reconstructable within the fleet’s storage budget, turning that requirement into a rate, window and retention specification. TechnoLynx then chooses between continuous and event-triggered capture, compression and on-device pre-processing, implements the pipeline on the target onboard hardware, and checks at fleet scale that it stays within budget.
Continuous capture is the expensive baseline. The AVS technical report (2025) puts full-fidelity logging across cameras, lidar, radar and GNSS/CAN at 14 TB per day. A 2024 review gives camera data rates of 20 to 40 MB per second and lidar rates of 10 to 70 MB per second. Forensics, as the AVS authors put it, needs seconds around an event, while analytics and training need sparse samples over months. rosbag2’s snapshot mode serves the forensic case. It keeps an in-memory circular buffer bounded by size and, optionally, duration, and writes it to disk only when triggered, for example by a detected error condition.
Worked example: what one triggered stall record costs
Assumptions: the AHEAD window of 30 s before and 10 s after the event (40 s in total), and the review’s per-sensor data rates.
- One lidar: 40 s × 10–70 MB/s = 400 MB to 2,800 MB raw per event.
- One camera: 40 s × 20–40 MB/s = 800 MB to 1,600 MB raw per event.
- Applying AVS’s reported codec ratios (6.56 for LAZ on lidar, 4.06 for JPEG on images) to these raw figures brings the upper bounds to about 427 MB and 394 MB.
- Using the AVSC 5 s window instead, one lidar drops to 50–350 MB raw.
- Fleet storage per retention period = per-event size × sensors × events per machine per day × machines × retention days. Lidar and image data dominate the storage footprint, as AVS reports.
Underestimate event frequency or window length and a selective scheme still breaks the budget. Retention itself is left to the manufacturer under the AVSC practice.
Tiered storage stretches the budget further. AVS ingests to an SSD hot tier for short-term queries and migrates data to an HDD tier for long-term retention. On the HDD tier, XFS showed no fragmentation, unlike EXT4, which matters for long sequential reads during replay. Overall AVS shrank logs to 4.0 GB on average, against 33.8 GB for raw ros2bag and 19.9 GB for per-message Zstd compression. Reduction needs care for diagnosis, though. AVS reports that its reductions preserve tracking and localisation accuracy, measured on those tasks rather than on fault diagnosis. We would keep raw sensor data inside the triggered window and apply reduction only outside it.
Can stall logging ship as a software update, or does it need new onboard hardware?
Stall logging can ship as a software update when the existing storage capacity, write bandwidth and compute headroom can sustain the target triggered rate and window. If they cannot, storage, bus or compute changes have to ship alongside the software. The risk is real-time interference. The TUM team (2025) notes that logging their entire stack consumes much of the available hardware and may disturb real-time execution.
Where the software route is open, the published fixes below move logging off the critical path. A thesis on autonomous vehicle software at Instituto Superior Técnico buffers log data during the control cycle and writes it only after the cycle completes, leaving close to 10 ms per cycle at 100 Hz. Writing to a pipe read by a lower-priority child process costs the critical process as little as 5 microseconds. The same work measured zlib deflate logging at 2 ms worst case, enough for compressed logging at up to 500 Hz. For rosbag2, the MCAP fastwrite preset skips CRCs and the message index to maximise throughput on resource-constrained robots, while the zstd_small preset gives very small bags but can be resource-intensive to write.
The hardware route AVS demonstrates is a separate storage computer. AVS runs on its own compute unit, decoupled from the autonomy compute, so that storage, compression and indexing never touch the perception–planning–control loops. Its prototype was a Raspberry Pi 5 with a 256 GB NVMe SSD and a 1 TB HDD, and p99 ingest latency stayed within 10 Hz and 50 Hz budgets.
Software-only feasibility checklist
- The snapshot buffer for the chosen window, at native sensor rates, fits in spare RAM, since snapshot mode holds it in memory.
- Storage write bandwidth absorbs a buffer dump without dropped messages; rosbag2 lists slow mass memory among the causes of drops at high recording rates.
- Compression runs in a separate low-priority process or after the critical path.
- Compressed event records can be offloaded over the fleet’s wireless links in the time available.
- End-to-end latency is measured with and without the new logging under the same load. In the ros2_tracing tests, a 10 Hz graph run for 20 minutes with and without tracing showed a difference of means of 0.1597 ms.
If any item fails, a software-only retrofit can force heavy compression or shorter retention, either of which degrades the reconstruction you were trying to buy. Even when every individual machine passes, continuous high-bandwidth offload across the fleet is impractical over intermittent connectivity.
After the replay: closing the stall out
The documented root cause, the fix, the regression scenario that proves it and the sign-off belong to a corrective-action process, which pairs naturally with the error budgets and incident response practice used for model serving. The AVSC practice also says the recording system should be tamper-resistant and compliant with ISO/SAE 21434. Audit-trail design is a separate question from diagnosis, though, and deserves its own treatment.
TechnoLynx treats incident-log design as part of production AI reliability engineering, which is distinct from model accuracy: its deliverables are artefacts that catch failures before customers do, not dashboards. The test of a stall logger is plain. Trigger a synthetic stall on one machine, pull the record, and time how long an engineer who did not build the logger takes to name the faulty layer.
