A lab score or a clean simulation run tells you how an autonomous system behaves in conditions someone chose to model. It does not tell you how the system behaves under the shadows, glare, snow and traffic of the place it will actually run. Getting from one to the other takes staged validation in the real world. First, harden perception against the new site’s lighting, weather and marking variation, using synthetic data and replay tests. Then, for any controller that acts on the world, run a shadow-mode trial. The controller’s decisions are logged next to the incumbent system’s on live traffic, and it gets no control authority until that comparison supports it. For anyone weighing the safety of autonomous vehicles or smart traffic infrastructure, that ordering matters more than any single benchmark number.
How do you validate robustness on real roads instead of trusting the lab?
You start by accepting that ordinary environmental conditions move perception performance by large margins, and then you measure the system under those conditions rather than assuming they average out.
On lane perception, the LanEvil++ benchmark (2026 preprint) found that environmental illusions such as shadows, reflections and tyre marks cut lane-detection accuracy by 5.27% on average, with shadows the most damaging at up to 7.20%. The authors grounded those illusion types in the US FARS/CRSS crash data and the UK DfT road safety statistics, which both list shadows, dazzling reflections and visibility obstruction as causal factors in accidents. The simulation figures are only the lead-in. When the same team rebuilt four routes on a physical scaled sand-table track, a lane-detection-driven robot’s success completion rate fell from 80.00% to 22.5% once shadow and road-damage illusions were added. A driving vision-language model fell further, from 57.5% to 10.0%.
Object detection shows the same pattern. Michaelis et al. (2019) found that standard detectors drop to 30 to 60% of their original performance on corrupted images. A detector trained in clear Cityscapes conditions lost almost half its performance under the strongest level of simulated fog (Foggy Cityscapes). A daytime-trained model also performed much worse at night. Location matters too: a University of Michigan roadside perception study (2023) shows a detector trained at one site degrading badly at a new one, and a training set with no night images giving poor night-time results even at the same intersection.
What is the quickest way to make a perception stack robust to new markings and lighting before a demo?
When time is short, stay at the software level. Generate synthetic variants of the new site’s lighting, weather and pavement markings, fine-tune the perception stack on them, and check the result by replaying real logged sensor data on the vehicle. New physical calibration only comes into it if the sensor rig or optics at the new site differ from the ones you trained on.
TechnoLynx runs this pre-demo hardening as a Generative & Agentic AI R&D engagement. The first step is to sort the current demo-site logs into appearance-driven and calibration-driven failures. After that, TechnoLynx chooses domain-randomisation parameters matched to the new site, builds and runs the hardening pipeline on the target hardware, replay-tests the baseline model against the hardened one, and packages the hardened model and configuration for the demo vehicle.
Can synthetic data add a new site’s visual conditions to training and test data?
Yes, and there is field evidence for it. The Michigan roadside study rendered 3D vehicles onto real background images captured under different weather and lighting. It used SUMO traffic simulation to place the vehicles, then a GAN-based enhancer (Contrastive Unpaired Translation) to make the rendered vehicles look photo-realistic. The team evaluated on real images from six field tests at the Mcity test facility in July and August 2022, covering sun, cloud, light and heavy rain, day and night. In harsh conditions, their model gained 6.1 mAP over a COCO-trained model, compared with 1.6 mAP in normal conditions. The gain was larger where the conditions were worse. The same pipeline also recovered snow-condition performance on a deployed system whose fall-only training data had left it degrading in winter.
Domain randomisation takes a different route to the same goal. Instead of matching the real world closely, it randomises the simulator so widely that reality looks like one more variant. Tobin et al.’s original paper on the technique designed it to allow low-quality renderers that were not matched to real textures or lighting. They also found that transfer degraded significantly with fewer than 1,000 random textures, while extra images added little once 1,000 textures were used. Stylised augmentation is a simple data-augmentation trick. Michaelis et al. report gains of about 12% on COCO and 41% on Cityscapes under corruption from adding a stylised copy of the training data, at a cost of 0 to 2% on clean images.
For lane markings specifically, a 2022 sim-to-real study using a CARLA-based generator found that detection and classification accuracy rose as the synthetic dataset grew. It also found that two-stage image-translation pipelines did worse, by 4 to 6% on detection, with lanes hard to identify in the translated images. The authors think this is likely because the translation stage was trained without the lane-task losses. Making synthetic images look real is not enough on its own.
Where synthetic hardening falls short
Synthetic data will not reproduce every piece of real lighting physics, so a sim-to-real gap remains. Sensor effects also contribute to that gap. Work on learned sensor-effect augmentation (2018) showed that transferring a real camera’s blur, exposure, noise, colour cast and chromatic aberration onto synthetic images narrows the domain gap. It also needed fewer training images than the domain-randomisation baseline in that study. That is why on-vehicle replay is the check that counts. Replay runs the hardened model on real logged sensor data, so a gain that only exists on synthetic images shows up as no gain.
Replay has its own limit, which is easy to forget the day before a demo. It only covers scenarios you have already logged, not the literal new site. A rushed hardening pass also trades depth for speed, so rare conditions can stay unhardened. Showing that scenario coverage is complete against a declared operational design domain is a larger exercise, and it belongs in a release-sign-off process, not a pre-demo sprint.
Does a new deployment site need new sensor calibration?
Only when the hardware or its geometry has changed. Lighting and marking variation is an appearance problem. The table below is how we would make the call from the logs.
| What the logs show | Likely cause | First action |
|---|---|---|
| Errors cluster in shadows, glare, wet roads, night or unfamiliar marking styles | Appearance variation | Synthetic augmentation and hardening, then on-vehicle replay |
| Projected detections are offset consistently from LiDAR returns or ground truth, whatever the lighting | Extrinsic misalignment between sensors | Recalibrate first, then re-test |
| Different camera, lens, mounting position or LiDAR model at the new site | Changed intrinsics or geometry | Recalibrate, then harden against appearance |
| Degradation tracks one camera’s exposure or noise behaviour | Sensor-domain shift | Sensor-effect augmentation matched to that camera |
Recalibration does not always mean a target-board session. TEScalib (Hu et al., 2022) estimates LiDAR-to-stereo-camera extrinsics from the geometry and photometry of the surrounding scene during driving, with no calibration targets, and adds an uncertainty estimate for the result.
What is a shadow-mode trial, and how long should it run before a controller gets live control?
In a shadow-mode trial, the candidate controller watches live traffic and computes the decision it would make. Every decision is logged without affecting the intersection, while the existing timing stays in control. In one vendor-reported case, the trial ran for about a month before the controller was given any live authority. We would treat a month as a floor rather than a target, because a trial covers only the conditions present while it runs.
TechnoLynx can turn “validated against real conditions” into operational criteria for a pilot corridor. That means designing the shadow-mode trial and the comparison method against SCATS, and building the pipeline that logs the candidate’s and the incumbent’s decisions on the same traffic. From there, TechnoLynx validates the controller’s decisions against the real downtown data and supports a phased move from shadow mode to limited live control.
One shadow-mode account comes from a vendor-reported case write-up of an AI traffic controller. The system watched traffic, worked out which signal plan it would choose, and logged every decision without touching the intersection. For about a month, those decisions were compared with the fixed timing, and the system moved to live control only after it beat the baseline plans in measured tests. The scope was also narrow by design: the engine chose from 10 already-approved signal programmes instead of inventing new timings. That constraint is worth copying, because it limits the harm a wrong decision can do once the system goes live.
On trial length, published validation guidance for adaptive signal control is useful. In the City of Mesa test, the adaptive system was switched off for approximately 30 days, spread randomly over two months. Over that period, the team collected green occupancy ratio, percentage of arrivals on green, platoon ratio, and route travel times and reliability. Spreading the days at random is the point: it stops one week’s events or weather from deciding the result.
Two limits stay in place however long the trial runs. Shadow mode cannot fully validate the effects of actuation, because the controller never changes a signal, so drivers and pedestrians never react to its choices. A short trial also covers only the conditions present while it ran. Even the careful Park City evaluation collected its data under fair weather and dry pavement only.
Can you show it works on real downtown traffic and pedestrians, not just in the simulator?
Only by running it there. A simulator reproduces the pedestrian behaviour and demand swings someone modelled, and a downtown corridor produces the ones nobody did. The Utah Department of Transportation took the same route: after several years of studying adaptive control in microsimulation, it installed a real system in Park City. It chose the site partly because event traffic there shifts demand sharply.
How do you compare an agentic controller’s decisions with what SCATS would have done on the same traffic?
Run two comparisons, because they answer different questions. The first is decision by decision: during shadow mode, log the candidate’s chosen action next to SCATS’s actual action for every cycle and intersection, then study where they disagree and what the traffic looked like at the time. SCATS adjusts cycle lengths, splits and offsets each cycle from stop-line detector measures of degree of saturation and flow. Each logged SCATS cycle is therefore a concrete decision you can line up against the candidate’s.
The second is outcome-level, and it needs the systems to take turns in control. The Park City evaluation of SCATS is a good template. It compared SCATS-on with SCATS-off weeks back to back in September 2007, using the same morning, afternoon and weekend midday peaks, with probe-vehicle travel times and intersection stopped-delay studies. Differences were tested at the 95% confidence level. Weekday morning travel times fell by an average of 7.6% under SCATS, but northbound afternoon travel times were not statistically different. A candidate that “wins” on raw averages can still be inside the noise for whole periods.
Do not assume the incumbent is a strong yardstick. In the Park City study, both arms had been tuned carefully: the fixed plans were maintained by five technicians, and SCATS had more than a year of parameter adjustment. SCATS’s own field record is described as much better than conventional timing, especially when that timing has not been updated recently. In FHWA’s field tests of adaptive systems from 1996 onward, results were mixed and none of the systems consistently improved travel time and delay. Beating a poorly tuned baseline proves little. Beating a well-maintained one, at significance, across seasons, is evidence. Also check that the inputs are healthy. FHWA’s Gahanna ACS Lite pilot found the system failed without good detection and often made traffic worse when a communications link failed.
The validation sequence, in order
- Simulate the modelled corridor to rule out obvious failures and tune the controller.
- Run shadow mode on real traffic with no control authority, for about a month, as in the cited trials, and long enough to cover the demand patterns and seasons that matter.
- Document the comparison against SCATS decision by decision, and back it with outcome measures from off and on periods, tested for significance.
- Only then grant limited live control: a subset of intersections, a pre-approved plan set and a defined handback to the incumbent.
None of these steps should be skipped for an investor demo or a schedule. A controller that has not been through step 3 has not been validated, however good the simulator plots look.
Where simulation, X-in-the-loop and real-road trials fit in ADAS testing
Treat these as rungs of one ladder, not alternatives. X-in-the-loop testing (model-, software- and hardware-in-the-loop) swaps simulated components for real ones step by step, while the environment stays simulated. LanEvil++ shows the ladder in miniature: the team ran closed-loop tests in CARLA with OpenPilot and LMDrive, then rebuilt the same routes physically to check whether the failures carried over. They did, with success on the physical track falling sharply. Each rung should be able to fail the system, and real-road or shadow-mode trials are the rung that tests the assumptions every earlier rung shares. Measured sustained latency and thermal behaviour on the target compute module belong in hardware-in-the-loop work as well, but that is a separate measurement problem.
Who can harden our perception stack and run the shadow-mode trial?
TechnoLynx runs the hardening work as a Generative & Agentic AI R&D engagement: triage of failure logs, synthetic augmentation, a hardening pipeline on your target hardware, and on-vehicle replay against your baseline. For the controller, TechnoLynx defines the shadow-mode criteria and comparison metrics against SCATS, builds the logging pipeline and supports the phased handover to limited live control. Our broader AI engineering for manufacturing and automotive page describes how we take on perception and control projects like these. Building this in-house makes sense if your team already owns the replay harness and the traffic data access.
If you need evidence that will stand up in front of an assessor, plan a separate pre-assessment and independent validation before the regulator-facing safety audit. If the real problem is retuning signal timings, a traffic engineering consultancy or your adaptive-control vendor is the more direct route. And if the work is still at proof-of-concept stage with no incumbent to compare against, a full shadow-mode programme is more than you need yet.
FAQ
What does ADAS stand for in vehicle safety?
ADAS stands for advanced driver-assistance systems. Lane keeping assist and adaptive cruise control are typical examples, and lane detection is widely used in both.
What does “perception driving” mean?
Perception is the part of a driving system that turns camera, LiDAR and radar data into a model of the scene: lanes, vehicles, pedestrians and signals. In LanEvil++ closed-loop tests, illusions such as shadows led perception-driven systems to incorrect driving decisions.
