CARLA is an open-source driving simulator built on Unreal Engine, and with co-simulation it has been extended to unpaved, rough terrain. Whether its results, or any simulator’s, can count toward safety sign-off has a two-part answer. First, the sensor and dynamics models have to be calibrated against logs recorded on your own machines and sites. Second, a safety board has to set and document the residual gap it will accept, instead of taking it from a simulator default. Scenarios the calibrated simulator still cannot reproduce credibly go to hardware-in-the-loop (HIL) or field testing.

Why both steps? UL 4600 lists arguing low risk from unvalidated simulation results alone as a pitfall, because simulation defects, modelling faults and abstraction simplifications can hide risks. EU type-approval also scrutinises results from virtual toolchains. Commission Implementing Regulation (EU) 2022/1426 sets criteria for assessing manufacturers’ validation results from virtual toolchains, and Annex III of NATM requires toolchain credibility to be established through code testing, verification and validation, calibration and sensitivity analysis.

Measuring fidelity is an engineering problem, and deciding how much infidelity is tolerable is a governance problem.

Can the CARLA simulator handle off-road, mining or field robots?

CARLA is widely used for virtual testing of autonomous road vehicles. Recent versions improved vehicle volumes and core physics such as wheel friction, suspension and centre of mass. For unpaved, rough terrain, CARLA needed enhancing. AVL extended CARLA by coupling it to its AVL VSM vehicle dynamics software and a soft-soil tyre model through the Model.CONNECT co-simulation platform, synchronising every wheel contact position. The setup was used to close the control loop for an autonomous tractor. A later release moved CARLA from Unreal Engine 4.26 to 5.5, and its Town 10 roads gained camber, potholes and speed bumps that affect both vehicle dynamics and sensors.

A survey of self-driving simulators argues that a simulator must trade scene realism against simplified vehicle dynamics. For a haul truck, an agricultural robot or a yard vehicle, we would treat CARLA as a capable host for sensors and scenarios. We would not trust it as the terrain-dynamics model on its own. We would take that physics from a coupled tool or from calibration against your own machine.

Is CARLA free, and what hardware runs sensor-heavy scenarios?

CARLA is open source. Its documentation lists Windows 10 and 11 and Ubuntu 20.04 and 22.04 as supported. It recommends a dedicated GPU equivalent to an NVIDIA 2070 or better with at least 8 GB of VRAM, plus about 20 GB of disk space and two TCP ports (2000 and 2001 by default). That is a starting point and does not size a multi-LiDAR rig. If RAM or GPU performance falls short, simulation quality degrades through frame drops, input lag and delays in sensor emulation.

CARLA’s LiDAR casts one ray per laser channel in every step, with points per channel per step equal to points_per_second divided by FPS times channels, so channel count and rate scale the compute directly. If CARLA shares GPUs with a second simulator or with training jobs, we would isolate or prioritise GPU resources explicitly.

Connecting CARLA to a ROS 2 autonomy stack

The CARLA ROS bridge for ROS 2 needs CARLA 0.9.11 or later, because earlier versions are not compatible. Its documentation targets ROS 2 Foxy on Ubuntu 20.04 and recommends matching the bridge version to the CARLA version. It is built from source with colcon, since there is no Debian package for ROS 2. CARLA 0.9.16 ships native ROS 2 integration with DDS-based message passing and time synchronisation. It connects to Foxy, Galactic, Humble and other distributions without a bridge tool’s latency.

The bridge runs in synchronous mode by default and waits for all the sensor data expected in the current frame, so results are reproducible. CARLA calls synchronous mode with a fixed time-step the best mode when synchrony and precision matter. Native integration removes bridge latency, but fixed-step simulated time need not track real time. Timing claims need the HIL treatment discussed further down.

How realistic are CARLA sensors against your own LiDAR logs?

CARLA’s LiDAR is a ray-caster with a few statistical controls. Its noise model perturbs points only along each ray, leaving angular positioning perfect. Huch et al. built real and simulated datasets of identical scenarios. LiDAR detectors trained on the simulated data still lost around 14% average precision on real data, and a simple noise model reduced that shift.

TechnoLynx builds this calibration for autonomy teams. TechnoLynx defines which dust, occlusion and degradation modes must be reproduced and chooses between parameter fitting and a data-driven noise model. TechnoLynx then builds the log-replay and calibration harness on your simulation stack and reports the gap per site and dust condition for safety-board review. The engagement’s first step is framing: which degradation modes matter, and which recorded logs exist to compare against. The calibrated model then goes into your regression pipeline with recalibration triggers, as part of TechnoLynx’s AI engineering for robots, drones and autonomous machines.

Fitting CARLA’s LiDAR parameters to recorded dropout

CARLA exposes three LiDAR attributes that matter here:

• noise_stddev (default 0.0) perturbs each point along its ray, which gives perfect angular positioning but noisy range.
• atmosphere_attenuation_rate (default 0.004) sets the intensity loss per metre.
• dropoff_general_rate (default 0.45) sets the proportion of points dropped at random.

Low reflectivity and low incidence angles cause points to drop on real sensors, and Huch et al. note that leaving this unmodelled widens the sim-to-real gap. They had to use random downsampling because material and incidence properties were not available in simulation, and that choice itself hurt detection performance.

The ALDUS model treats dust as spherical, homogeneous particles with configurable size, density and cloud volume, and it estimates both energy attenuation and backscattered intensity. Its authors report good agreement with their own experimental measurements. PCGen takes another route with a small learned surrogate for the laser head. The surrogate predicts each ray’s return probability from distance, incidence angle and simulated intensity, so it does not need pixel-wise correspondence between real and simulated scans.

A calibration loop for dust-degraded LiDAR

1. Replay logged scenarios. Rebuild the recorded trajectories and scenes so that any differences come from the sensor model, not the scenario. Huch et al. used scenario-identical datasets for exactly this reason: to separate sensor-model effects from distribution shift.
2. Make every sweep reproducible. CARLA’s physics determinism requires synchronous mode with fixed delta seconds.
3. Compare statistics per range band and dust condition. In Huch et al.’s data there was no domain shift at mid-range but a clear one at close and long range, so a single averaged figure hides where the model is wrong.
4. Fit the parameters. For the dust cloud these are particle size, density and cloud volume. For the sensor they are noise, attenuation and dropout, or a learned raydrop model. Robot-simulator calibration offers objectives: SimOpt evolves parameters against a real-versus-simulated trajectory discrepancy loss, and one benchmark optimises mean squared error between trajectories.
5. Score at the perception level as well. PCGen’s Learned Point Cloud Similarity metric correlates negatively with detection mAP, which makes it a gap estimate from the perception model’s point of view.
6. Hold logs out. Keep separate reference, validation and test trajectories so the fit is not judged on the data it was tuned to. Many parameter combinations can produce the same output, which is the motivation BayesSim gives for keeping uncertainty over parameters.

As a sanity check on fitted values, one lidar degradation framework sets its severity tiers at roughly 15 to 40% dropout and 1 to 5 cm of Gaussian range noise. The authors base these ranges on empirical observation and sensor physics.

Should calibration be redone per site or per dust condition? We would recalibrate per dust source, and whenever the site material or the sensor changes. On the LiDARDustX mining data, natural dust mainly increased false positives and false negatives, while dust kicked up by vehicles degraded orientation precision and increased false positives. The effect of dust on detectors depends on its source. Calibration only covers the conditions the logs actually captured, so more severe or novel dust remains unvalidated until it has been recorded.

What gap between simulated and real dropout will the safety board accept?

There is no industry-standard threshold for simulation fidelity. The S2R-Bench authors say so explicitly, and they chose 30% as their own operational threshold for ‘satisfactory simulation’ based on their experimental data. Measured gaps vary widely by condition. For LiDAR in fog, in one of the benchmark’s LiDAR comparisons, the closest method was 35.1% below real data in the 3D view and 50.1% below it in bird’s-eye view, and in tunnels the drops were 49.0% and 48.2%. A study of a wheel loader digging in deformable terrain found an overall sim-to-real error of about 10%, which its authors judged sufficient to transfer their force-feedback controller. The same authors argue that the gap has no objective measure: it counts as small when it is smaller than the natural variation between instances of the real system, and it depends on the task.

So the threshold is a risk decision. ISO 21448 (SOTIF) fixes no value for acceptable residual risk. Setting the criteria involves regulators, society, OEMs and suppliers, and the standard offers principles such as GAMAB, positive risk balance, ALARP and minimal endogenous mortality. Acceptable risk can be budgeted per scenario, and budgets can move between scenarios during verification and validation as long as the overall criterion holds. UL 4600 makes the acceptable level depend on how critical the risk is and on its contribution to item-level risk as the safety case establishes it.

Should the threshold vary by ODD or dust severity? UL 4600 requires the safety case to justify safety for a defined operational design domain. Robustness also changes with context: in one lidar degradation study, indoor sequences failed far more abruptly at extreme severity than outdoor ones. A single global number is the wrong unit. We would set one per ODD slice and dust-severity tier, because the error a perception stack can tolerate in light haze is not the error it can tolerate in a blasting plume.

The board that owns the safety case signs off on whether the simulator is close enough. The engineers supply the measured gap and its coverage limits, and calibration never replaces that governance step. One validation methodology for safety-critical simulation toolchains frames this as a loop. It repeats validation until performance targets are met or an informed decision is made to accept known limitations, and those limitations are then documented explicitly in a Toolchain Validation Report.

Why generic CARLA validation figures will not give you your gap

Huch et al. point out that comparing a dataset recorded in CARLA with the real Waymo dataset cannot show whether the gap comes from the sensor model, the distribution shift or both. Task matters too. A study of CARLA’s dynamic vision sensor found that detectors trained only on its synthetic events performed poorly on real data, and its authors caution that the gap varies by task and architecture. In semantic segmentation, KPConv accuracy dropped by up to 38% on the Paris-CARLA-3D dataset compared with real data.

Datasets for dusty mining conditions are scarce. LiDARDustX measures how real dust hurts detectors: PointPillars lost 17.3% AP in dusty scenes. S2R-Bench compares simulated and real sensor anomalies in snow and fog. Whatever CARLA version you run, the gap that counts is the one from a scenario-matched comparison against your own logs for the same scenario class. We would repeat that comparison when the sensor suite or the ODD changes.

Finding which simulated condition diverges most (dust density, terrain roughness or payload) uses the same method. Compare each condition against field logs, recalibrate where the logs close the gap, and collect more real data where they cannot, for example in a sparsely logged load range. Remember that correlation with avoidance errors does not prove cause. The wheel loader study shows the terrain side: it matched terrain parameters to the site gravel and calibrated bucket friction and stiffness to measured dig forces.

Which scenarios still need hardware-in-the-loop or field validation?

Simulation evidence can support a claim for scenarios that are common and well characterised, provided their sensor and dynamics models have been calibrated against logs and the residual error is acceptable. Scenarios that are rare, safety-critical, or dependent on behaviour the simulator captures poorly need HIL or field testing as well, or instead.

TechnoLynx works out which safety claims need simulation, HIL or field evidence, and builds the pipeline that compares CARLA, Gazebo or AirSim sensor and dynamics outputs with logged fleet or flight data. TechnoLynx then quantifies the residual gap, defines acceptance thresholds together with your safety team, and routes the scenarios simulation cannot carry into the HIL and field plan. The engagement starts by mapping each safety claim to the evidence type it needs.

What assessors look for before simulation evidence counts

The TÜV SÜD paper on SOTIF says simulation can map the known scenario area once its trustworthiness has been shown. It suggests finding unknown hazardous scenarios through extended validation drives or re-simulation of a dataset not used during development, and it insists that the validation data be kept separate from development data. The same methodology argues that version-controlled requirements and revised validation reports give independent assessors a verifiable chain of evidence, from the first discrepancy found through to compliance. We would agree the evidence format with the assessor early.

Routing scenarios to simulation, HIL or field

Scenario type	Where the evidence comes from	Why
Common, well-characterised scenarios with calibrated sensor and dynamics models	Calibrated simulation, with re-simulation on held-out logs	SOTIF allows simulation to map known scenarios once trustworthiness is shown
Rare hazards that are unsafe or expensive to stage (GPS loss, collisions)	Simulation first, then a reduced set confirmed on hardware	Simulation is valuable for hazards that are hard to replicate safely
Timing-sensitive interfaces, fault paths, watchdogs	HIL	Software-only checks miss timing overruns, signal scaling errors and watchdog issues
Wind gusts, GPS degradation, sensor-noise edge cases	HIL plus controlled field tests	Single-simulator set-ups struggle to emulate these effects, and simulated noise profiles often differ from real hardware
Real radio links, packet loss and latency jitter	HIL with the mission’s actual wireless hardware	These effects are hard to model synthetically
Unknown hazardous scenarios	Validation drives on routes that match expected use	SOTIF guidance favours real-world drives selected for the expected use

HIL has fidelity limits of its own. Its results are only as trustworthy as the plant model fidelity the question needs, and a simple model cannot prove plant behaviour it does not represent.

Calibrating Gazebo or AirSim wind and GPS-loss models against flight logs

We would count wind and GPS-loss models as evidence only after comparing them with flight logs from your airframe. Rasheed (Sensors, 2026) quantifies discrepancies between PX4 software-in-the-loop logs and real quadrotor missions. He argues that security validation has to be defined by the threshold of natural physical variance. The calibration itself follows three stages: collect trajectories safely on real hardware, train an estimator, then use the real trajectories to estimate parameters and calibrate the simulator. When real and simulated trajectories differ by a small delay, dynamic time warping avoids the error build-up that a plain Euclidean comparison produces.

Field results provide the reference the threshold is set against. In one drone testing pipeline, landings averaged about ±15 cm of error, mostly from wind during final descent. The acceptable gap for obstacle avoidance or landing is then a figure the reviewers agree to, and it should sit relative to that natural spread.

Is the firmware in your HIL rig the same build that runs on the robot?

It should be the same build or a characterised equivalent, and control-loop timing should be measured in both environments. If you only show functional equivalence, timing remains an unexamined cause of any sim-real gap. PX4 shows how this works in practice. Its HITL mode runs normal PX4 firmware on real flight-controller hardware, connected over USB or UART to Gazebo or jMAVSim on a development computer. Even so, core modules such as commander and sensors start in HITL modes that bypass some normal functionality, so simulated data enters at a different point than real sensor data does.

Could simulation time-step differences explain part of the gap?

Plausibly, yes. CARLA substeps its physics by default, with up to 10 substeps and a maximum physical delta time of 0.01 s. Above that delta, velocity and acceleration start to deviate. In autonomous racing, worst-case latency often determines stability more than average latency does, which is why tail percentiles across the pipeline describe execution robustness better than means. A HIL bench closes part of this gap by running the plant model on a fixed clock that the controller cannot pause.

The diagnostic logic follows from this. If firmware and timing are confirmed identical or within a characterised tolerance, attribute the remaining behaviour gap to models, perception or planning. If they differ, rule out timing before blaming the autonomy stack. Stress tests make the comparison concrete: inject sensing delay, reduce control-loop frequency, perturb dynamics and drop sensor data partially, then read the sensitivity curves. Three caveats apply. Bit-identical timing between simulation and hardware is not a practical target. Rare scheduling spikes can destabilise a controller operating near its dynamic limits. And confirming equivalence only narrows down the cause of a gap without closing it.

When Gazebo, AirSim or Isaac Sim fit better than CARLA

Gazebo is known for its flexibility and tight ROS integration. In return, worlds are built by hand, with 3D models, physics and positions defined in XML, and new sensor models can be added through plugins. Sensor noise is set per sensor in SDF. Gazebo’s tutorial uses Gaussian noise with a standard deviation of 0.01 m and calls that reasonable for Hokuyo lasers. In robotics more broadly, Gazebo, Webots and CoppeliaSim are the most popular simulators, chosen for their balance of physical realism, sensor fidelity and ROS integration.

For drones, AirSim supports software-in-the-loop with PX4 and ArduPilot and hardware-in-the-loop with PX4. Microsoft has announced that the original repository will be archived with no further updates, and its focus has moved to Project AirSim. That matters for any validation evidence you expect to maintain over a product’s lifecycle.

Our rule of thumb is this. Use CARLA when the ODD resembles a road scene with traffic agents and a rich sensor suite. Use Gazebo for ROS-centred ground robots and PX4 work. For aerial platforms, choose a tool with an actively maintained future. Isaac Sim faces the same test as the others: its sensor models enter the safety case only after the same comparison against your logs.

Do CARLA’s simulated pedestrians behave like people on your site?

CARLA pedestrian results count only after the domain gap has been measured against CCTV, field recordings and near-miss logs from the same ODD, and the scenario generator calibrated to match. TechnoLynx can build the pipelines that align site recordings and near-miss logs with CARLA scenario outputs and compare trajectory distributions against held-out recordings, with the AI Porting & Deployment Pack getting the calibrated generator running on your target compute.

Who can build a sim-to-real calibration pipeline for our robots?

TechnoLynx builds the engineering that makes simulation evidence defensible for robots, drones and autonomous machines. That covers log-replay pipelines, calibrated sensor and dynamics models, gap metrics a safety board can read, and routing of the scenarios simulation cannot carry to HIL and field tests. Building in-house makes sense if you already have simulation engineers who also understand your sensors and your logged data. A specialist helps when the calibration harness, the compute stack and the evidence format for the safety case all have to come together on a deadline. When choosing a partner, ask how they separate scenario mismatch from sensor-model error, and what they hand over to the safety board. Running the calibrated workload on edge or embedded targets falls under the AI Porting & Deployment Pack, and engineering for autonomous machines covers the wider work.

There are limits to what we do. We do not sign off safety cases: UL 4600 conformance is an independent assessment of the safety case, and the acceptance decision stays with your board and your assessor. A team with no recorded field data yet needs a data-collection campaign before calibration has anything to fit against. The other realistic routes are an in-house robotics simulation team, support from the simulator vendor for its own models, and an independent assessor for the conformance step.

The same split outside robotics: epidemic models

The calibrate-then-accept pattern also applies to an epidemic model checked against a recorded outbreak: compare its peak timing, peak size and spread with the record, and recalibrate the disease parameters where it misses.

Acceptance then rests with the public-health authority that will use the forecasts, on a documented protocol of back-testing against historical outbreaks.

Recalibration triggers belong in the safety case

A calibrated simulator only stays calibrated until the site, the sensor or the ODD moves. UL 4600 describes lifecycle monitoring to confirm claims that design-time evidence cannot adequately support, including assumptions. In the TÜV SÜD SOTIF workflow, an incident moves the boundary between known and unknown scenarios and can force design changes and a restart of verification and validation. In practice, the recalibration triggers (a new pit, a new LiDAR model, a new dust source, an unexplained field incident) should be written into the safety case next to the threshold they protect.

FAQ

What is the main difference between hardware-in-the-loop and software-in-the-loop simulation?

Software-in-the-loop should catch logic errors cheaply. HIL puts a real controller against a simulated plant to stress interfaces and timing before expensive hardware is exposed. In PX4, HITL runs standard firmware on real hardware, while SITL actually runs more of the standard system code.

What is sim-to-real transfer in robotics?

The sim2real gap is an umbrella term for the discrepancies between simulation and field deployment, including sensing and dynamics models. The authors of one ROS 2 drone framework argue that it also includes integration failures at the interfaces between software and hardware components.