Scenario-based testing serves as ISO 21448 SOTIF evidence when it forms one traceable chain. We would build that chain from a defined operational design domain (ODD), to scenario classes derived from the ODD and the hazard analysis, to per-class metrics with thresholds set from the residual-risk argument, to versioned results and a safety-case argument linked to each piece in both directions. ISO 21448:2022 defines SOTIF as the absence of unreasonable risk from hazards caused by functional insufficiencies, whether in the specification of the intended functionality or in the performance of the electronic elements that implement it. For a perception and SLAM pipeline, the evidence therefore has to show how detection and localisation behave in each condition. An aggregate score going up does not show that.

How ISO 21448 classifies scenarios, and what it leaves to you

SOTIF sorts operational scenarios along two axes: whether they are safe or unsafe, and whether their occurrence is known or unknown in advance. The goal is to maximise or maintain the safe area and minimise the unsafe area while keeping the function reasonably available. The standard expects both unsafe-known situations (such as inputs outside the ODD) and unsafe-unknown ones (such as samples outside the training distribution) to be reduced until the residual risk is acceptable. What is distinctive about SOTIF is that it deals with the unknown-unsafe region explicitly, through an iterative approach that reduces it as much as possible with an acceptable level of effort.

ISO 21448 applies where safe behaviour depends on situational awareness derived from complex sensors and processing algorithms, including emergency intervention systems and driving automation levels 1 to 5. That brings in camera-based functions such as AEB, ACC and lane keeping, which rely on machine-learning perception. For those functions, dependable machine learning that reduces false positives and false negatives already matters a great deal, even though the driver remains responsible.

The standard stops short on how to generate scenario suites. One analysis of scenario generation reads two requirements out of ISO 21448 for a scenario suite. The suite must sufficiently cover the scenarios that can occur in the ODD, and it must help minimise the set of hazardous scenarios. The same work points out that the standard does not say how to define a coverage criterion for the first requirement, or how to judge whether a scenario contributes to the second. A test engineer therefore cannot tell from the text alone whether a suite is good enough.

Two approaches from the research literature help close that gap. The first is structural: scenarios can be described at three abstraction levels as development proceeds. Functional scenarios use linguistic notation, logical scenarios use parameter ranges in a state space, and concrete scenarios give a specific value for every parameter. Concrete scenarios are what a test executes, so they must be reproducible, free of inconsistencies and machine-readable. The second is statistical. Sampling that prioritises scenarios by criticality gives statistical coverage and avoids the curse of dimensionality that comes with enumerating every combination at a fixed resolution. Parameter sampling with a feedback mechanism lets testers balance exploring the ODD against concentrating on likely-hazardous regions. The taxonomy and hazard catalogue that feed those levels are treated separately in our guide to building a defensible scenario taxonomy and safety case for SOTIF.

What pass/fail thresholds can you defend for occlusion, low-light and dynamic-obstacle scenarios?

We would treat a threshold for occlusion, low-light or dynamic-obstacle scenarios as defensible if it comes from your own hazard analysis and residual-risk acceptance for that class, is scored separately for detection and localisation, and has been calibrated against recordings known to be good or bad. A number borrowed from another lab brings that lab’s sensors and ODD with it.

TechnoLynx builds this per-scenario-class scoring and gating into a test harness through its Production AI Monitoring Harness engagement. The first step is to define the scenario classes with your team and agree what a pass means operationally in each, before any metric is chosen.

Two metric families, one verdict per class

For detection, the usual measures are mean average precision (mAP) and precision-recall per object class. Autoware’s perception evaluation in TIER IV’s Driving Log Replayer, for example, computes mAP and related indices from the recognition output. For localisation, the TUM RGB-D benchmark proposes two trajectory metrics with automatic evaluation tools. Relative pose error (RPE) measures the difference between estimated and true motion and suits drift evaluation. Absolute trajectory error (ATE) aligns the two trajectories and then compares absolute poses, which requires absolute ground truth.

A per-class verdict can use both families. We would combine them as a conjunctive gate, where detection must pass its class threshold and localisation must pass its own, rather than blending mAP and ATE into one weighted number. With a blended score, a localisation regression can hide behind a detection gain, and nobody reading the report can tell which subsystem moved. So the report carries the joint verdict with both component scores next to it.

How ROS 2 replay pipelines gate perception and localisation

Published replay evaluation for Autoware uses the same structure on a smaller scale. In TIER IV’s Driving Log Replayer, a perception frame succeeds when the share of true positives meets a criterion set per distance band. In the documented example that is 75% for objects within 0 to 50 m and 25% beyond 50 m. The scenario succeeds when the proportion of successful frames meets a configured pass rate. The localisation use case checks NDT convergence on three per-frame conditions: lateral distance within 0.2 m, execution time within 100 ms and at most 30 iterations. It passes when at least 95% of attempts converge. A run also fails if localisation diagnostics publish errors at too high a rate.

Two lessons carry over. In those pipelines, thresholds are set in the scenario configuration, not as one global number, and a SOTIF gate needs the same shape. The tool’s database evaluation mode averages results across datasets. That is convenient for tracking regressions, but averaging is exactly how a model passes overall while failing one class badly, so the per-class verdict has to be recorded before any averaging happens.

Per-class evidence and threshold sources

Scenario class	Detection evidence	Localisation evidence	Where the threshold comes from
Standard conditions	Per-class mAP and recall within distance bands	ATE and RPE against ground truth	Hazard analysis
Occlusion	Recall and false-negative rate per occlusion level	RPE through occluded segments	Residual-risk acceptance for missed objects behind occluders
Low light	Per-class mAP per illumination bin	ATE, RPE and convergence rate per illumination bin	Hazard analysis per bin, with the ODD boundary as a hard edge
Dynamic obstacles	False-negative rate on moving objects in path; track continuity	Pose stability when much of the scene is moving	Severity of the hazardous event

The decision logic behind the table is simple. We would give safety-critical dynamic-obstacle classes the tightest gates. A degraded-sensing class such as low light gets a looser gate only where the function design accepts degraded behaviour there, for example a warning and handover. The threshold is then calibrated: it should pass every known-good recording and fail every known-bad one, and it should be compared with published gating practice without adopting that practice unmodified. Calibration only means something if each class has enough recordings, and a class with a handful of clips produces a threshold that is mostly noise. A gate like this serves as release-gate evidence. Producing it is reliability engineering, which is a separate discipline from improving model accuracy.

What reviewers expect in the validation write-up and the coverage claim

We would treat two things as the minimum for scenario test results to count as approval-grade evidence. The write-up states the methodology, its assumptions and the pass/fail thresholds with their origin, and links every reported number to the raw results and tool versions that produced it. The coverage claim is measured against the ODD rather than counted in scenarios.

TechnoLynx can design that validation methodology and build the traceability layer that ties each claim in the report to the scored records behind it. The work starts with the thresholds, the strata the results must be broken down by, and the criteria the reviewer will apply.

The write-up: methodology, assumptions, thresholds and links to raw data

A paper on scenario-based safety assessment of automated driving systems says test results only have value in a safety case if the methods and tools used are themselves validated. It says documentation must cover the scenario database and selection tools, the scenario generation and sampling methods, proving-ground equipment, and the simulation and sensor models. On the reporting mechanics, one guide to ISO 26262 verification recommends checking tool output into source control and referencing it from the report rather than pasting it in. The report then summarises who ran the regression, when, and where the results and coverage reports are stored. For a perception pipeline, we would break every reported figure down at least by scenario class, sensor configuration and model version. Thin strata undermine any confidence level claimed for them.

SR 26-2, the US banking agencies’ revised model risk guidance, offers a comparison: internal audit generally does not duplicate validation but evaluates whether model risk practices are rigorous and effective.

A scenario coverage claim measured against the ODD

ISO 34503 makes the ODD definition an input to the scenario-based safety evaluation framework of ISO 34502. It also requires a comparison between the test scenario set and the ODD definition to analyse test-space coverage. Testing should cover correct operation inside the ODD, behaviour across its boundary, and responses outside it, such as rejecting attempts to engage. The ODD taxonomy has three top-level attributes (scenery elements, environmental conditions and dynamic elements), and each attribute needs an objective measurement. Granularity matters as well. If a system is designed for light rain only, below 2.5 mm/h, the standard says the ODD should be specified at that granularity.

A purchased or licensed scenario library fits this framework rather than replacing it. Scenario databases such as SAFETYPOOL, STREETWISE, ADSCENE and SCENARIO.CENTER store scenarios that do not depend on the ADS technology. Mapping a licensed library onto your ODD attribute cells shows which cells it fills and which it leaves empty. The empty cells still need their own scenarios or a written justification. The SYNERGIES project described by TNO is developing metrics for the representativeness and ODD coverage of a selected scenario set, along with tools that record the search and its metadata so the selection can be reconstructed later. The same paper frames the evidence as a test matrix per requirement, with a pass or fail for each test case and an account of how the matrix was compiled, including its coverage of the ODD.

Traceability from scenario to safety case, in both directions

Bi-directional traceability means every requirement can be shown to be verified somewhere, and every verification step can be shown to support one or more requirements. For SOTIF, we would run the chain from ODD attribute to triggering condition, then to safety goal, scenario class, concrete scenarios, test result, and finally the claim in the safety case. Every link should be walkable in both directions. A safety case is a structured argument backed by traceable evidence, not a collection of documents. It is built from claims, the argument for why each claim follows, and the evidence that supports the argument. Goal Structuring Notation makes gaps visible, such as a goal without evidence or evidence without a reference, before an assessor finds them. Trace links that point at current work products show which parts of the argument need re-evaluating when an artefact changes. How that chain is packaged for a functional safety sign-off is covered in building a defensible ADAS validation evidence package for functional safety sign-off.

How should SOTIF test evidence be recorded and versioned to survive an IATF 16949 audit?

We would build SOTIF test evidence for an IATF 16949 audit as a structured, version-controlled package generated by the pipeline that produced the results. The package would record who assessed what, when, how and against which criteria, with approver sign-off and a stable identifier that cross-references the PPAP submission. Confirm each customer’s specific requirements before fixing the record’s final shape.

TechnoLynx designs and implements automated evidence-pack generation inside the existing test pipeline as part of its AI governance and trust work. It starts by identifying which configurations, logs, metrics and approvals make up the assessment, and which PPAP cross-references the record needs.

A functional safety checklist for automotive quality systems from Quality Support Group calls data management critical, naming configuration management, traceability, cataloguing of artefacts and keeping their status current. In a perception context, that translates into a record with these contents:

• the scenario catalogue version and the ODD definition version it was compared against;
• model, dataset and sensor-calibration versions;
• the harness code commit and container image used for the run;
• the thresholds file version, with a reference to the hazard-analysis rationale for each threshold;
• the location of raw per-frame results, plus the summary derived from them;
• approver identity, timestamp and the part or PPAP identifier.

The pack should be generated, not reconstructed. Traceability added afterwards to data collected without consistent metadata is unreliable, and generation only preserves traceability if the pipeline’s artefacts are themselves immutable and versioned. We would let assessment volume decide the mechanism. For a low-volume, infrequent assessment, a templated manual record with sign-off can be enough. Recurring or pipeline-driven assessments call for automated generation tied to the run.

Bosch’s supplier quality requirements classify test processes by kappa: 0.9 or above is capable, 0.7 to 0.9 is conditionally capable, and below 0.7 is not capable. We would treat a perception pass/fail gate as a test process in that sense. We would show that it gives the same verdict on repeated runs of the same recordings. Whether a generated package is accepted is still not guaranteed. A mock audit or an early conversation with the auditor is cheaper than finding out during the surveillance audit.

How do you show a detector’s degradation mode under fog, flare or occlusion?

We would show a detector’s degradation mode by scoring detection and fusion separately for each scenario class and severity level, plotting the degradation curve, and breaking the misses down by condition, object class and distance. Diagnostic analysis then explains the dominant failure patterns. An overall mAP only shows that failures happen. A curve per condition shows where they start and how fast performance falls.

LynxBenchAI scores detection and fusion for each scenario class and severity level and reports the degradation curve per condition, and the first step is to agree those classes and severity levels with your team.

TechnoLynx’s first step is to agree the scenario classes, metrics and report format with your functional safety process.

Benchmarks of deep network robustness show unexpected prediction errors on simple perturbations. Natural corruptions from the camera lens, snow, rain and fog need robustness work of their own. In the rain-rendering evaluation discussed below, some algorithms’ performance curves cross as rain increases, so a model that ranks near the top in clear weather can rank last in a storm. A single score averaged across conditions cannot show a crossing like that.

Reproducibility makes the curve evidence and not just an anecdote. We would have the harness fix the dataset splits, scenario taxonomy, random seeds and evaluation code, and version all of them, so a score can be regenerated and compared across model versions. Misses also need to be traceable to frames. Driving Log Replayer’s output, for instance, lists the frame numbers evaluated as true positive, false positive, true negative and false negative. That list lets an engineer go from a recall drop in fog to the specific frames where it happened.

Explaining why those frames failed takes more than metrics. We would use error attribution, failure clustering and saliency analysis to turn a miss list into a statement about cause. For sensor fusion, we would add indicators that measure correlated failure. Koopman’s summary of UL 4600 gives examples of safety performance indicators, including a correlated camera and lidar false-negative rate and a tolerance for gaps in lane markings. A fused stack is only as robust as the conditions in which both sensors fail together, so correlated misses deserve their own row in the report.

Two limits apply. What counts as fog, flare or occlusion has to be defined and agreed with the reviewers before the numbers mean anything, and a harness that scores robustness is one input to the SOTIF argument, not proof of safety. A practical catalogue of the edge cases such a harness should exercise is in what edge cases a perception robustness audit actually covers.

Who can build a SOTIF scenario test harness and evidence pipeline for our perception stack?

TechnoLynx builds the scenario-class scoring and thresholding logic into a test harness. On building in-house versus bringing in a specialist: the team that owns the perception stack knows its failure modes best. The harness engineering (reproducible evaluation, versioned scenarios and thresholds, generated evidence packs) is where we would expect outside help to shorten the path. When choosing a partner, ask how they stop per-class verdicts from being averaged away and how a reported number traces back to individual frames. The engagement is the Production AI Monitoring Harness, and it starts by defining scenario classes and pass criteria. It sits within TechnoLynx’s wider work on AI engineering for manufacturing and automotive.

The engagement has clear limits. It is engineering, not safety certification or regulatory sign-off. The safety assessment itself is a judgement by a qualified independent body, internal or external, which recommends acceptance, conditional acceptance or rejection. A company’s functional safety process may also be qualified by an independent body such as TÜV, DNV or UL.

What rain conditions does an LDWS regression suite need to cover, and can synthetic rain fill the gaps?

We would build an LDWS rain regression suite from rain-intensity bins in mm/h taken from the ODD and hazard analysis, crossed with the lane-marking degradation conditions the hazard analysis names. It should use real footage where you have it and synthetic rain only where the generation has been checked against real footage. We would set its thresholds per bin, and freeze and rerun the whole suite before every release. The acceptable false-negative rate is not a universal number. We would derive it from the residual-risk argument, since the SOTIF verification and validation strategy must show residual risk below the acceptable level.

TechnoLynx can turn rain and visibility triggering conditions from your item definition and hazard analysis into a structured scenario taxonomy, and implement the suite as an automated pre-release gate scored per bin.

Choosing the intensity bins

Rainfall intensity is measured as the average fall over an hour. As a reference, moderate rain is roughly 10 mm/h, heavy rain is above 50 mm/h, and in non-tropical regions rates of 200 mm/h or more usually last only a few minutes. Halder et al. (ICCV 2019) rendered rain at 0, 1, 5, 17, 25, 50, 100 and 200 mm/h for their evaluation. ISO 34503 shows how an ODD can draw the line, with a light-rain-only system specified below 2.5 mm/h, and allows boundaries with temporal qualifiers such as up to 2 minutes of heavy rainfall. We would place one bin just inside the ODD rain boundary and one just outside it, because those two bins test whether the system knows when to hand over.

What the rain-rendering studies measured

The extended 2020 version of the physics-based rain-rendering work can generate rain at a known, calibrated rate from 5 mm/h up to beyond 200 mm/h. On real nuScenes images, with weather labels cross-checked against a historical weather database by GPS location and time, object detection mAP was 16.30% in rain against 32.53% in clear conditions. On synthetic rain at 200 mm/h, detection was always at least 12% worse than in clear weather. Fine-tuning on synthetic rain improved object detection on real rainy scenes by 20.7% with physics-based rendering and 21.0% with the hybrid GAN and physics approach.

These are object detection results on public datasets, so they show how to measure rain degradation. They are not a prediction for lane-marking detection on your camera, since the studies measured object detection, segmentation and depth estimation.

Checking synthetic rain against real footage

Synthetic rain can fill gaps at rare intensities, but only after its generation has been checked against real footage in the bins it is meant to fill. That means comparing per-bin detection behaviour on real and rendered rain, not just whether the images look plausible. Physical rain rigs need the same care. A paper on the credibility of simulated rainfall in perception tests cites ROADVIEW project calibration results from the CARISSMA outdoor rain field and the Cerema PAVIN fog and rain platform, showing clear deviations between set and measured rainfall intensity and non-uniform rain. A nominal intensity label on a test run is a setting, not a measurement.

The pre-release rain gate, in order

1. Derive the intensity bins and lane-marking conditions (worn, wet-reflective, partially covered) from the hazard analysis and the ODD rain attributes.
2. Fill each bin with real footage where it exists and with validated synthetic rain where it does not, recording which is which.
3. Set a threshold per bin from the residual-risk argument and record its rationale next to the hazard it addresses.
4. Freeze the catalogue, thresholds and harness version under one tag.
5. Run the suite before every release, score every bin, and fail the release if any bin fails, whatever the aggregate shows.
6. Change the catalogue only when the ODD or hazard analysis changes, and version that change like code.

A suite that passed last release says nothing about a condition nobody put in the catalogue.

FAQ

What is a false negative rate?

The false negative rate is the share of real objects or events that a system fails to detect: false negatives divided by the sum of false negatives and true positives.