The safety standards leave pass/fail details of perception testing open, so robustness evidence scoring cannot work as a lookup. UL 4600 sets no pass/fail criteria, and the automotive safety standards mostly leave the design of perception tests to the manufacturer. The insurer has to build the rubric itself. Anchor it in ISO 21448 (SOTIF) for performance limitations, add ISO/PAS 8800 where the perception uses machine learning, and structure the scenarios with ISO 34502. Then write down why each threshold sits where it does, and withhold full credit until an independent party has re-run part of the evidence.
Which standards should we score third-party ADAS perception evidence against?
Score the evidence against SOTIF (ISO 21448) as the primary frame. Layer ISO/PAS 8800 on top where the kit relies on learned models, and map the vendor’s tests onto a scenario catalogue built the ISO 34502 way. The standards require safety-case arguments about perception but leave the tests open.
TechnoLynx can build that risk-tiered rubric with an insurer. TechnoLynx would start by assessing which standards apply to the specific kit, then map the vendor’s scenario-based test evidence onto the chosen catalogue and risk tiers to produce the pass/fail scorecard, and finally document the applicability assessment and scoring logic so a reinsurer can follow it. TechnoLynx delivers that documentation as an engineering output a non-engineer underwriter or reinsurer can read and sign against.
The perception frame: SOTIF first, ISO/PAS 8800 for the learned parts
ISO 21448 names the performance limitations behind hazardous behaviour in sensing-based systems. Hazardous behaviour can come from a system that is free of ISO 26262 faults but cannot correctly understand the situation, including functions that use machine learning, or that lacks robustness to sensor input variations and diverse environmental conditions. The standard defines functional safety and SOTIF as distinct and complementary aspects of safety. ISO 21448:2022 applies where situational awareness comes from complex sensors and processing algorithms, across driving automation levels 1 to 5.
ISO/PAS 8800 covers the risk of unsafe vehicle-level behaviour caused by output insufficiencies, systematic errors and random hardware errors in AI elements. It is meant to complement ISO 26262 and ISO 21448, not replace them, and its guidelines apply whenever a safety-critical component includes an AI model. This matters for retrofit kits. A perception component rated QM under functional safety analysis can still introduce SOTIF hazards, and one analysis argues that AI-based QM components need a full SOTIF assessment even if their FuSa rating is unchanged.
Aftermarket scope is where the rubric needs an explicit rationale. ISO 21448:2022 is written for E/E systems installed in series-production road vehicles. ISO/PAS 8800 excludes the development of AI elements that are not part of the vehicle, though it can serve as a reference where no domain-specific guidance exists. A bolt-on kit sits awkwardly between those boundaries. The scorecard should therefore say in writing why the insurer applies these standards by analogy, rather than assume the vendor is formally in scope.
The clinical parallel: what AI medical device evidence is benchmarked against
The same logic applies to clinical AI submissions, only with a regulator already behind the frame. In the EU, MDCG 2019-11 sets the qualification and classification criteria for medical device software under the MDR and IVDR. Under its Rule 11, software that informs diagnostic or therapeutic decisions is class IIa by default. It rises to IIb where those decisions may cause serious deterioration or surgery, and to class III where they may cause death or irreversible deterioration. The guidance also refers to the IMDRF risk framework, which categorises software by how significant its information is to the healthcare decision and by the patient’s condition.
FDA’s CADe guidance describes the multiple reader multiple case (MRMC) design as the typical clinical performance method for CADe devices: clinicians read cases unaided and aided by the device. The same guidance accepts a direct comparison with a predicate CADe device as one possible control arm for substantial equivalence. Predicate comparison works as context alongside the named rubric. It does not replace it. FDA’s guidance on AI-enabled device software functions calls for clinically justified pre-specified success criteria, test data kept apart from development, subgroup performance analysis and post-market monitoring. Those four elements carry over into an ADAS rubric.
Does ISO 26262 ASIL decomposition apply to an aftermarket ADAS retrofit kit?
Not as the main frame. ISO 26262 targets systematic and random hardware and software failures. It was not designed for the performance limitations and functional insufficiencies typical of AI perception, which is the gap SOTIF was created to fill. SOTIF hazards arise without any component breaking, for instance when a camera with limited contrast detects a dark car too late in poor light. ISO 21448 is built to analyse that kind of performance limitation together with its triggering conditions.
ISO 26262 should not be excluded by reflex, though. Where the kit interfaces with safety-critical vehicle functions such as braking actuation, the rubric should check those integration points against ISO 26262 explicitly. Our piece on functional safety in automotive perception covers how the two standards divide the work on a production system.
A robustness evidence scoring scorecard underwriters can defend
A scorecard for non-engineers works when each row asks one question the vendor’s report can answer with yes or no, and every “no” has a consequence decided in advance. The perception testing literature describes a qualitative criterion becoming pass/fail by setting an interval on a related metric. AMLAS warns against over-reliance on a single performance measure for an ML component. Its classifier example names the trade-off between false positives and false negatives as one to define. The rubric below follows both points.
| Rubric row | What the vendor evidence must show | Pass | Fail or cap |
|---|---|---|---|
| Applicability frame | Written scoping of SOTIF, ISO/PAS 8800, ISO 34502 and any ISO 26262 integration points for this kit’s architecture | Rationale documented and consistent with the kit’s sensors and integration | No rationale: scoring does not proceed |
| Scenario coverage | Each test mapped to a catalogue class, from functional to logical to concrete scenario | Every in-ODD class the insurer lists has tests behind it | Uncovered class flagged; tier capped |
| Per-class thresholds | A pre-specified metric interval per class, covering false negatives and false positives separately | Meets the insurer’s documented threshold in every safety-relevant class | A failure in any safety-relevant class fails that class, whatever the average |
| Test independence | Test data separate from training data; verification data hidden from developers | Separation and data access documented | Undocumented: treated as self-reported |
| Independent re-run | A third party re-runs a sample of scenarios on an equivalent setup | Results reproduce within the agreed tolerance | No re-run: tier cannot exceed the ceiling |
| Tool validity | Simulation, replay and proving-ground tools validated | Validation evidence supplied | Unvalidated tools: their results discounted |
| Version match | The tested model and software version is the one deployed | Versions match | Mismatch: rescore before renewal |
The rubric uses three tiers. The top tier needs every row passed, including the independent re-run. The middle tier allows full self-reported evidence with no independent sample, and it is capped regardless of how good the numbers look. Anything with an uncovered safety-relevant class or a missing applicability rationale goes to resubmission. The rows and tiers are this rubric’s own design; two rows rest on published sources. AMLAS requires verification to be sufficiently independent of development, with a log showing that verification data was not exposed to the development team. An operationalisation of the UNECE assessment approach notes that test evidence only has value when the tools and models that produced it have been validated.
Simplifying to a scorecard loses nuance, and edge-case kits will occasionally land in the wrong tier. The cut-points should be recalibrated against observed claims outcomes, with minimum auditor qualifications so that a light-touch audit cannot inflate a tier. How deep each row’s validation needs to go, and how the artefacts get packaged, is covered in scoping validation depth from risk tiering to evidence packs.
Is there an industry-standard pass rate per scenario class we can benchmark an OEM against?
No. ISO 34502 gives the structure for scenario-based safety evaluation but takes its safety test objectives from ISO 21448 or regulation, leaving the rate itself open. TechnoLynx uses its AI Readiness Scorecard to score the programme against a named published rubric with an evidence trail, so the scoring step has something fixed to measure against.
What ISO 34502 does and does not settle
ISO 34502 conforms to SOTIF and adds detail to it through a scenario-based process. That process identifies risk factors and the critical scenarios that follow from them. It builds on ISO 21448:2022 Clause 7 by structuring reasonably foreseeable risk factors into a scenario catalogue for testing, with the aim of maximising coverage of known hazardous scenarios. It then supports Clause 9 by defining the concrete scenarios to test and the platforms to test them on, and it explores unknown hazardous scenarios by varying the attributes of known ones.
On thresholds, the standard specifies the form of an objective but leaves the value open. A safety test objective can be stated as a boundary on an acceptable occurrence rate, for example that the perception element forwards incorrectly perceived objects less than once per y hours of operation in the operational domain. The objectives come from ISO 21448 or a related regulation rather than from ISO 34502 itself. The y is left as a placeholder. ISO 34502 also scopes itself to automated driving systems on limited-access highways. For a Level 2 ADAS kit, it is a structural reference the insurer adopts, since its guidance targets ADS.
Where the insurer’s own thresholds should come from
AMLAS lists three sources for risk acceptance criteria: comparison with an existing system, existing standards for safety-critical systems, and the views of domain experts and users. ISO 21448 frames acceptance through principles such as GAMAB, positive risk balance, ALARP and minimal endogenous mortality. These principles aim to keep residual risk no greater than that of a competent human driver or the incumbent system. Those global criteria are then broken down into scenario-class validation targets that can be measured in simulation, on a track or on the road. Regulators face the same problem. The EU’s type-approval rules require ADS to be free from “unreasonable risk” against a competent and careful human driver, and turning that into quantitative acceptance criteria remains open methodological work.
On the actuarial side, LexisNexis Risk Solutions reports from a sample of 11 million vehicles from model years 2014 to 2019 that ADAS features reduce loss cost by coverage. The company presents this as grounds for feature-based discounts. That is a vendor white paper about features on newer model vehicles, so it supports the case for discounting in principle. It says nothing about where a retrofit kit’s thresholds should sit.
Once set, a threshold has to be applied the same way to every OEM and vendor the insurer underwrites. Bespoke bars that move from deal to deal are easy to challenge as arbitrary if they are ever disclosed. Two caveats belong in the rubric’s own documentation. The rubric should map ISO 34502’s categories explicitly onto the insurer’s own classes, such as lane-keeping or oncoming-traffic detection. And the rubric should not compare raw pass rates across OEMs whose scenario libraries and severity definitions differ. The standards landscape is also moving, so a published threshold standard could appear later and would replace the insurer’s own numbers.
Were the vendor’s pass/fail thresholds set independently, and can a third party re-run the tests?
Treat any threshold the OEM or vendor set for itself as self-reported until two things hold: it traces to a published framework, and an independent party has re-run a sample of the scenarios. Third-party auditing exists to address the incentive problem in self-reporting, provided the auditor has enough information about the system’s activities. TechnoLynx can design and run that independent audit. TechnoLynx would compare the vendor’s stated thresholds against the ISO 34502 framework to find gaps, sample a subset of the claimed test cases, re-run them on equivalent target hardware or simulation, and document every discrepancy against the reported pass/fail outcomes. Where the re-run needs hardware comparable to the vendor’s edge platform, LynxBenchAI measures throughput, latency and memory across accelerators. Those results help choose an equivalent device. They do not stand in for re-running the vendor’s scenarios.
LynxBenchAI measures throughput, latency and memory across accelerators to find a device equivalent to the vendor’s edge platform for the re-run, and the first step is naming that platform.
These credibility checks apply before a vendor’s numbers enter the scorecard.
- Threshold provenance: each per-class threshold names its source (a standard, a regulation, an incumbent-system comparison) or is labelled as the vendor’s choice.
- Scenario definitions in a portable format: ASAM OpenSCENARIO 2.0 is designed to support ISO 34502 scenario-based verification, and its files are meant to run unchanged on different compliant toolchains.
- Fail labels defined up front: in OpenSCENARIO’s SOTIF workflow, monitoring conditions are derived from the hazard under study, and a concrete scenario that triggers one is labelled as failed.
- Approved scenarios only: the OpenSCENARIO concepts separate scenario reviewers from approvers. Unapproved scenarios can be used for preliminary evaluation but not for certification or production purposes.
- Data separation: the perception testing literature requires concrete test scenarios and data to be kept apart from those used in development, and reference data quality to fit the validation objective.
- A test matrix per requirement: the safety case structure described for the UNECE NATM approach lists each test against its requirement with a pass or fail result.
An independent re-run has limits. It is only as good as the auditor’s access to the same test environment and scenario definitions the vendor used. Alignment with ISO 34502 also leaves the implementer some threshold-setting discretion, so it reduces self-selection risk rather than removing it. Where the industry has no agreed threshold for a class at all, the scorecard should record that as an open gap, not as a hidden vendor choice.
Who can audit an ADAS vendor’s perception evidence for an insurer?
TechnoLynx can build the automotive AI engineering side of this for an insurer. TechnoLynx defines the risk-tiered rubric and the rationale for each threshold, maps the vendor’s scenario evidence onto a catalogue, and runs an independent sample re-run of the claimed tests. On the build-or-buy question: an in-house team can maintain the scorecard once it exists, but the first mapping and the re-run need people who read SOTIF analyses and scenario files fluently. To choose a partner, ask whether they can re-execute scenarios, not just review reports. The engagement starts by fixing the criteria before any evidence is scored.
This is not safety certification, type approval or legal sign-off. We produce the artefacts an audit needs, not the verdict.
What a scorecard cannot certify
Scenario coverage bounds what was tested. SOTIF lists scenario factors as a starting point but gives no concrete method for finding novel triggering conditions. The perception testing literature also points out that the open, changing real world makes any operational design domain description and its test coverage incomplete. So the most useful thing an underwriter can add to the rubric is a re-scoring trigger, not a tighter threshold. A model update, a sensor swap or a new camera revision should send the kit back through the version-match and re-run rows before renewal, because those are the changes that quietly invalidate the evidence the discount was priced on.
