Test scenario coverage analysis measures which parts of a declared scenario space your test set has actually exercised, and reports that as a ratio against a denominator you have written down. For automated driving, three ISO documents supply candidate denominators. ISO 34503 defines the operational design domain (ODD) attributes under three headings: scenery elements, environmental conditions and dynamic elements. ISO 34504 defines scenario categories through tags. ISO 34502 covers identifying hazardous scenarios and deriving a representative set of test scenarios that argues sufficient coverage of the relevant scenario space. A coverage percentage only holds up when the taxonomy, the discretisation and the rule for what counts as “covered” are published alongside it.

How is scenario coverage different from code coverage?

The ISTQB glossary defines coverage as the degree, expressed as a percentage, to which a specified coverage item has been exercised by a test suite. It defines code coverage as the analysis of which parts of the software the test suite executed. What separates the two is the coverage item. For code coverage the item is a statement, a decision or a condition. For scenario coverage it is a scenario class or category.

Hawkins and Alexander point out that testing until every code path has run will not necessarily find all faults, because many faults take the form “fails to deal with situation X”, and no software component may ever have been designed to deal with situation X.

ISTQB also defines coverage analysis as measuring achieved coverage against predetermined criteria, to decide whether more testing is needed and which test cases it should include. A coverage figure exists to tell you what to test next, not to decorate a slide.

Should coverage count test scenarios or test cases?

Count scenarios for coverage, and count only scenarios that ran inside a test case with a real evaluation criterion. The automotive literature describes scenarios at three levels of abstraction. Functional scenarios are written in human-readable language. Logical scenarios describe the same situation as parameter ranges in a state space. Concrete scenarios fix a specific value for every parameter. In scenario-based approaches, a test case consists of at least one concrete scenario plus one or more evaluation criteria. Steimle and colleagues argue that the scenario itself should not contain those criteria.

Two consequences follow. First, only concrete scenarios can be turned directly into test cases and executed. A logical scenario with continuous ranges can produce any number of concrete scenarios. So one concrete run does not “cover” a logical scenario in any strong sense. It covers one point, or one cell once you discretise. Second, a run with no meaningful pass/fail criterion is weak evidence. A 2026 guide to coverage gap analysis puts it plainly: a linked test with no meaningful assertion may be partial, not covered. The coverage unit is the scenario category. The evidence that fills a category is an executed test case.

Mapping a test set onto ISO 34502, 34503 and 34504 categories

Finding which taxonomy categories your current test set never touches is a set-membership exercise. It only works if the taxonomy is fixed before you look at the logs. Under ISO 34504, a tag is a meta-attribute attached to a scenario, and a scenario belongs to a scenario category if every tag of that category also applies to the scenario. Tags cover dynamic entities, scenery elements, environmental conditions, additional scenario information and intended test usage. They are arranged in hierarchical trees, and stakeholders may extend them as long as the extensions do not conflict with the specified tags. ISO 34504 itself notes that relating an ODD to scenario categories can be easier than relating it to every possible scenario.

The steps run in this order:

1. Declare the ODD with the ISO 34503 attributes, at the granularity the system really has. The standard’s own example: if the system handles only light rain (below 2.5 mm/h), the ODD should be specified at that level of detail.
2. Derive the tag set from that ODD. ISO 34503 ODD definitions can use the tags listed in ISO 34504.
3. Define scenario categories as tag collections.
4. Tag every executed test or log segment, then apply the membership rule to assign it to categories.
5. Count scenarios per tag and category, N(L, C). Every zero count is a tag-category combination with no scenario.
6. Classify each scenario as inside, at the boundary of, or outside the ODD. ISO 34503 says testing should cover correct operation within the ODD, behaviour across its boundary, and responses outside it, for example refusing to engage.

ISO 34503 requires this comparison explicitly: a test scenario set must be compared with the ODD definition to analyse test space coverage. If the ODD itself is unsettled, settle it first. Our piece on building auditable ODD specifications covers that step, because the ODD becomes the denominator of every figure that follows.

Mapping recorded data onto ISO 34502 needs one caution. ISO 34502 builds critical scenarios from three groups of risk factors: perception disturbances, traffic disturbances and vehicle-control disturbances. For general vehicles on highways, combining three components gives 24 traffic disturbance scenarios. A formalisation study tested those descriptions on the highD dataset and found two problems. Read strictly, they missed a significant number of disturbance traces. They were also not adequate for classification, which the authors expect, since the set was designed for comprehensive coverage rather than for classifying recordings. Before you classify, write down how you interpret each ISO 34502 description. Your coverage figure depends on that interpretation as much as on the logs.

How do you turn “we tested on past logs” into a defensible coverage percentage?

The answer is to compute named metrics against named inputs, since no single percentage represents every coverage relationship. The paper on coverage metrics for scenario databases separates two questions. Type I asks whether the collected scenarios cover all relevant aspects of the ODD. Type II asks whether they cover all relevant aspects present in the driving data. Its tag-based coverage, CoverageTag(n), answers Type I. For every combination of tag and scenario category it counts the scenarios, caps each count at n, and divides the total by n times the number of tags times the number of categories. Time-based, actor-based and actor-over-time coverage answer Type II.

Type II coverage can point to scenario categories you may have missed. In that paper’s example, about 75% of time instants were covered by at least one scenario. In the remaining 25%, no actor matched any category description or there was no other actor, and adding an “ego vehicle has no leading vehicle” category would bring time coverage to 1.

Worked example: one set of logs, three defensible numbers

Assume a feature tree with three binary features: turning left or driving straight, oncoming traffic or none, a crossing pedestrian or none. That gives 2³ = 8 possible scenario classes. If the logs contain only one of them, scenario class coverage is 1/8, which is 12.5% (arithmetic on that definition).

Now take CoverageTag with ten cut-in scenarios that carry the tag “actor at left”. With n = 1 that cell contributes fully, since min(1, 10) / 1 = 1. With n = 20 it contributes only half, since min(20, 10) / 20 = 0.5 (arithmetic on the published formula). The data has not changed. Only the evidence threshold per cell has.

Changing the classifier projection changes the figure too. In the tree-based classification study, run on simulated CARLA drives, the layer 1+2 projection reached 100% of its scenario classes, the full classifier only 26%. Every such figure is a statement about a declared discretisation of a declared space, and the scenariocov tool states the point directly: complete coverage of that space is not evidence that the system is correct.

The same logs can also give high coverage with poor completeness. If actor trajectories are missing from the data, coverage can still look high while important information is absent, so completeness has to be checked separately. Report each percentage with its tag set, its n, its projection and a completeness check. A bare percentage will not survive the first question.

How much confidence does each covered cell carry?

Coverage does not give you confidence on its own. Coverage records whether a class was observed, while scenario instance count records how often. A cell filled by one run and a cell filled by two hundred runs both count as covered. The coverage-metrics paper notes that n may need to be raised above 1 to get more accurate statistics, and that choosing n calls for other methods.

Report the per-cell count next to the per-cell result, and treat thinly populated cells as weak evidence, not as passes. Avoid presenting a resampling score as a probability. The scenariocov tool says its confidence output is not a calibrated probability: it is the fraction of bootstrap resamples that place a candidate in the sparsest quantile, and it measures sensitivity to the executed sample and nothing else. Choosing the interval method for small samples is a separate problem, covered in our piece on statistically defensible validation under limited real-world data.

There is one more caveat about what coverage proves at all. In Hawkins and Alexander’s study over a 216-cell situation space, coverage-guided generation filled about 80% of cells against 36% for random generation with the same compute. Yet the correlation between situation coverage and runs that found faults was poor, with r² = 0.1681. Higher coverage tells you more of the space was visited. It does not, on that evidence, tell you more faults were found.

Recording a gap versus a compensating control

A control standing in for an untested scenario needs to cover the same population, the same period and the same risk scenario. Where it is detective rather than preventive, record the timing gap as well. One guide on compensating controls in security audit names three dimensions where substitution falls short: population (narrower scope), period (runs less often, leaving unmitigated windows) and response latency (detective rather than preventive). It adds that whether a shortfall is tolerable depends on the risk scenario and operating context, not on the fact that the shortfall exists. It changes when protection happens, and the record should say so.

Before you accept a compensating control, check four things:

• Equivalence: does it cover the same population, period and risk scenario as the missing test?
• Independence: does it share an upstream dependency (a data feed, a sensor, a process owner) with the thing it replaces? If so, both fail together.
• Currency: a test result shows the control worked during the tested period, not that it still works. The retest interval belongs in the acceptance record before approval.
• Explicit acceptance: an accepted gap records the missing evidence, risk rationale, compensating control, owner, approval, review or expiry date, and escalation condition.

The 2026 coverage-gap guide makes the last point well: acceptance is an explicit decision, not an empty matrix cell. Define the states up front (covered, partial, unknown, accepted gap, not applicable) so that the matrix can tell “nobody looked” apart from “we looked and accepted it”.

What a test scenario coverage analysis report should contain

The report is the part a reviewer reads, so each element should answer a question they would otherwise ask. The structure below draws on the post-campaign report of the open-source scenariocov tool and on the ISO 34503 boundary requirement.

Report element	What it states	Why a reviewer needs it
Declared space	Tag set, scenario categories and n per cell	Without it the percentage has no denominator.
Discretisation	Bin edges or levels per parameter, stated as an assumption	Coverage is a statement about a declared discretisation.
Occupancy	Cells filled out of total, plus the largest empty block	Shows where gaps cluster, not just how many there are.
Uncovered classes and tuples	Every missed class and feature pair, by name	Gaps are named, not summarised.
Boundary coverage	Each boundary hit or missed, with the nearest approach and margin in parameter units	A hit flag is a floating-point comparison. Read the margin.
ODD position	Scenarios inside, at the boundary of, and outside the ODD	ISO 34503 expects all three to be tested.
Per-cell counts	Instance count behind each covered cell	Separates one run from many.
Gap register	Accepted gaps with owner, rationale, compensating control, expiry and escalation	Turns empty cells into decisions.
Completeness check	Missing trajectories or data omissions	High coverage on incomplete data misleads.

We treat this report as an engineering output, not a policy document. Where it goes into a larger submission, our guide to building a defensible evidence pack for audit and procurement review covers the pack around it.

The same coverage matrix outside automated driving

The method carries over to other domains because coverage is a relationship between an obligation and evidence. The obligation can be a requirement, a risk or a regulatory control, and the evidence can be a test, a monitor or a review. What changes from domain to domain is the external taxonomy, and choosing it correctly is a domain decision.

A FATF typology coverage matrix for a transaction-monitoring rule book

For a bank’s transaction-monitoring rule book, the obligations are the typologies it must detect and the evidence is the rule that would detect each one. The structure is the same as above: an identifier, source, owner and acceptance rule for every obligation, and an explicit state for each cell. Deciding which FATF reports form the canonical source list, and what the model-risk function will accept at examination, is a compliance judgement. We would not make it from engineering evidence alone.

A crosswalk from radar-camera fusion logs to a child-presence detection matrix

Where a draft protocol defines its scenario matrix as a product of axes, the crosswalk maps fields in your log schema onto those axes. Cells with no runs are zero-coverage, and cells with few runs carry weak evidence. Where a field does not map cleanly onto an axis, document that ambiguity as an interpretation, just as with the ISO 34502 descriptions.

Presenting “what I tested and what I didn’t” to a defence committee

Presenting coverage to a capstone or thesis committee is a different job from producing the analysis. The analysis supplies the material, but deciding how to present it is a matter of academic format. The parts that carry over are the honest ones: name the declared space, show the uncovered classes by name, and state that the figure depends on your discretisation.

Which scenario catalogue should a rain robustness matrix be mapped to?

Encode the concrete rain runs in ASAM OpenSCENARIO so other labs can replay them. TechnoLynx can define the rain parameter space (rain rate, visibility, lane-marking contrast), encode it in OpenSCENARIO, map each severity level onto the ISO taxonomy with traceability documentation, and run the suite on your simulation or sensor-in-the-loop stack. The first step is framing the problem against the lane-detection requirement and the ODD boundaries. This is part of our production AI reliability work, which includes release-readiness reviews.

If you only need internal reproducibility, OpenSCENARIO alone may be enough. Recognition by other institutes or OEMs also needs the taxonomy mapping. Two limits remain. Neither standard guarantees that a specific OEM will accept your matrix, and mapping severity bands onto attributes is an interpretive step that neither standard fully prescribes, so it has to be documented and justified.

Limits that no coverage figure removes

Even a perfectly reported figure has limits. Reaching CoverageTag = 1 is not always possible and may not be necessary: some scenario categories hardly ever occur with certain road layouts, such as a cut-in on a single-lane road. Achieving high Type I and Type II coverage together is difficult, and the authors leave appropriate thresholds as context-dependent future work. Any threshold you set is therefore your argument, and it should be stated as one.

The harder limit is the denominator itself. ISO 34503 distinguishes the ODD from the target operational domain (TOD), the real-world conditions the system will actually meet. The TOD is often a superset of the ODD, and in all practical cases an ODD definition will not be exhaustive enough to cover it. How far the ODD covers the TOD is still an open research question. An honest report says that too, and points to the fallback behaviour that handles an ODD exit.