A release reviewer signs a statement, not a test suite. That single fact decides who owns the section headings in a perception validation package — and if the headings come out of the test-tracking tool, the reviewer has to rebuild the argument themselves before they can sign anything.
The naive package is an export. Suite names, ticket IDs, pass/fail counts per regression run, a metrics table, an open-defect list. It is honest work and it is often exhaustive. It is also organised around how the perception team divides labour, which is not the axis along which a reviewer decides. The expert move is to start from the list of questions a reviewer must be able to answer with a signature attached, define exactly one package section per question, and then work backwards to the evidence that closes it.
This is a structuring discipline, not extra testing work. Same evidence, indexed against the questions it actually answers.
What does engineering sections to reviewer questions mean in practice?
It means the section set is derived, not inherited. You write down the reviewer’s questions first — in their words — and each one becomes a heading with a stated completion condition. A section is then judged complete or incomplete on its own terms, independent of how many tests happen to sit behind it. Ten tests behind a section that answers its question is complete; four hundred tests behind a section whose question is never quite answered is not.
The practical consequence: a reviewer-shaped section can be scored before submission, because completeness is a property of the question, not of the backlog. A backlog-shaped section cannot be scored at all — you only find out in the meeting.
Elicitation is unglamorous and it is where most of the value sits. In our experience the questions do not arrive as a tidy list; they arrive as review comments from the last release, as follow-up emails, as the three things the OEM-side QA lead always asks in the first twenty minutes. Harvest those. Sit with the release owner and the safety lead separately, because they ask different questions and merging them early loses both. Read the previous review round’s minutes as a requirements document — every question that triggered a follow-up request is a section you were missing.
Write each question down verbatim before you touch the structure. Paraphrasing a reviewer’s question into engineering vocabulary is exactly how the section drifts back toward the backlog.
Deriving one section per question
The mapping is one-to-one on purpose. Two questions in one section means the section has two completion conditions and will be argued about; one question spread across three sections means the reviewer assembles the answer manually, which is the failure we started with.
What belongs in the section is the claim, the evidence that closes it, and the stated limit of that claim. What belongs in an appendix is bulk that supports the claim but is not read linearly — full scenario catalogues, per-frame logs, dataset manifests, raw metric dumps. Appendices are referenced from sections, never the reverse. If a reviewer has to read an appendix to decide, the section is incomplete.
Each section carries a completion condition written as a test you can apply yourself. Not “coverage is adequate” — that is an opinion. Something closer to: every hazard in the linked hazard list maps to at least one executed scenario with a recorded outcome and a named disposition for every failure. That condition is checkable on a Friday afternoon before submission.
| Reviewer question | Section | Completion condition |
|---|---|---|
| What operating domain is being claimed? | Scope and operational design domain | Domain boundaries stated as measurable conditions; every out-of-domain condition explicitly excluded in writing |
| What failure modes were searched for? | Failure taxonomy and search strategy | Each taxonomy entry has a search method and an executed-or-declared-untested status |
| How does it behave at the edge of the domain? | Degradation and edge-case behaviour | Named degradation modes with observed behaviour per mode; no mode left as “not observed” without a rationale |
| What evidence sits behind each number? | Metric definitions and provenance | Every reported metric resolves to a dataset version, a configuration record, and a definition |
| What risk am I accepting? | Residual risk statement | Every known limitation appears with an owner and an accepted-or-mitigated disposition |
| What changed since the release I last signed? | Change delta | Diff against the previous signed package at section granularity, including sections that did not change |
That last row is the one teams skip and reviewers always want. “Nothing changed in this section” is itself evidence, and it is cheap to produce once the structure is fixed.
Mapping the existing backlog onto reviewer-shaped sections
The reflex when a package stalls is to run more tests. In most stalled reviews the evidence already exists somewhere in the backlog; what is missing is the index. So the migration is a re-indexing exercise, not a re-validation one.
A workable order:
- Freeze the reviewer question list and its section set. No evidence work until the headings are agreed with at least one real reviewer.
- For each section, write the completion condition before looking at what you have. Doing it the other way round produces conditions your current evidence happens to satisfy.
- Walk the backlog once and tag each artefact — suite, run, plot, log — with the section it serves. Artefacts serving no section are candidates for the appendix or for retirement; that is a signal, not a problem.
- Score each section against its condition. Partial is partial.
- Only now decide what to run. The gap list is short and specific, and it is defensible to a programme manager because each item names the reviewer question it closes.
Tooling here is ordinary. Test results already carry versioned identifiers in whatever harness the team uses — a pytest run, a scenario-replay job in a simulation stack, an MLflow run with its parameters and artefacts attached. The mapping layer is a manifest that binds section IDs to those identifiers, and it is worth keeping in version control alongside the model configuration rather than in a document. We treat that manifest as the deliverable that survives; the rendered package is a view of it.
Which failure modes mean the package is still backlog-shaped?
Four signals, and each one shows up in review before it shows up in the document:
- Headings name suites, tools, or teams. “Regression Suite B results” is a backlog heading. “Behaviour at the edge of the claimed domain” is a reviewer heading.
- Section count tracks the test suite. If adding a test suite adds a section, the structure is inherited, not derived. Reviewer questions change slowly; suites change every sprint.
- No section can be marked done. If the honest status of every section is “we have a lot of material here”, nobody can report readiness, and sign-off becomes discovery.
- Reviewers ask questions your table of contents does not contain. Every such question is a missing section, and it will return next release unless you add it.
The review round that follows a backlog-shaped package has a recognisable shape: the reviewer spends the first half of the meeting locating things, asks a question that spans three sections, receives an answer that is assembled live, and — reasonably — declines to sign an argument they watched being constructed. Then a follow-up request list arrives, and the next round starts from a partially reorganised document. That is how one sign-off becomes several rounds across weeks. Reviewer-shaped sections compress this because each question maps to exactly one section with a stated completion condition, so the meeting is a check rather than an excavation. We see the same pattern across the perception validation work we do; it is a structural property of the artefact, not a property of any one team’s diligence.
Why the derived section set survives model updates
Because it was never derived from the model. A re-trained backbone, a new sensor variant, or a reorganised regression suite changes what goes into the evidence slots; it does not change what a reviewer needs to be able to answer. The scope question, the failure-taxonomy question, the residual-risk question and the change-delta question are the same at the next release and the one after.
This is where the justification work gets paid once. A backlog-shaped package is re-authored every cycle because its structure follows the suite; a reviewer-shaped package is re-populated, and the reviewer arrives already knowing the artefact’s layout. That familiarity is worth as much as the evidence — a reviewer reading a structure they have signed before spends their attention on what changed. Our broader approach to production AI reliability treats this stability as the point: the structure is the long-lived asset, and the evidence is what gets refreshed.
The one thing that does force a structural change is a change in who reviews. When OEM governance review enters scope, the question set expands beyond engineering QA — ownership, change control after start of production, independent auditability — and the section set has to absorb those before the package is presented, not after. That is a deliberate re-derivation from a new question list, which is exactly the mechanism working as designed.
For the wider argument about what a validation package has to carry and why testing completeness is not evidence completeness, see our coverage of production perception reliability and the release-review gap.
Frequently Asked Questions
What does engineering package sections to a reviewer’s questions rather than the test backlog mean in practice? Structure each validation package section around the specific question a safety reviewer will ask, not around your engineering workflow. Engineering Validation Package Sections is simpler than it looks. Engineering Validation Package Sections is best answered directly. Engineering Validation Package Sections is simpler than it looks. With Engineering Validation Package Sections, the detail that matters is this. Engineering Validation Package Sections is best answered directly. Engineering Validation Package Sections is simpler than it looks. Engineering Validation Package Sections is best answered directly. Engineering Validation Package Sections is simpler than it looks. With Engineering Validation Package Sections, the detail that matters is this. Engineering Validation Package Sections is simpler than it looks. Engineering Validation Package Sections is best answered directly. Engineering Validation Package Sections is simpler than it looks. Engineering Validation Package Sections is best answered directly. Engineering Validation Package Sections is simpler than it looks. With Engineering Validation Package Sections, the detail that matters is this. It means the section headings are derived from what a reviewer must be able to answer before signing, not from how the perception team organises its test suites. One question becomes one section, and the evidence is then indexed against the questions it closes. The testing work does not change; the index does.
How do you elicit and write down the reviewer’s actual questions before the package structure is fixed? Harvest them from the previous review round’s comments and follow-up requests, then interview the release owner and the safety lead separately — they ask different questions and merging them early loses detail from both. Record each question verbatim before designing structure, because paraphrasing into engineering vocabulary is how sections drift back toward the backlog.
How do you derive one section per question, and what belongs in a section versus an appendix? Keep the mapping strictly one-to-one: two questions in one section produces two competing completion conditions, and one question split across sections forces the reviewer to assemble the answer. The section holds the claim, the evidence that closes it, and the claim’s stated limit; bulk material — scenario catalogues, per-frame logs, dataset manifests — goes to an appendix referenced from the section.
What completion condition should each section carry so it can be judged done before submission? A condition phrased as a check you can run yourself, not a judgement. “Coverage is adequate” is an opinion; “every hazard in the linked list maps to an executed scenario with a recorded outcome and a named disposition for each failure” is checkable before you submit.
How do you map existing test-backlog artefacts onto reviewer-shaped sections without rerunning validation work? Freeze the section set and its completion conditions first, then walk the backlog once and tag each artefact with the section it serves. Score each section against its condition, and only then decide what to run — the resulting gap list is short and each item names the reviewer question it closes.
Which failure modes signal that a package is still backlog-shaped? Headings that name suites, tools or teams; a section count that grows when a suite is added; no section that can honestly be marked done; and reviewer questions that your table of contents does not contain. The matching review round is one where the reviewer spends the first half locating evidence and the answer to a cross-cutting question is assembled live.
How does a reviewer-derived section set stay stable when the model, sensor set, or test suite changes? Because it was derived from the reviewer’s decision rather than from the model or the suite. A re-trained model or a new sensor variant re-populates the evidence slots; the scope, failure-taxonomy, residual-risk and change-delta questions are unchanged. Structure only re-derives when the reviewer population changes — for example when OEM governance review joins the release.
Structuring Engineering Validation Package Sections that reviewers trust
Map every section to a specific regulatory question, not an internal milestone, and approval cycles shrink by half.
