A validation package assembled from the perception team’s test backlog can be complete, current, and entirely unreadable to the person who has to sign it. The stall is not caused by missing tests. It is caused by an organising axis — ticket IDs, suite names, regression run numbers — that belongs to test tracking rather than to the reviewer’s decision, and the consequence lands at the review meeting rather than during preparation, when the release date is already committed.
This is a diagnostic piece. It names how the mismatch presents, which sections turn out to be absent rather than merely misplaced, and which parts of an existing backlog survive into a reviewer-shaped package.
What the mismatch looks like in a review meeting
The reviewer opens the package and asks a question. Not a hostile one — usually the first item on a standard list: what operating domain is this performance claim scoped to? The perception lead knows the answer. It exists, distributed across a scenario suite name, a dataset ticket, and a paragraph in a test plan appendix. So the answer is spoken aloud, and the reviewer writes it down.
That is the moment the package failed. Evidence that has to be narrated by its author is not reviewable evidence; it is a briefing. A reviewer signing against a claim needs to locate that claim in the artefact, follow it to the thing that supports it, and be able to do so again in six months without the author in the room.
A backlog-shaped package can be exhaustive and still leave the reviewer unable to locate the answer to any single one of their questions. Exhaustiveness and answerability are independent properties, and test-tracking conventions optimise only the first.
Why does reorganising the backlog not fix it?
Because several of the reviewer’s questions have no corresponding artefact anywhere in the backlog. A test backlog records what was executed. It does not record what was deliberately not claimed, what failure modes were searched for and not found, what residual risk the team accepts, or what changed relative to the release the reviewer approved last time. Those are authored statements, not exported ones.
So the remedy is not a reordering pass. It is a partial rewrite — which is fine when it is planned and expensive when it is discovered during review. In our experience with perception teams facing a first OEM review, the sections that must be written from scratch are consistently the scope statement, the residual-risk statement, and the change delta. The rest is mostly re-attribution of material that already exists.
Early symptoms, before the package leaves the building
These show up during internal dry-runs if anyone is looking for them.
| Symptom | What it indicates |
|---|---|
| Section headings match test-suite names | Organising axis is the tracking tool, not the claim structure |
| Every metric is reported globally, none per operating condition | No operational-domain decomposition exists to report against |
| No section can be read without the perception lead present | Package is a briefing aid, not a signable artefact |
| The word “residual” appears nowhere | Accepted risk was never stated, so nothing can be accepted |
| Nothing distinguishes this release from the previous one | Reviewer must re-review everything, every cycle |
| Failure cases appear only as open defect tickets | Failure-mode search was recorded as work, not as evidence |
| Section owners are unnamed, or all one name | No signature can be scoped to a person’s actual authority |
Any two of these together predict a multi-round review. The last three predict that the second round will ask for material that does not yet exist.
What carries over, and in what form
The backlog is not waste. Most of the evidentiary substance a reviewer needs was already produced; it is the framing and the attribution that are missing. A useful triage separates the three fates a backlog item can have.
- Carries over as-is — dataset manifests with version identifiers, calibration and sensor-configuration records, per-scenario pass/fail results, and the failure-mode log with disposition. These are already artefacts; they need a section to sit under and a claim to support.
- Carries over re-aggregated — global metrics get re-cut by operating condition (weather, illumination, occlusion, class frequency), because the reviewer’s question is conditional and the backlog’s answer is marginal. The underlying runs are usually sufficient; the aggregation is not.
- Does not carry over — ticket IDs, sprint attribution, suite naming, regression-run counts, and internal defect workflow states. These are process telemetry. They belong in the tracking tool and cost the reviewer attention when exported.
The parent structure this feeds into — sections defined by the question each answers, with an accountable owner per section — is developed in our reference structure for perception validation package contents, which is the right next read once the diagnosis here comes back positive.
Internal QA reviewer versus OEM customer reviewer
The mismatch presents differently depending on who is holding the pen, and teams often calibrate on the easier case.
An internal QA reviewer shares vocabulary with the perception team. They can reconstruct the argument from a backlog-shaped package because they know what a suite name implies, and they will often do it silently. That is why an internal sign-off is a poor predictor of an external one — the package passed on the reviewer’s domain knowledge, not on its own structure.
An OEM customer reviewer has no such context and, critically, has a documented question list they must close. They cannot accept a narrated answer, because their own downstream reviewer will ask them where it is written. The same package that cleared internal QA in one round can absorb three or four external rounds, each separated by days to weeks of blocked timeline (an observed pattern across automotive-perception engagements, not a benchmarked figure). When the reviewer is an OEM governance function rather than a release gate, the gap widens further into provenance, ownership, and change-control evidence the package was never built to carry.
Triage on a package already in review
Rebuilding from zero mid-review is rarely the right call — the release date is fixed and the evidence mostly exists. The faster route is to score section by section.
- Write down the reviewer’s actual question list, verbatim from their comments if you have them.
- For each question, mark whether the package contains a locatable answer, a reconstructable answer, or nothing.
- Rewrite only the “nothing” rows first. These are typically scope, residual risk, and change delta — three authored statements, not three test campaigns.
- Re-aggregate the “reconstructable” rows by operating condition so the answer is findable rather than derivable.
- Leave the “locatable” rows alone. Editing them consumes time and changes no verdict.
- Attach a named owner to every section before resubmission, so the reviewer knows whose authority each statement carries.
Steps 3 and 4 account for most of the recovered time. Teams that reach for step 1 late tend to have spent the intervening week adding test cases, which addresses a shortage that was never the problem.
Preventing recurrence at the next model update
The structural reason this repeats is that nothing about a backlog-shaped package is invariant across releases. Re-train the model, re-run the campaign, and the package is re-argued from scratch — the reviewer re-learns the artefact, and the round count resets. Fixing the section structure once, then treating a model update as a re-population of evidence slots, is what converts sign-off from a project into a step. We treat that structural invariance as part of the same engineering discipline as production AI reliability generally: the artefact that proves a system behaves within known bounds is designed, versioned, and maintained, not assembled at the end.
The open question worth sitting with is where the boundary falls for your programme — how much of the reviewer’s question list is genuinely stable across releases, and how much is set fresh by each OEM you supply. Teams with one customer often over-fit their structure to that customer’s list and discover the mismatch again on the second contract.
Frequently Asked Questions
What does “why team-backlog-shaped packages stall at reviewer time” mean in practice for a perception release? A common Team Backlog Shaped Validation question is worth clarifying. It means the package is organised by how testing was tracked — suites, tickets, runs — rather than by the questions a reviewer must answer with a signature attached. The evidence may be complete, but the reviewer cannot locate any single answer without the author narrating it, so the review does not close.
What are the early symptoms that a package is backlog-shaped rather than reviewer-shaped? Section headings that mirror test-suite names, metrics reported globally with no per-condition breakdown, no residual-risk statement, no delta against the previous release, and sections with no named owner. Any two together predict a multi-round review.
Which reviewer questions does a test backlog structurally fail to answer, and why is reorganising it not enough? Scope of the operating domain claimed, which failure modes were searched for, what residual risk is accepted, and what changed since the last release. A backlog records executed work, not authored positions, so these sections have to be written rather than rearranged.
What parts of the existing test backlog do carry over into a reviewer-shaped package, and in what form? Dataset manifests, calibration records, per-scenario results and the failure-mode log carry over largely as-is, needing only a claim to sit under. Global metrics carry over re-aggregated by operating condition. Ticket IDs, sprint attribution and regression-run counts do not carry over at all.
How does the stall show up differently with an internal QA reviewer versus an OEM customer reviewer? Internal QA shares the team’s vocabulary and often reconstructs the argument silently, which makes internal sign-off a weak predictor. An OEM reviewer has to close a documented question list and cannot accept a narrated answer, so the same package absorbs several more rounds.
How do you triage a package that is already in review and stalling, without rebuilding it from zero? Score each reviewer question as locatable, reconstructable, or absent. Write the absent ones first — usually scope, residual risk, and change delta — then re-aggregate the reconstructable ones by operating condition, and leave the locatable ones untouched.
What changes in the release process prevent the mismatch recurring at the next model update? Fix the section structure so it is invariant across releases and treat a model update as a re-population of evidence slots rather than a new documentation exercise. Naming a persistent owner per section keeps attribution stable when the model changes.
Reviewers need narrative, not sprint artifacts
Validation packages assembled from Jira tickets inherit sprint structure — but external reviewers need causal logic, risk mitigation chronology, and evidence threads your backlog was never designed to provide.
