A gate that has never failed a release is not a gate; it is a signature line. Most release-readiness reviews for AI features do fail — in the wrong direction. The gate runs, gaps are found, someone writes them down as follow-up tickets, and the release ships on schedule with exactly the evidence holes the review was built to catch. Two cycles of that and the review is theatre: attendance is high, the outcome is predetermined, and nobody is fooled except the people reading the sign-off.
The decision this article is about is narrow and uncomfortable. When a review finds missing evidence, is the correct answer no-go, or is it conditional pass with an exception? Getting that call right in advance — before the room is full and the launch date is public — is what keeps a gate load-bearing.
What does failing a release-readiness review actually mean?
It does not mean the model is bad. It means the release candidate cannot be defended with evidence that exists today, and the honest record of that is a no-go rather than a softened finding.
The distinction matters because reviewers instinctively treat a no-go as a political cost and price it accordingly. Findings get rounded down. A missing drift baseline becomes “nice to have”. An untested rollback script becomes “documented, we’ll rehearse next sprint”. Each individual softening is defensible; the cumulative effect is a gate whose thresholds are negotiated at review time, which is the same as having no thresholds.
A failed review is only useful if it produces a remediation contract rather than a delayed ship date. That is the divergence point. A delay with the same evidence gaps intact buys nothing — the team returns in a week with a newer build and the same unanswerable questions. A remediation contract names the missing evidence, the owner who will produce it, and the trigger that brings the release back for re-review.
Hard fail or conditional pass: the threshold table
Decide these categories once, at the process level, and publish them. Deciding them in the room is how gates degrade.
| Missing evidence | Verdict | Why | What a conditional pass would require |
|---|---|---|---|
| No rollback path, or one that has never been executed | Hard fail | Without a rehearsed rollback, every other mitigation is theoretical; the blast radius is unbounded by construction | Nothing. Rehearse it, then come back |
| No kill-switch rehearsal for the AI-specific path | Hard fail | A flag that has not been flipped in a live-like environment is an assumption, not a control | Nothing |
| No named owner for an AI-specific failure mode | Hard fail | Unowned failure modes have no time-to-detect; escalation resolves by chance | Nothing |
| No eval coverage on the behaviour that changed | Hard fail | The release is unmeasured precisely where it is new | Nothing |
| Drift baseline defined but not yet recorded in production telemetry | Conditional pass | Recordable inside a bounded window; risk is detectability lag, not absent control | Time-boxed exception, narrow rollout, dated owner |
| Eval coverage thin on a low-traffic segment | Conditional pass | Exposure is bounded by traffic share and can be capped | Rollout cap on that segment, expiry date |
| Runbook incomplete for a failure mode that is owned and alarmed | Conditional pass | Owner plus alarm gives a detection path; the runbook shortens response, not detection | Exception with expiry, owner named in the alarm |
| Monitoring dashboard exists but nobody has confirmed the alert routes to a human | Hard fail | An alert that pages nobody is not monitoring | Route it and prove it; ten minutes of work |
The pattern behind the table: the hard-fail line falls where the missing evidence removes a control, and the conditional-pass line falls where it only delays a control’s precision. Missing rollback removes a control. An unrecorded drift baseline delays precision. That test generalises better than any list, which matters because the list will not cover every case you meet.
One more rule worth codifying: a conditional pass with no expiry date is a hard fail wearing a different hat. If nobody will notice when the exception lapses, the exception is permanent.
The remediation contract
A no-go is only as good as the document it produces. In our experience the difference between a gate that survives its first contested release and one that quietly dies is whether the fail output is structured enough to be actionable by someone who was not in the room.
A remediation contract contains five things and nothing else:
- The specific evidence that is missing — named artefact, not a category. “Rollback rehearsal log against release candidate 4.2, with timings” rather than “rollback testing”.
- The named owner of that artefact — a person, not a team.
- The re-review trigger — usually “artefact attached to the evidence pack”, occasionally “artefact plus one clean canary window”.
- A duration. A remediation window longer than one release cycle means the gap is structural and belongs on the roadmap, not on an exception.
- The scope of the re-review — partial, covering only the failed categories, unless the model or data version changed in the interim, in which case the eval evidence is stale and the full pack is back in scope.
The [production-AI validation pack](Production AI Monitoring Harness) is the natural home for all five, because it is already the artefact the review scores. A remediation contract that lives in a meeting note has a half-life of about a week; one that lives as unmet rows in the evidence pack survives reassignment, holidays, and the reorganisation.
Cycle time from failed review to approvable re-submission is the metric we would watch most closely. It is the honest test of whether remediation was scoped or open-ended. Alongthe parent framework’s operational measures — post-release incident count, time-to-detect on regressions, time-to-rollback — it tells you whether the gate is teaching the release process anything. If re-submission cycle time keeps growing, the fails are being used to defer work rather than to bound it.
How a no-go survives escalation
Escalation is where discretionary gates lose. A no-go framed as reviewer judgement invites a counter-judgement from someone more senior, and seniority wins that argument every time. A no-go framed as this named evidence does not exist is much harder to overturn, because overturning it requires stating on the record that the release proceeds without that evidence.
That reframing is the whole trick, and it is why the fail thresholds have to be published before the contested release rather than derived during it. The reviewer is not asserting the release is unsafe. The reviewer is reporting that row seven of the evidence pack is empty against a threshold everyone agreed to in a calmer month.
Overrides should exist — a gate with no override path gets bypassed rather than argued with — but they should cost evidence. A defensible override names the accountable executive, records which specific rows are being waived, states the compensating control substituted in their place (a tighter rollout cap, a staffed watch window, a manual kill-switch operator on shift), and sets an expiry. Overrides that cost nothing get used routinely; overrides that cost a signature and a compensating control get used when they are genuinely warranted. Either way, the count of overrides by evidence category is one of the most informative numbers a platform team can track — it points straight at where the release process is structurally thin. If drift baselines are waived on four releases in a row, the problem is not the reviews. It is that nobody has built drift baselining into the pipeline.
The evidence-scoring discipline that makes this work — scoring artefacts rather than assertions — is the same discipline a reliability audit applies to a system already in production. A release-readiness review is that audit run early, with the option to say no still available.
When to fail outright rather than ship behind a flag
Flags and narrow rollouts are the standard answer to a partially-ready release, and usually the right one. They stop being the right one in three situations.
The first is when the flag itself is the untested component. Shipping behind a control you have not exercised means the mitigation and the risk share a failure mode. The second is when the failure is silent — a retrieval quality regression, a subtle grounding failure, a slow accuracy decay on a segment nobody dashboards. A narrow rollout limits exposure only if someone notices; without a drift baseline and a routed alert, a 2% rollout can run degraded for a month. The third is when rollback is not clean because the release writes state — cached embeddings, user-visible artefacts, downstream records. Turning the flag off does not undo those, and “we can flag it off” is a false comfort.
In all three cases the flag is doing less work than it appears to, and a fail is the cheaper option. That is the underlying economics of a defensible no-go: it converts an unbounded post-release incident into a bounded pre-release engineering cost. The cost is visible and annoying, which is exactly why it is undervalued.
Reading the fail rate
A gate with a zero percent fail rate is not working. Neither is one with a seventy percent fail rate — that means readiness criteria are not reaching teams early enough to influence how work is planned, and the review has become a bottleneck rather than a check.
Between those extremes, look at the shape rather than the number. Healthy signs: fails cluster in a small number of evidence categories, those clusters shift over time as the platform closes them, and re-submission cycle time is short and stable. Unhealthy signs: fails cluster in the same category for six months, conditional passes outnumber both passes and fails, exceptions lapse without anyone noticing, and overrides arrive from the same person every quarter.
We treat the distribution of fails by evidence category as a leading indicator of platform gaps, not as a scorecard for the teams being reviewed. The moment it becomes a team scorecard, findings get softened again — and the gate is back to being a signature line.
For the four operational signals a review scores against in the first place, and how they interact, see our framework for deciding whether an AI feature is ready to ship, which sets the thresholds this article decides how to enforce.
Frequently Asked Questions
What does failing a release-readiness review mean in practice, and what happens next?
Blocking a release requires courage, but shipping a model that fails silently in production costs far more than a two-week delay. Fail Release Readiness Review turns on one distinction. It means the release candidate cannot be defended with the evidence that exists today — not that the model is bad. What happens next is a remediation contract: the specific missing artefact, a named owner, a re-review trigger, and a duration. A fail that produces only a delayed ship date, with the same gaps intact, has accomplished nothing.
Which missing evidence is a hard fail versus a conditional pass with a time-boxed exception?
Hard fail when the gap removes a control: no rollback path, no kill-switch rehearsal, no named owner for an AI-specific failure mode, no eval coverage on the behaviour that changed, or alerts that route to nobody. Conditional pass when the gap only delays a control’s precision — an unrecorded drift baseline, thin coverage on a low-traffic segment, an incomplete runbook for a failure mode that is already owned and alarmed. Any conditional pass without an expiry date should be treated as a hard fail.
What does a remediation contract contain, and how long should it run?
Five elements: the named missing artefact, its owner as a person, the re-review trigger, a duration, and the scope of the re-review. If the remediation window needs to exceed one release cycle, the gap is structural and belongs on the roadmap rather than on an exception.
Who has the authority to override a no-go, and what should that override cost in evidence terms?
Overrides should exist, because a gate with no override path gets bypassed rather than argued with. A defensible override names the accountable executive, records exactly which evidence rows are waived, substitutes a compensating control such as a rollout cap or a staffed watch window, and carries an expiry. Track overrides by evidence category — repeated waivers in one category point at a pipeline gap, not a reviewer problem.
How do we tell a healthy fail rate from a gate that has become theatre?
Look at shape, not percentage. Healthy gates show fails clustered in a few evidence categories that shift as the platform closes them, with short and stable re-submission cycle times. Theatre shows up as conditional passes outnumbering passes and fails, exceptions lapsing unnoticed, and the same category failing for months without anyone changing the pipeline.
Two red flags that demand a delay
Missing rollback procedures or untested cost caps mean you’re flying blind—both are fixable in days, not weeks. Revisit it when your workload shifts.
