An AI feature passes its accuracy threshold on the held-out set, the demo behaves, the vendor benchmark looks strong — and the team approves the release. Three weeks later, an input distribution nobody anticipated drives a quiet regression that the dashboard never flagged, support tickets climb, and the post-mortem reveals there was never a rollback plan because nobody owned the decision to make one. That sequence is not a freak event. It is the default outcome when release approval rests on the one lever the team can see — accuracy on a static evaluation set — instead of on a measured commitment about how the feature behaves in production. The question “is this model good enough?” is the wrong question. The right one is: what evidence do we have that this feature is ready to operate, fail, and be withdrawn under real conditions? This article lays out a release-readiness decision framework for AI features: the named go/no-go signals, the evidence a gate must produce before it fires, and the conditions under which the correct answer is to roll back rather than patch. It is a decision instrument, not a maturity score — it tells you whether this release candidate should ship, not how sophisticated your organisation is at AI. What Signals Tell Us an AI Feature Is Ready for Production? A release-readiness gate is a structured review that converts a binary accuracy judgement into a multi-axis operational judgement. The naive approach approves on a single visible lever and instruments later. The expert approach produces an approval-grade evidence pack before the release fires. The difference matters because most AI features that ship without a release-readiness gate regress in production in ways the pre-release evaluation never surfaced — this is the structural pattern the framework exists to counter, not an isolated anecdote. Accuracy on a held-out set tells you how the model performs on data that resembles the test distribution. It tells you nothing about input drift, latency under concurrent load, the blast radius of a bad output, or whether anyone is on the hook to pull the feature when it misbehaves. A ready feature clears five axes, not one: Axis Go signal No-go signal Eval coverage Evaluation set covers known edge cases, adversarial inputs, and the input segments that matter operationally — not just the convenient distribution Eval is a single accuracy number on data sampled the same way as training Drift baseline Input-distribution and output-distribution baselines are recorded, with alert thresholds defined before launch No baseline exists, so any future drift is undetectable by definition Kill-switch rehearsal The feature can be disabled in production, and someone has executed that path in staging A kill switch is “documented” but never exercised Ownership matrix A named owner holds the go/no-go decision, the on-call rotation, and the rollback authority Ownership is implicit or shared across a team, which means it belongs to no one Rollback plan A specific, rehearsed sequence reverts to the prior version with bounded data-consistency risk “We’ll figure it out” — rollback is theoretical This table is the spine of the framework. A release that clears all five is a measured commitment. A release that clears one is a bet against the operational track record of AI features — and that track record is not in your favour. How Do We Run a Release-Readiness Review Without It Becoming Theatre? The fastest way to kill a release gate is to turn it into a checkbox ritual that everyone games. A gate becomes theatre when the signals are subjective (“the team feels confident”), when the evidence is asserted rather than produced, and when no signal can actually return no-go without political cost. Three properties keep a gate honest. First, every signal must be backed by an artefact, not an opinion — a recorded drift baseline, a staging log showing the kill switch fired, a named owner in the on-call tool. If a signal cannot point at an artefact, it is not a signal; it is a hope. Second, the gate must have the standing to block. A review that can only ever say yes is a formality. Third, the evidence must be reproducible across model versions, because the next release candidate will face the same gate and the artefacts must not silently go stale. This is exactly the discipline that release engineering for AI features formalises at the pipeline level — the gate is where that engineering practice meets the go/no-go moment. The review is not a meeting; it is the act of inspecting an evidence pack that the pipeline already produced. We see this pattern regularly: teams that treat the gate as a document fail it the same way teams that skip it entirely fail — the document describes a feature that was never actually tested against the conditions it will meet. The gate has to test the candidate, not describe it. What Evidence Does a Release-Readiness Gate Need to Produce? The output of a gate is an evidence pack — a self-contained record that an engineering lead and a product owner can read and sign. It is the difference between “we think this is fine” and “here is what we measured.” A complete pack contains: Eval report — performance broken down by the operationally relevant input segments, with the edge-case and adversarial results called out, not averaged away. Drift baselines — recorded distributions for the inputs and outputs, with the alert thresholds and the detection latency you expect (observed-pattern: in our experience, detection latency is the metric teams most often forget to define, and it is the one that determines how bad an incident gets before anyone notices). Kill-switch rehearsal log — evidence that the disable path was executed in a staging or canary environment, with the time-to-disable recorded. Ownership matrix — named human owners for the go/no-go decision, the on-call response, and the rollback trigger. Rollback plan — the specific revert sequence, its data-consistency implications, and the rehearsed time-to-rollback. The gate, executed against a real release candidate, is what a production AI monitoring harness is built to deliver — the harness is the instrument; the evidence pack is its output. If you want the gate run against your candidate as an engagement, that is the Production AI Monitoring Harness we operate; it sits inside the broader AI infrastructure and SaaS surface for teams standardising this across many features. To be clear about what this framework does not claim: a release-readiness gate does not replace engineering judgement, and no gate promises zero-incident production AI. The gate raises the floor on what you know before you commit. It does not eliminate the risk; it makes the risk a measured, owned decision instead of an accidental one. When Is a Rollback or Kill-Switch the Right Answer Instead of a Remediation Sprint? The most expensive mistake after a bad release is not the release — it is the reflex to fix forward when you should revert. Engineers default to remediation because rolling back feels like an admission of failure. Operationally, it is often the cheaper and safer move. Use this rubric at the moment of an incident: Condition Right answer Regression has a bounded, understood cause and a fix can be validated within the time the impact stays tolerable Remediation sprint Regression cause is unknown, or the blast radius is growing, or impact is user-facing and severe Kill switch now, diagnose after Prior version is known-good and rollback data-consistency risk is bounded Roll back, then remediate offline Rollback itself carries unbounded data-consistency or state-migration risk Kill switch (degrade gracefully), then forward-fix under control The metrics that decide this are time-to-detect, time-to-rollback, and the avoided cost of leaving a bad release live. A release-readiness gate that rehearsed the kill switch and recorded a rollback plan has already pre-decided most of this — which is the entire point. You do not want to be designing the rollback during the incident. How Does Release-Readiness Differ from AI Readiness or AI Maturity? These terms get conflated, and the conflation causes teams to answer the wrong question. Release-readiness is about a specific feature candidate: is this version safe to ship today? AI readiness and AI maturity are organisational measures — how capable is the team, how good is the tooling, how repeatable is the process across the portfolio. A mature organisation runs a release-readiness gate as routine. But a high maturity score does not make any individual release ready, and an immature team can still gate a single critical feature properly if it produces the evidence pack. The gate is per-release; maturity is per-organisation. Keep them separate, because a maturity assessment will never tell you whether to ship on Tuesday. This is the same separation that governs the cost side of production-AI decisions — why cost-per-request is the right optimisation target anchors a release decision in a named per-unit KPI rather than an organisational posture, exactly as release-readiness anchors the go/no-go in named operational signals. Both reject the maturity-score proxy in favour of a measured commitment about a specific decision. FAQ What signals tell us an AI feature is ready for production? A ready feature clears five axes, not just accuracy: eval coverage that includes edge and adversarial cases, recorded drift baselines with pre-defined alert thresholds, a rehearsed kill switch, a named ownership matrix, and a rehearsed rollback plan. Passing an accuracy threshold on a held-out set addresses only one of these and tells you nothing about how the feature behaves, fails, or gets withdrawn under real load. How do we run a release-readiness review without it becoming theatre? Back every signal with an artefact rather than an opinion — a recorded baseline, a staging log of the kill switch firing, a named owner in the on-call tool — give the gate the standing to actually return no-go, and make the evidence reproducible across model versions. A review that can only say yes, or that describes a feature instead of testing the candidate, is theatre regardless of how thorough the document looks. What evidence does a release-readiness gate need to produce? An approval-grade evidence pack: an eval report broken down by operationally relevant input segments, recorded drift baselines with alert thresholds and expected detection latency, a kill-switch rehearsal log with time-to-disable, a named ownership matrix, and a rehearsed rollback plan with its data-consistency implications. The pack is what an engineering lead and product owner read and sign — the difference between “we think this is fine” and “here is what we measured.” When is a rollback or kill-switch the right answer instead of a remediation sprint? Remediate forward only when the cause is bounded and understood and a fix validates within the time the impact stays tolerable. Hit the kill switch when the cause is unknown or the blast radius is growing; roll back when a prior version is known-good and revert risk is bounded. Time-to-detect, time-to-rollback, and the avoided cost of leaving a bad release live are the metrics that decide it — and a gate that rehearsed these has pre-decided most of the call. How does release-readiness for an AI feature differ from general AI readiness or AI maturity at the organisation level? Release-readiness is per-release — is this specific version safe to ship today? AI readiness and maturity are per-organisation measures of team capability, tooling, and process repeatability. A high maturity score never makes an individual release ready, and an immature team can still gate one critical feature properly by producing the evidence pack; conflating them answers the wrong question at the wrong scale. What belongs in a release-readiness checklist for an AI feature, and how do we keep it from going stale across model versions? The checklist is the five-axis evidence pack: eval coverage, drift baselines, kill-switch rehearsal, ownership matrix, and rollback plan. It stays current because each artefact is produced by the pipeline against the new candidate rather than copied forward — the drift baseline is re-recorded, the kill switch is re-rehearsed, and ownership is re-confirmed for every version, so a stale artefact fails the gate the same way a missing one does. A release that passes this gate is not a guarantee — it is a measured commitment with named owners and a rehearsed exit. The harder question the framework forces you to answer is the one teams avoid until it is too late: not “is the model accurate enough?” but “if this feature fails next Tuesday, who decides, how fast do we know, and how fast can we pull it?” The reliability-audit discipline that instruments this gate end to end is covered in what a production AI reliability audit actually tests — the audit is where the evidence pack stops being a one-off review and becomes a repeatable instrument.
