Almost no team decides to ship an AI feature early. The decision is rarely made at all — the launch date arrives, nobody holds an instrument that can produce a defensible no, and the release goes out on confidence rather than evidence. That is the failure mode: not accepted risk, but absent instrumentation. An AI feature ships before it is ready when the release decision is not falsifiable.
This matters because the two situations look identical from the outside. A team that assessed the risk and shipped anyway, and a team that never had a mechanism to assess it, both produce the same artefact: a release note and a live feature. The difference only becomes visible when the regression lands, and by then the question “who decided this was ready?” has no answer anyone can point at.
Why does release pressure override engineering judgement on AI features?
Conventional software has a natural brake. A test suite goes red, a build fails, a migration throws. The brake is mechanical and it is embarrassing to override, because overriding it means editing something in the pipeline that leaves a trace. Release pressure exists, but it argues against an instrument.
AI features frequently have no equivalent brake. The eval suite produces a number, and a number is negotiable in a way that a red build is not. Is 0.91 acceptable? It depends — on the segment mix, on what the previous model scored, on what the product owner promised, on how much slack the roadmap has. Every one of those is a conversation, and conversations under date pressure resolve towards the date. We see this pattern regularly: the model looks fine in review, the demo behaved, and nobody in the room can articulate what evidence would have justified a delay.
There is a second structural reason. Most AI failure modes are not present at release. Drift, distribution shift in a specific customer segment, prompt regressions after a hosted-model update — these arrive on day 20, not day zero. A release process that only asks “does it work now?” is structurally blind to the class of risk that actually produces the incident. The gate exists to convert an unaskable future question into a present-tense evidence requirement.
The four patterns that precede an early ship
In our experience, an early ship is almost always preceded by one of four specific, recognisable conditions. Each has a signature in the release artefacts, which means each is detectable before rollout rather than after.
Thresholds set on aggregate metrics. The eval reports one headline number over the whole test set. A model that has degraded badly on a 4% segment and improved slightly elsewhere passes cleanly. In production the regression is concentrated, visible, and attributable to the release — which is the worst combination for user trust, because the affected cohort experiences it as consistent rather than random.
Evals frozen at the version that passed. The suite was written against the model or prompt configuration of six months ago. New behaviour has shipped since; the eval does not exercise it. The suite still passes, and its passing carries no information about the current candidate. This is the quietest of the four, because the artefact looks healthy right up to the point where it is inspected for coverage.
Kill-switches that have never been fired. A feature flag exists. Nobody has executed it against production traffic, so nobody knows whether disabling the model leaves the downstream service in a coherent state, whether cached inferences continue to serve, or how long propagation takes. A documented rollback path and a rehearsed one are different artefacts; only the second is evidence.
Ownership diffused across three teams. The model belongs to the ML team, the serving layer to platform, the user-facing surface to product. Each assumes one of the others is watching the AI-specific signals. There is an on-call rota, and it does not cover “the model got worse.” Diffused ownership is the pattern that turns a 40-minute incident into a four-hour one, because the first two hours are spent establishing who should be looking.
Which gate signal catches which pattern
The gate is not a general quality opinion. It is a set of signals, each of which produces a named artefact against a specific release candidate, and each of which maps to one of the patterns above.
| Early-ship pattern | Gate signal that catches it | Evidence the signal must produce |
|---|---|---|
| Aggregate-metric thresholds | Eval coverage stated by slice | Per-segment results with the segment definitions named, plus which segments were not covered and why |
| Stale evals | Eval-to-candidate version binding | The model/prompt version the suite was authored against, and a diff of behaviour added since |
| Untested kill-switch | Rehearsed rollback and disable path | A dated record of an executed rollback, with observed propagation time and downstream state |
| Diffused ownership | Named owner per AI failure mode | A person (not a team) per failure mode, confirmed on the rota, with the alert routed to them |
| Latent post-release risk | Drift baseline recorded pre-rollout | The baselined signals, the thresholds that fire, and who receives the alert |
Read the table as a diagnostic rather than a checklist. If your release process cannot produce the right-hand column for a given row, that row is currently unguarded, and the corresponding pattern in the left column is the one your next incident will come from. This is the decomposition the four gate signals article treats as a readiness framework; here it is being used in reverse, as a failure taxonomy.
Saying no to a fixed date without the gate being dismissed
The objection arrives immediately: a gate that blocks releases will be worked around. That is correct, and it is why the design goal is not blocking. The goal is making disagreement visible, dated, and signed.
When the date and the evidence disagree, three things need to be true. The disagreement is recorded with the specific missing artefact named, not as a general concern. An override is available, because a gate with no override is a gate that gets deleted. And the override has a named owner with the authority to carry the risk — usually the person who owns the launch date, which is precisely the point, because the cost lands where the pressure originated.
A gate that only documents risk and then waves the release through is a different instrument from one that can stop it. Both are better than nothing; only the second changes behaviour. The distinction we hold to is that a small set of gaps — no rollback path, no kill-switch rehearsal, no named owner, no eval coverage on the changed behaviour — are hard fails rather than override candidates. Everything else is negotiable on the record. When to fail a release-readiness review works through where that line sits and what happens after a no-go.
How to tell whether your process is an instrument or theatre
There is one metric that separates the two, and it is uncomfortable: gate-block rate should be non-zero. A release-readiness process that has never failed a candidate is not evidence that the candidates were all sound; it is evidence that the process cannot distinguish. In our reliability work this is the first number we look for, ahead of any eval score.
The supporting measures are ordinary incident metrics read at release granularity: post-release incident count per release, time-to-detect on regressions that slip through, and time-to-rollback once detected. The last two are the ones the gate moves fastest, because a rehearsed kill-switch and a named owner shorten both directly. Incident count moves more slowly, and it moves for reasons other than the gate, so treat it as a trailing indicator rather than proof.
The measurable outcome of a working gate is a release that does not ship — the early-ship failure caught before rollout, avoiding the incident, the emergency remediation sprint, and the trust cost of a user-visible AI regression. That is a hard thing to celebrate internally, which is part of why gates decay. Naming the block rate as a health metric rather than a friction metric is how it survives contact with the roadmap.
For teams standing this up rather than auditing it, the gate has to be executed against a real candidate before it means anything — that is the work our Production AI Monitoring Harness performs, and the wider context for reliability engineering on AI platforms sits with our AI infrastructure and SaaS practice. The instrument is not the checklist; it is the executed run that produces the artefacts in the table above against the version you are about to ship.
The open question we have not solved cleanly: how much of this survives when the model is hosted and changes without notice. A drift baseline assumes the thing you baselined is the thing still serving. When it is not, the gate’s evidence has an expiry date nobody set.
Frequently Asked Questions
What does “AI features ship before they’re ready” mean in practice, and what does it look like inside a real release cycle?
AI Features Ship They is a frequent topic. It means the release went out without evidence that would have justified the decision, not that someone knowingly took a risk. Inside a real cycle it looks unremarkable: a passing eval, a demo that behaved, a fixed date, and a review meeting where nobody could name what would have constituted a no. The failure becomes visible on day 20, when a segment-specific regression or a drift event surfaces and there is no baseline to compare against.
Why does release pressure override engineering judgement on AI features specifically, more than on conventional software?
Conventional software has mechanical brakes — a red build or a failing migration is awkward to override and leaves a trace. AI evals produce a number, and numbers are negotiable under date pressure. Compounding this, most AI failure modes are absent at release time, so a process that asks only “does it work now?” is structurally blind to the risk class that produces the incident.
Which gate signal catches which failure pattern, and what evidence does each signal have to produce?
Slice-level eval coverage catches aggregate-metric thresholds; eval-to-candidate version binding catches stale suites; a rehearsed rollback catches untested kill-switches; a named owner per failure mode catches diffused ownership; and a pre-rollout drift baseline catches latent post-release risk. Each must produce a dated artefact tied to the specific release candidate — per-segment results, an authored-against version, an executed rollback record, a person on the rota, or the baselined signals and their thresholds.
How do we tell whether our release-readiness process is a real instrument or just a checklist nobody has ever failed?
Look at the gate-block rate. If it is zero over many releases, the process cannot distinguish sound candidates from unsound ones, regardless of how thorough the checklist reads. Supporting evidence comes from time-to-detect and time-to-rollback on regressions that did slip through — a real gate shortens both, because the kill-switch and the owner were established before rollout rather than during the incident.
Pressure to ship versus readiness to scale
Product wants velocity, engineering wants observability—the gate process exists because both sides are right. Revisit it when your workload shifts.
