A classical service tells you when it is broken. An AI feature stays up, answers, and is wrong — and nothing on a standard release checklist is instrumented to notice the difference.
That single divergence in failure mode is why reusing a mature SRE release checklist unchanged for a model-backed feature feels safe and is not. The checklist is not wrong. It is a floor, and it was designed against a class of failure — the process dies, the latency spikes, the error rate climbs — that AI features exhibit far less often than the failure they actually exhibit, which is confident, available, incorrect output.
Most teams shipping their first AI feature already own the classical artefact: tests green, canary configured, dashboards wired, on-call rota named, rollback documented. The work is not replacing it. The work is naming the delta — which line items are inherited from SRE practice and carry over unchanged, and which have to be authored specifically for the model.
What carries over unchanged
More than teams expect. This is worth stating plainly, because the opposite error — declaring AI “different” and rebuilding release engineering from scratch — wastes a working control system.
Deployment mechanics carry over completely. Blue/green and canary routing, feature flags, staged percentage rollouts, container image immutability, infrastructure-as-code review, secrets handling, capacity headroom checks, dependency CVE scanning, on-call escalation paths, incident severity definitions, post-incident review process. A model behind an HTTP endpoint is, at the transport and orchestration layer, an ordinary service. Kubernetes does not care that the container holds an ONNX graph rather than a Java process.
Availability, latency and saturation SLOs carry over too — they are just no longer sufficient. Keep them. A model server that OOMs on a long input, or that regresses p99 latency because someone changed the batch scheduler, is a classical failure and the classical checklist catches it correctly.
Why don’t availability and latency SLOs catch model quality regressions?
Because a quality regression is not an error. It is a valid response with the wrong content.
An availability SLO measures whether a response was produced. A latency SLO measures how long it took. A quality regression produces a well-formed, fast, HTTP-200 answer that happens to be worse than last week’s answer for a subset of inputs — a document class that started arriving in a slightly different format, a customer segment underrepresented in the training window, a prompt template edit that quietly broke instruction-following on multi-part questions. Every classical signal reads green.
The measurable consequence is time-to-detect: quality regressions that trip no availability or latency SLO are typically first reported by users, not by monitoring, because no automated signal is watching the axis on which the system actually degraded. That is the gap the model-specific gates exist to close, and it is why release-gate coverage — how many failure modes the checklist can detect before rollout rather than after — is the metric to improve rather than checklist length.
There is a second-order effect we see in release reviews at teams with strong SRE practice. Because every classical item passes cleanly, the review develops false confidence: the gate ran, it was green, therefore the feature is ready. The gate was simply blind on the relevant axis.
The delta: gates a classical checklist structurally misses
Four categories. Each one exists because a property of AI systems has no analogue in a deterministic service.
| Gate | Classical equivalent | Why the classical item is insufficient | Evidence that closes it |
|---|---|---|---|
| Eval coverage over the input distribution | Unit and integration test pass rate | Tests assert fixed input→output pairs. A model’s behaviour is defined over a distribution, and a 100% pass rate on a curated suite says nothing about the slices not in it. | Eval results reported per slice (segment, language, document type, length band), with the uncovered slices named rather than omitted. |
| Drift baselines | Static SLO thresholds | An SLO threshold is absolute and set once. Drift is relative to a reference window that must be captured before rollout, or there is nothing to compare against later. | A recorded pre-rollout baseline for input feature distributions and output distributions, with a threshold, a firing rule, and a named recipient. |
| Non-code artefact versioning and gating | Git commit SHA, image digest | Code version does not pin behaviour. Training data snapshot, weights, prompt template, retrieval index, tokenizer and inference runtime version each change behaviour independently of the commit. | A release manifest binding the code SHA to the data snapshot ID, weights hash, prompt template version, and eval run that produced the approval evidence. |
| Silent-degradation detection | Error rate and exception alerting | The failure produces no exception. Nothing raises. | A quality proxy logged in production — abstention rate, confidence distribution shift, human-override rate, downstream task completion — with a defined alert condition. |
The self-containment test for that table is deliberate: each row should be liftable into your own release-review template without the surrounding prose.
Versioning what is not code
The artefact-versioning row deserves expansion, because it is the item teams most often believe they have already solved.
A conventional release identifies exactly one thing: the commit. For an AI feature the deployable unit is a tuple, and any element can change behaviour while the commit stays fixed. Retraining on a fresher data snapshot changes behaviour. Swapping a hosted model version behind the same API name changes behaviour, sometimes without notice from the provider. Editing three words in a system prompt changes behaviour. Rebuilding a retrieval index changes behaviour. None of these produce a diff a code reviewer would see.
The gate item is a release manifest, not a policy. In practice that means MLflow or a comparable tracking store holds the run that produced the candidate, the data snapshot has an immutable ID (a DVC pointer, a warehouse snapshot timestamp, a versioned S3 prefix), the prompt template lives in version control rather than a config UI, and the release record names all of them together with the eval run ID that generated the approval evidence. The reason to insist on the manifest is not tidiness. It is that rollback without it is guesswork: you can revert the container and still be serving different behaviour, because the index rebuilt overnight.
Our reliability audits usually start here, and the assessment is short — we ask a team to reproduce, from their release record alone, the exact behaviour that shipped six weeks ago. The answer is diagnostic.
Adding the items without turning the review into theatre
Four extra gates is small. Four extra gates phrased as intentions is worthless, and worse than worthless after two cycles, because reviewers learn the rows are decorative.
Three constraints keep the delta honest:
- Each added row names an artefact, not an assertion. “Drift monitoring in place” is theatre. “Baseline recorded at
runs/2026-08-28-eval-441, threshold 0.15 PSI on the top five input features, pages the ML platform on-call” is evidence. If a row cannot name a file, dashboard, or run ID, it is not ready to be a gate. - Each added row has a single named owner, and it is usually not the SRE team. The classical items belong to whoever owns the service. The model-specific items belong to whoever can actually produce the evidence — typically the ML platform lead for drift baselines and artefact manifests, the feature team for eval coverage, and the SRE owner for the rollback and kill-switch rehearsal that both sides depend on. Split ownership is the normal state; unassigned ownership is the failure state.
- Each added row is scoped to the change. A prompt-only change does not need a full retraining-provenance review. Gate on what moved. Reviews that demand identical evidence regardless of blast radius get routed around within a quarter.
The payoff we see when the model-specific evidence items are named up front rather than discovered mid-review is procedural: fewer go/no-go decisions stall for missing evidence, because the team knew before the meeting what it had to bring. The [broader release-readiness framework for production AI](Production AI Monitoring Harness) sets out how those evidence items are executed against an actual release candidate, and the framing here is deliberately narrow — this is the delta list, not the whole gate.
Where the boundary actually sits
If you want one test for whether a checklist item belongs to the classical set or the AI-specific set, use this: does the item detect a failure that announces itself? If yes, SRE practice already owns it and you should not rewrite it. If the failure is silent — the system stays available, responds in time, throws nothing, and is wrong — then no inherited item covers it and you are looking at a gate that has to be authored for the model.
That boundary is not stable forever. As tooling matures, some model-specific gates will become standard platform features and drift back into the inherited set, the way canary deployment did for classical services. The open question for teams building this now is which of the four categories above will still require bespoke authorship in three years — and drift baselines, which depend entirely on knowing what your own inputs looked like before rollout, look like the one least likely to be solved by a platform vendor.
The wider question of how AI reliability engineering fits into a SaaS platform organisation is covered on our AI infrastructure and SaaS practice page.
Frequently Asked Questions
Which items on an existing SRE release checklist still apply unchanged to an AI feature?
Nearly all deployment and operational mechanics carry over untouched: canary and blue/green routing, feature flags, image immutability, capacity headroom, secrets handling, dependency scanning, on-call escalation, incident severity definitions, and post-incident review. Availability, latency and saturation SLOs also stay — they remain correct, they are just no longer sufficient on their own.
Which AI-specific gates does a classical release checklist structurally miss — and why?
Four: eval coverage stated per input slice rather than a test pass rate, a drift baseline captured before rollout rather than a static threshold, versioning and gating on training data, weights and prompts alongside the code SHA, and a silent-degradation signal. Each exists because a property of model behaviour — distributional correctness, reference-relative change, non-code dependency, failure without an exception — has no analogue in a deterministic service.
How do availability and latency SLOs fail to catch model quality regressions?
Because a quality regression is a valid, fast, HTTP-200 response with worse content. Availability measures whether a response was produced and latency measures how long it took; neither reads the answer. The practical consequence is that such regressions are usually first reported by users rather than by monitoring.
Who owns the model-specific gate items when the SRE team owns the classical ones?
Ownership splits by who can produce the evidence: the ML platform lead typically owns drift baselines and the artefact manifest, the feature team owns eval coverage per slice, and the SRE owner keeps the rollback and kill-switch rehearsal. Split ownership across two groups is normal and workable; a row with no named owner is the state that turns the gate into theatre.
Why classical checklists break for AI releases
Unlike deterministic services, AI features ship with residual error baked in, so your go/no-go criteria must quantify acceptable failure modes rather than eliminate defects entirely. If AI Release Readiness vs is on your roadmap, the next step is to map it onto your own constraints rather than copy a reference architecture.
