A release-readiness gate produces paperwork. The question is whether that paperwork can be checked. An evidence pack is the dated, versioned bundle assembled against one specific release candidate, and every item in it has to be verifiable by someone who was not in the room when it was produced.
Most teams get the order wrong. They write the checklist first, then hold the review, and the review runs on assertions: monitoring is in place, we can roll back, someone owns this. Each of those sentences is a claim about the world that nobody has tested. The eval number is usually the only item an approver can actually inspect, which is why it ends up carrying the whole decision.
The pack fixes the asymmetry by making each item falsifiable. An approver can either check that a thing exists and was exercised, or they can only check that someone said it was. That is the entire divergence point, and it applies to every row.
What does a release-readiness evidence pack contain, in practice?
Five artefacts, each attached to a release candidate identifier rather than to a date range or a team name.
| Artefact | What it must contain | The falsifiability test |
|---|---|---|
| Eval results | Headline metric plus per-slice breakdown, coverage stated as which slices were measured and which were not, and the baseline version compared against | Can a reviewer name a slice the eval did not cover? |
| Drift telemetry plan | The named signals logged in production, the baseline distribution each is compared against, the threshold that fires, and the rota the alert reaches | Is there a signal in the plan that is not yet emitting in staging? |
| Rollback runbook | Ordered steps, the artefact versions rolled back to, expected duration, and a dated record of at least one staging execution | Who ran it, on what date, and how long did it take? |
| On-call ownership matrix | Named humans (not teams) against the AI-specific failure modes, with escalation and out-of-hours cover | Can the approver page the right person in under two minutes? |
| Kill-switch rehearsal record | The switch’s mechanism, who is authorised to flip it, and a dated rehearsal result | Was the feature actually disabled in a live-like environment? |
The pack is a bundle of artefacts with owners and dates, not a slide deck. If an item cannot be attached as a file, a dashboard link, or a timestamped log, it is not evidence yet.
Eval results: coverage is the claim, not accuracy
A single accuracy figure tells an approver almost nothing about the release. What matters is which parts of the input distribution were measured, and — more usefully — which were not. We ask for coverage stated as an explicit inclusion and exclusion list: these fifteen slices were evaluated with sample counts, these four were not, and here is why.
The exclusion list is the part reviewers learn to read first. A model that scores well overall and was never evaluated on the segment where the feature will see most of its traffic is not a passing eval; it is an untested release with a reassuring number attached. In our experience, the slices that get skipped are the ones that are expensive to label, which is also where quality regressions tend to surface after rollout (observed pattern across TechnoLynx engagements; not a benchmarked rate).
Comparison baseline belongs in the same artefact. Each eval result names the previous production version it was scored against, using the same harness and the same slice definitions. That is what turns the current pack into the reference point for the next one.
The drift telemetry plan: signals, thresholds, and who wakes up
Monitoring is in place is not a plan. A drift telemetry plan is a table of rows, and each row has four fields: the signal, the baseline it is compared to, the threshold that fires, and the destination the alert reaches.
Signals worth naming explicitly include input-feature distribution shift, prediction-distribution shift, the rate of low-confidence or abstained outputs, upstream schema and null-rate changes, and — where a feedback loop exists — delayed label agreement. Thresholds have to be recorded before rollout, because a threshold chosen after the first anomaly is a threshold fitted to the anomaly.
The destination field is the one that fails review most often. An alert that routes to a Slack channel nobody owns is telemetry without a consumer. Time-to-detect on a drift regression is bounded by whether the signal was named before release — no amount of retrospective log-diving recovers a signal that was never logged. This is where the harness layer matters in practice: emitting the signals, holding the baselines, and firing to a named rota is what a [production AI monitoring harness](Production AI Monitoring Harness) is for, and it is why the plan and the instrumentation should ship together rather than in sequence.
What makes a rollback runbook credible?
Execution. A rollback runbook that has never been run is a hypothesis written in imperative mood.
Rehearsal in staging means someone followed the written steps, in order, without improvising, and recorded what happened. Three things usually break on the first attempt: the runbook rolls back model weights but not the prompt or feature-transform config, so the restored system is a combination that was never evaluated; a step depends on credentials or a console the on-call engineer does not hold; and the measured duration is several times the estimate, which changes the incident-response maths.
The record itself is short — operator name, date, candidate version, elapsed time, deviations from the written steps. That last field is the valuable one. Deviations are how the runbook improves between releases. Time-to-rollback is bounded by whether the runbook was rehearsed, not by whether it was written well.
The kill-switch rehearsal is a separate record for a reason. Rollback returns the system to a previous known state; the kill switch removes the AI behaviour entirely and falls back to a deterministic path or a graceful decline. Different mechanism, different authorisation, different test.
Ownership, versioning, and waivers
On-call ownership is recorded as named individuals against failure modes — degraded output quality, drift alert, latency or cost blow-out, upstream data break — with escalation and out-of-hours cover. Team names are not owners. The practical target is that an approver reading the pack can identify who to page within minutes rather than hours, which is a property of the matrix, not of the org chart.
Versioning keeps the pack cheap to refresh. Each item carries the model, dataset, and config version it was produced against, so when the model version changes, the review is a diff rather than a rebuild: eval results are re-run, drift baselines are recomputed, and the rollback runbook and ownership matrix are re-confirmed unless the deployment path changed. Teams that keep the pack versioned find each subsequent review shorter than the last, because the previous pack supplies both the baseline and the format.
Waivers are legitimate and should be written down. An item can be waived by engineering judgement when the risk it covers is genuinely absent — a feature with no user-visible output may not need a kill switch distinct from its rollback path. A waiver records what was skipped, who accepted the risk, why, and when it expires. An undated waiver is how a gate quietly becomes theatre. Which items are never waivable is a decision the review process itself has to settle in advance; we look at that boundary in when to fail a release-readiness review.
Where the pack sits in the wider gate
The pack is the output of the readiness gate, not the gate itself. The parent framework — the four signals that decide whether an AI feature is ready to ship — sets out which commitments a release has to make and why model quality alone cannot carry the decision. This article is about the artefacts those commitments leave behind, and the standard each has to meet to be worth reviewing. For teams running AI features as part of a platform product, the same evidence discipline shows up across the AI infrastructure and SaaS delivery pattern.
The honest uncertainty is where the pack stops paying for itself. Five artefacts, versioned and rehearsed, is a load a small team can carry per release. Twenty rows with per-slice sign-off is not, and packs that grow that way tend to get filled in from memory — the exact failure the pack was built to prevent. We do not have a clean rule for the ceiling. What we do watch for is the first review where an item is attached but nobody can say when it was last exercised.
Frequently Asked Questions
What does a release-readiness evidence pack contain for an AI feature, in practice? Five artefacts attached to a specific release candidate: eval results with coverage stated by slice, a drift telemetry plan naming signals and thresholds, a rollback runbook with a dated staging execution, an on-call ownership matrix with named humans, and a kill-switch rehearsal record. Each item must be attachable as a file, dashboard link, or timestamped log.
What do eval results need to show beyond a headline accuracy number, and how is coverage stated? Coverage is stated as an explicit inclusion and exclusion list — which slices were evaluated with sample counts, which were not, and why. The exclusion list is the part reviewers should read first. The result also names the previous production version it was scored against, using the same harness and slice definitions.
What belongs in the drift telemetry plan — which signals, which thresholds, and who gets paged? Each row names a signal (input-distribution shift, prediction-distribution shift, low-confidence rate, upstream schema or null-rate change), the baseline it is compared against, the threshold that fires, and the named rota the alert reaches. Thresholds are recorded before rollout; a threshold chosen after the first anomaly is fitted to that anomaly.
What makes a rollback runbook credible, and what does rehearsing it actually involve? Credibility comes from execution: someone followed the written steps in staging, without improvising, and recorded operator, date, candidate version, elapsed time, and deviations. Rehearsal usually exposes three gaps — config not rolled back alongside weights, missing credentials, and a duration several times the estimate.
How is on-call ownership recorded so an approver can see who owns the feature after release? As named individuals — not teams — mapped to the AI-specific failure modes, with escalation paths and out-of-hours cover. The working test is whether an approver reading the matrix can page the right person in minutes rather than hours.
How does the pack get versioned and refreshed when the model version changes? Every item carries the model, dataset, and config version it was produced against, so a new model version makes the next review a diff: re-run evals, recompute drift baselines, re-confirm the runbook and ownership matrix unless the deployment path changed. The previous pack supplies both the baseline and the format.
Which items can be waived by engineering judgement, and how should a waiver be documented? An item can be waived when the risk it covers is genuinely absent for that feature — for example, a kill switch distinct from rollback on a feature with no user-visible output. The waiver records what was skipped, who accepted the risk, the reasoning, and an expiry date. Undated waivers are how a gate degrades into theatre.
Six artifacts every evidence pack needs
Assemble evaluation results, error-category breakdowns, latency percentiles, rollback runbooks, monitoring dashboards, and a one-page risk summary before your launch review. If Release Readiness Evidence Pack is on your roadmap, the next step is to map it onto your own constraints rather than copy a reference architecture.
