A RAG feature has no single accuracy number you can approve against. That is the whole difficulty. A classifier gives you one score on a held-out set and a reviewer can argue about whether 0.91 is enough; a retrieval-augmented feature gives you a prompt suite that looked good in review, a vendor eval score produced on someone else’s corpus, and a demo that happened to answer the hard questions. Teams approve on that basis and file retrieval quality, grounding and prompt regressions as things to watch after rollout.
The gate does not change. What changes is how it is instantiated. A RAG release candidate has two moving parts you own — the retriever and the generator — and a third you do not: the hosted model version, which can change without a deploy on your side. This article walks the same release-readiness gate through those three parts, using a support-answer feature over an internal documentation corpus as the worked example.
What does release-readiness mean when the release candidate is a RAG feature?
It means the gate’s evidence is produced against a pinned tuple, not against “the feature”. The tuple is: corpus snapshot ID, embedding model version, index build ID, top-k and reranker config, prompt template version, and provider model version pin. Change any element and the eval evidence expires.
That is the first practical divergence from a single-model release. With a classifier, the reviewer asks which weights. Here the reviewer asks which six things, and if the answer is incomplete the gate has nothing to approve. Eval evidence for a retrieval-augmented feature is only valid against the exact retriever-plus-generator-plus-model-pin tuple it was produced on; a change to any one element invalidates it. We have seen a corpus re-index alone move answer quality more than a prompt rewrite, which is why the index build ID belongs in the pack and not in a runbook footnote.
Splitting eval coverage across retrieval and generation
A single end-to-end score tells you the answer was wrong. It does not tell you whether the retriever failed to surface the right passage or the generator ignored a passage it was given. Those are different fixes, owned by different people, and conflating them is how RAG post-mortems stall.
So the eval set is scored twice on every run:
| Layer | Signal | What a failure means | Fix owner |
|---|---|---|---|
| Retrieval | Recall of the gold passage within the top-k the feature actually uses | The context never contained the answer | Retriever / index config |
| Retrieval | Rank of the gold passage inside k | Answer present but crowded out by near-duplicates | Reranker / chunking |
| Generation | Grounding-failure rate — assertions not supported by retrieved context | Model invented content it had no source for | Prompt / model choice |
| Generation | Refusal rate on answerable questions | Feature is uselessly cautious | Prompt / threshold tuning |
| Generation | Refusal rate on unanswerable questions | Feature answers when it should decline | Prompt / guardrail |
Two details matter more than the table suggests. First, retrieval recall must be measured at the deployed k, not at k=50 where recall always looks comfortable — a recall figure taken at a k the feature does not use is not evidence about the feature. Second, refusal is measured in both directions. A release that drops grounding failures to near zero by refusing a third of answerable questions has not passed; it has traded one failure for a quieter one.
The unanswerable slice is the one teams skip because it is tedious to build. It is also where the release candidate’s real behaviour lives, and where the four gate signals described in the parent framework on when an AI feature is ready to ship get their teeth for a RAG system.
Drift baselines: corpus, queries, and the model you do not control
Classical drift monitoring watches the input distribution. A RAG feature has three distributions drifting independently:
- Corpus drift — documents added, edited, deprecated. A stale passage that still ranks first is a grounding failure with a correct-looking citation. Baseline: document count, chunk count, and mean age per source, snapshotted at release.
- Query drift — users asking things the eval set does not represent. Baseline: intent-cluster distribution over the first fixed window after rollout, compared to the eval set’s coverage.
- Provider model drift — the hosted model behind the same API name behaves differently after a provider update. Baseline: a small canary prompt set with deterministic settings, replayed on a schedule, with output diffs alerting rather than a quality score alerting.
The third is the one with no equivalent in a conventional release, and it is the reason a pinned model version belongs in the evidence pack even when the provider offers pinning. Pins expire. A hosted model version is a runtime dependency that can change without a deployment on your side, so the release-readiness gate must carry a detection path for it, not only a version pin. Time-to-detect on a provider-side change is a signal you can state a target for; when we assemble monitoring for retrieval-augmented features, that canary replay is usually the cheapest instrument on the list and the one nobody had.
The kill-switch when the LLM path is the feature
For a classifier you can often fall back to a previous model. For a RAG feature the switch has to answer a harder question: what does the user see instead? A blank screen is not a degraded path, it is an outage with a nicer name.
The rehearsal — and it is a rehearsal, not a documented procedure — covers three levels:
- Roll back the tuple. Restore the previous prompt version, index build and model pin together. Measured as time-to-rollback; partial rollbacks that restore the prompt but not the index are the common failure.
- Degrade to retrieval only. Serve ranked source passages with citations and no generated summary. Less useful, still useful, and it removes the generator from the risk surface entirely.
- Degrade to the non-LLM path. Keyword search over the same corpus, or the pre-existing support flow. Ugly, and the one users can still complete their task with.
Level 2 is what makes this worth building. Because retrieval and generation are separable, a RAG feature has a genuine intermediate degradation that a single-model feature does not, and it is available at the flip of a flag if — and only if — the retrieval-only view was built and tested before release rather than sketched during the incident.
The cost signal RAG makes visible
One thing a retrieval-augmented release exposes that a classifier release does not: unit economics per request. Tokens consumed and retrieval calls made per resolved request — not per API call, since a request that took three turns to resolve cost three times what the dashboard suggests. A prompt change that adds 40% more context to lift grounding accuracy is a quality win and possibly a margin loss, and the gate is the last place to notice that before the invoice does.
Recording cost-per-resolved-request at the gate turns a later argument into a comparison against a baseline. It also gives the reviewer a legitimate reason to fail a candidate that passed on quality, which is uncomfortable and correct.
What the evidence pack contains, and who signs it
The pack our [production-AI monitoring and validation work](Production AI Monitoring Harness) is assembled around holds:
- the pinned tuple (six elements above), dated;
- retrieval scores at deployed k, by corpus source and query slice;
- grounding-failure and bidirectional refusal rates against a fixed eval set;
- the three drift baselines, with thresholds and the named person each alert pages;
- the kill-switch rehearsal record — who ran it, when, what the degraded path served, measured time-to-rollback;
- cost per resolved request at candidate and at baseline.
Sign-off splits along the same seam as the eval: the retriever/index owner signs the retrieval evidence, the feature owner signs grounding and refusal, and whoever carries the on-call pager signs the rehearsal record. One signature across all of it means nobody checked the half they do not own.
None of this removes engineering judgement about how much grounding error the feature can carry. A support-answer feature with visible citations can tolerate error a billing-explanation feature cannot. The gate’s job is to make that tolerance an explicit, dated, owned number before the release fires — rather than a number the first cohort of users discovers on your behalf. For teams shipping these features inside an AI infrastructure or SaaS platform, that is usually the difference between a launch and a launch you can defend.d.
Frequently Asked Questions
What does release-readiness applied to a RAG or LLM-augmented feature mean in practice? Release Readiness RAG LLM is best answered directly. Release Readiness RAG LLM has one honest answer. It means the gate’s evidence is produced against a pinned tuple — corpus snapshot, embedding model, index build, top-k and reranker config, prompt version, provider model pin — rather than against “the feature”. Change any element and the evidence expires and the gate must be re-run.
How do we build eval coverage that separates retrieval failures from generation failures? Score every eval run twice: retrieval recall and gold-passage rank at the top-k the feature actually deploys, then grounding-failure rate and refusal behaviour on the same items. A wrong answer with the gold passage in context is a generation defect; the same answer with the passage missing is a retrieval defect, and the fixes sit with different owners.
What drift baselines matter for a RAG feature — corpus, query distribution, or provider model version? All three, and they drift independently. Snapshot document and chunk counts plus mean source age for the corpus, intent-cluster distribution for queries, and a deterministic canary prompt set replayed on a schedule to catch provider-side model changes that arrive without a deploy on your side.
What does a rehearsed kill-switch look like when the LLM path is the feature, and what does the degraded path serve instead? Three tested levels: roll the whole tuple back together, degrade to retrieval-only (ranked cited passages, no generated summary), or degrade to the pre-existing non-LLM flow. The retrieval-only level is the one worth building in advance, because it removes the generator from the risk surface while leaving the user able to finish the task.
How do we handle a hosted model version changing under a release that already passed the gate? Treat the hosted version as a runtime dependency with its own detection path, not as a settled pin. The canary replay diffs outputs on fixed inputs and alerts a named owner; the response is the level-1 rollback to the previous tuple while the candidate is re-evaluated against the new provider behaviour.
How does this differ from running the gate on a conventional single-model feature? Three ways: eval coverage splits across two components instead of scoring one, drift monitoring tracks a corpus and a third-party model rather than only an input distribution, and the degraded path has a genuine intermediate level. Cost per resolved request also becomes a gate signal, because token and retrieval spend scale with quality decisions.
What belongs in the evidence pack for a RAG release candidate, and who signs it off? The dated pinned tuple, retrieval scores at deployed k by slice, grounding and bidirectional refusal rates, the three drift baselines with thresholds and named alert owners, the kill-switch rehearsal record with measured time-to-rollback, and cost per resolved request at candidate and baseline. Sign-off splits by seam — index owner, feature owner, and on-call owner each sign the part they can actually verify.
Implementing guardrails for RAG-augmented releases
Every RAG LLM feature requires specific validation checkpoints before production deployment. Release Readiness RAG LLM rewards teams that measure first and argue later — start with the smallest instrumented slice and let the numbers settle the design.
