A monitoring harness section is not a paragraph of description. It is a claim, and every claim needs a retrievable artefact behind it — a versioned dataset, a run ID, a baseline with its computation window, a signed review record. The test is blunt: when someone external asks “show me the run that produced this number,” either you can open it or you cannot.
That question usually arrives late. A customer reliability questionnaire, an internal audit, a regulator, or simply the first retrain after handoff. By then it is too late to build the evidence trail retroactively, because the measurement conditions no longer exist. This is why we treat evidence as a property of each harness section rather than an appendix at the end of the deliverable.
Why does a harness section without evidence fail at model update time?
Because the update is precisely when the section has to be re-executed rather than re-read. A section that says “the model achieves 94.1% precision on the acceptance slice” without naming the dataset version and the run that produced it cannot be compared against the retrained model. You are left with two numbers and no way to know whether the difference came from the model, the data, or the metric definition.
A harness whose sections have no evidence trail cannot be re-verified after a model update — which is exactly the moment reliability claims are being relied upon. The practical consequence is a documentation exercise: someone spends weeks reconstructing what was measured, and the reconstruction is unverifiable by definition. Evidence-per-section discipline replaces that with a re-run of named suites.
The second failure is quieter and more common at handoff. A section states a threshold — alert at 0.08 population shift, fail the gate below 0.91 recall on the pedestrian slice — and nobody can point to the measurement that produced the number. It was chosen in a meeting. Nothing is wrong with a judgement-based threshold, but it must be recorded as one, with the reasoning attached, rather than presented as if it were derived.
The evidence table, section by section
Each row states what counts as evidence, where it lives, and who is accountable for keeping it current after a model update. The storage locations below are the pattern we most often see hold up in practice; the specific tools matter less than the requirement that the artefact be addressable by an identifier a reviewer can paste into a search box.
| Harness section | What counts as evidence | Where it is stored | Accountable owner |
|---|---|---|---|
| Eval harness | Versioned eval dataset (content hash or dataset version tag), the metric definitions as code, and the run ID that produced each reported figure. Reported metrics alone are not evidence. | Dataset registry or object store with immutable versions; runs in the experiment tracker (MLflow, Weights & Biases) | ML lead who owns the eval suite |
| Regression suite | Frozen baseline results keyed to a specific model version, fixed seeds and pinned environment (container digest), plus the pass/fail record of the latest run. | CI system artefacts, with baselines committed alongside the suite code | Engineering lead |
| Drift telemetry | The baseline distribution and its computation window (which dates, which population, which feature set), the threshold, and a written note on how the threshold was chosen. | Monitoring store for live signals; baselines versioned in the repo or feature store | MLOps owner |
| Alert quality | Precision, recall and false-alarm rates measured over a named period with a named incident list — “Q2, 41 alerts, 9 true positives adjudicated by the on-call rota” — not a hypothetical rate. | Incident tracker export plus the adjudication sheet, both dated | Operations owner on the customer side |
| Release-readiness review | A signed pass/fail record naming the model version, the suites executed, the exceptions granted and by whom. | Release record in the repo or change-management system | Named reviewer per section (QA for acceptance criteria) |
| Audit-evidence pack | An index that resolves every claim in the harness to one of the artefacts above, with the retrieval path for each. | Alongside the harness document, versioned with it | The engagement’s delivery owner |
Two rows deserve a note. Drift baselines fail more often than any other section, because the baseline is computed once, the window is never recorded, and a year later nobody can say whether the reference population included the seasonal peak. And alert-quality evidence is the row buyers almost never ask for — a false-alarm rate quoted without a period and an incident count is an opinion.
The reviewer’s four-question test
A reviewer does not need the domain expertise to re-derive a threshold. They need to establish whether an artefact exists. Four questions catch most untraceable sections:
- What identifier retrieves this number? A run ID, a commit SHA, a dataset version. “The evaluation notebook” is not an identifier.
- Over what window and population was it measured? If the answer is absent, the number cannot be reproduced even with the code.
- If the model is retrained tomorrow, what re-executes? Name the suite. If nothing executes, the section is descriptive prose.
- Who is accountable for this section’s evidence being current? A role, not a team. We treat per-section accountability as a separate design decision from evidence definition, and it is covered in who signs each section of a monitoring harness.
The point of the test is that a buyer can run it before signing scope. Ask a reliability vendor to show the evidence layer for one section of a previous engagement — with the client specifics redacted — and the answer separates teams that produce executable harnesses from teams that produce documents.
What this buys, and what it does not
The return is re-verifiability. A model update triggers a re-run of named suites instead of a fresh authoring cycle, and an audit or customer questionnaire can be answered from stored run artefacts in days rather than reconstructed from memory. That efficiency is the same mechanism as the reliability improvement: the harness stays connected to measurements, so a regression shows up as a failed suite instead of a surprise in production.
What it does not buy is correctness of the thresholds themselves. Evidence discipline proves a number came from somewhere; it does not prove the number is right for the operational context. A drift threshold traceable to a well-documented baseline can still be set too loose for the decision the model drives. That judgement belongs to the release gate, and the gate consumes exactly this per-section evidence to make it — which is why an incomplete evidence layer usually shows up first as a release decision nobody feels able to sign.
There is also a real cost. Immutable dataset versions, pinned environments and adjudicated incident lists are ongoing work, not one-time setup. In our experience the discipline pays off from the second model update onward, and rarely before the first — which is a reasonable argument for building the evidence layer at harness-design time rather than adding it later, when reconstruction is the only option left.
The broader structure — which sections a harness contains, and why the section list is what makes reliability work comparable across engagements — is developed in our coverage of production AI reliability. The open question we keep returning to: how much evidence is enough for a section before the marginal artefact stops reducing risk and starts producing paperwork? We have not found a general answer, only per-domain ones.
Frequently Asked Questions
What does “what evidence belongs behind each monitoring harness section” mean in practice? For Evidence Belongs Behind Monitoring, the question is straightforward. It means every section of the harness names a retrievable artefact rather than describing an activity. A section on the eval harness cites the dataset version and run ID; a section on drift cites the baseline, its computation window and the threshold rationale. In practice the test is whether a reviewer can retrieve the artefact from an identifier without asking the author.
What counts as acceptable evidence for an eval harness section — datasets, run IDs, or reported metrics alone? Reported metrics alone are not evidence, because they cannot be reproduced or compared across model versions. Acceptable evidence is the versioned eval dataset (immutable version tag or content hash), the metric definitions as executable code, and the run ID that produced each figure. All three together let a reviewer re-execute the measurement.
What evidence backs a regression suite section, and how are baseline results versioned against model versions? The evidence is the frozen baseline result set, keyed to a specific model version, together with fixed seeds and a pinned environment such as a container digest. Baselines are committed alongside the suite code so that the pairing of model version to expected behaviour is visible in version history. The latest run’s pass/fail record completes the trail.
What evidence supports drift telemetry claims, including baseline windows, thresholds and the reasoning behind them? Three artefacts: the baseline distribution itself, the window and population it was computed over, and a written note on how the threshold was chosen. The window is the part most often missing, and without it the baseline cannot be reproduced or defended. A judgement-based threshold is acceptable evidence provided it is recorded as a judgement rather than presented as derived.
How is alert-quality evidence produced — precision, recall and false-alarm rates measured over which period? By adjudicating a named set of alerts over a named period — for example every alert raised in a stated quarter, each marked true or false positive by the on-call rota. The evidence is the incident-tracker export plus the dated adjudication sheet, not a rate quoted without a denominator. A false-alarm rate with no period and no incident count is an opinion.
Where is harness evidence stored, and who is accountable for keeping each section’s evidence current after a model update? Datasets in a versioned registry or object store, runs in the experiment tracker, regression baselines in the repository beside the suite, live drift signals in the monitoring store, and adjudicated incidents in the incident tracker. Accountability is assigned per section to a named role — ML lead, engineering lead, MLOps owner, customer operations owner — because the sections need different competences to keep current.
How does a reviewer spot a harness section whose stated threshold has no traceable measurement behind it? Ask what identifier retrieves the number, over what window and population it was measured, and what re-executes if the model is retrained tomorrow. A section that cannot answer the first question is descriptive prose regardless of how precise the number looks. The tell is a specific figure paired with a vague source, such as “based on our evaluation”.
The honest read on Evidence Belongs Behind Monitoring
Evidence Belongs Behind Monitoring is rarely the hard part — knowing which of its failure modes you can live with is. That answer is workload-specific, and it is worth writing down before you build.
