Ask one question of any artefact that claims to demonstrate reliability work: can it be executed, re-run against a new model version, and failed? A model card cannot. A dashboard screenshot cannot. A handoff deck with accuracy tables cannot. Each of those describes model behaviour at a single moment; none of them produces a verdict when the model behind them changes.
That question is the whole distinction, and it is worth applying early. The expensive version of this mistake is accepting a summary artefact at handoff, then discovering at the first retrain that there is nothing to run.
What “a monitoring harness is not a model card” means in practice
A model card is authored. Someone writes down the intended use, the training data provenance, the known limitations, the headline metrics, and the ethical considerations. It is a document, and its correctness is a property of the moment it was written.
A monitoring harness is executed. It consists of eval suites that run against a candidate model, regression suites with frozen baselines and pass/fail assertions, drift telemetry with thresholds attached to actions, alert-quality tuning grounded in measured precision, and a release-readiness review that emits a signed pass or block. Its correctness is a property of the last time it ran.
The two artefacts diverge at model update time. Retrain the model, and the model card silently goes stale — the numbers in it now describe a model that is no longer deployed, and nothing in the document knows that. The harness, on the other hand, re-runs. It either signs the new version off or blocks it. The harness has an opinion about a model it has never seen; the card does not.
We see this land badly most often in teams that did everything else well. The documentation is genuinely good — thorough limitation sections, honest error analysis, clean lineage notes in MLflow. What is missing is the CI job that would have caught the regression on the third retrain.
The executable/signable test
Two properties, applied to each claimed deliverable. An artefact belongs in the harness only if it has both.
Executable — there is a command, a pipeline step, or a scheduled job that runs it. Not a person who reads it. If the artefact requires human interpretation to produce a result, it is a summary.
Failable — running it can produce a negative outcome that blocks something. A metric that is only ever recorded, never compared to a threshold with a consequence attached, is telemetry theatre.
| Claimed deliverable | Executable? | Failable? | Verdict |
|---|---|---|---|
| Model card (intended use, limitations, training data) | No | No | Summary — legitimate, but not the harness |
| Handoff deck with accuracy tables | No | No | Summary |
| Screenshot of a metrics dashboard | No | No | Summary |
| Live dashboard with no thresholds or owners | Partly | No | Telemetry without a decision |
| Eval suite pinned to a versioned dataset | Yes | Yes | Harness section |
| Regression suite with frozen baselines and slice assertions | Yes | Yes | Harness section |
| Drift monitors with thresholds wired to an action and an owner | Yes | Yes | Harness section |
| Release-readiness review producing a pass/fail record | Yes | Yes | Harness section |
The parent hub develops why these sections belong together and what a complete section list looks like — see what a production AI monitoring harness actually contains for the full shape and how it maps to a validation pack.
Where a model card is genuinely useful — and where it stops
None of this is an argument against model cards. They do a job nothing else does. A card records the intended operating domain, the populations the model was and was not trained on, the licence and provenance of the training data, and the limitations the team knew about at release. That is context, and context is not reproducible by re-running a test. An auditor reading a regression suite six months later cannot recover from it that the model was never intended for paediatric cases; a model card can carry that.
The usefulness stops at the point where a decision has to be made about a specific new model version. Ask a card whether build 47 is safe to promote and it has nothing to say. It was written about build 12. It has no mechanism for noticing that build 47 exists.
The clean way to hold both is to treat the card as one input to the harness rather than a competitor to it. The release-readiness review reads the card to check that the new build is still inside the declared operating domain, and reads the eval and regression results to check that it behaves. Neither substitutes for the other.
What breaks at the first retrain
The failure is quiet, which is what makes it expensive. Nothing errors. The deck is still in the shared drive. The dashboard still renders. Someone retrains on three months of fresh data, deploys, and the only thing that changed is that every artefact describing the system is now describing the previous system.
A few things are typically discovered in the same week:
- There is no held-out eval set anyone can point to as the eval set — several notebooks contain variants, none versioned.
- The accuracy table in the deck cannot be reproduced, because the split that produced it was not recorded.
- Drift monitors exist but their thresholds were tuned against the old model’s output distribution, so they either fire constantly or never.
- Nobody has authority to block the release, because there is no artefact that produces a block.
Rebuilding from that position is not a documentation task. It is a fresh engineering cycle: reconstruct the eval corpus, re-derive baselines, re-tune thresholds, and re-establish who signs. Teams that inherited a real harness instead measure the cost of a model update in CI minutes per version, not in review cycles.s.s.
Which parts can never be a document
Some harness sections are structurally impossible to represent as prose, and this is a useful sharpening of the test rather than a separate rule.
Alert quality is one. The measured property is precision over a named incident window — how many pages were real. That number only exists if alerts fired against real traffic and someone adjudicated them. A document can state a target; it cannot hold the measurement, because the measurement is produced by the running system.
Regression coverage is another. What matters is not a list of test names but the fact that the suite executed deterministically against a specific model artefact and reported per-slice results. Determinism is a runtime property. In our experience this is the section vendors most often describe and least often ship — the description is easy and the fixed-seed, pinned-dependency execution environment behind it is not.
The release-readiness record is the third. Its value is that it is dated, attributable, and tied to one model version. Rewriting it as a general policy statement destroys exactly the property that made it evidence.
Wording acceptance criteria so this cannot be handed over
The correction is contractual, and it is short. Do not ask for “monitoring” or “documentation of model behaviour”. Ask, per section, for an artefact that satisfies four conditions:
- A command or pipeline step the client can run themselves after handoff, in an environment the vendor specifies.
- A named model version it was last executed against, with the run record retained.
- An explicit pass/fail criterion — the condition under which it blocks a release, not just the metric it reports.
- A named reviewer who signs the result for that section.
A proposal that cannot answer all four for a given section is offering a summary of that section. That is worth knowing at first review rather than at handoff, and it is the cheapest filter available in a reliability procurement. Buyers who apply it can reject non-comparable proposals in the first pass; the alternative is discovering the gap during the week the harness was supposed to start earning its keep.
The harder question sits one level down, and it is where we usually end up in scoping conversations: for each section you are asking a vendor to build, what specific model failure would it have caught last quarter? If nobody can name one, the section may be well-specified and still be the wrong section.
Frequently Asked Questions
What does “a monitoring harness is not a model card or a slide deck” mean in practice?
Unlike model cards and slide decks, monitoring harnesses execute runtime validation rather than documenting static metadata. It means the deliverable that constitutes reliability work is something that runs, not something that is read. A model card and a handoff deck describe how a model behaved at one point in time; a harness re-executes against each new model version and returns a verdict. At retrain time the described artefacts go stale silently while the harness either signs off or blocks.
What is the concrete test that separates an executable harness from a descriptive summary artefact?
Two properties together: executable (there is a command, pipeline step, or scheduled job that runs it without human interpretation) and failable (running it can produce a negative outcome that blocks a release). Eval suites, regression suites with frozen baselines, thresholded drift monitors, and release-readiness reviews satisfy both. Cards, decks, and dashboard screenshots satisfy neither.
What does a model card legitimately cover, and where does its usefulness stop?
A model card carries context that no test can reproduce: intended operating domain, training-data provenance and licence, populations excluded, and known limitations at release. That is genuinely valuable to an auditor reading the system months later. It stops being useful the moment a decision is needed about a specific new build, because the card was written about a previous one and has no mechanism for noticing.
What breaks at the first model retrain when the deliverable was a deck or a dashboard screenshot?
Nothing errors, which is the problem. The team typically discovers in the same week that no versioned eval set exists, that the accuracy table cannot be reproduced because its split was never recorded, that drift thresholds were tuned to the old output distribution, and that no artefact has the authority to block the release. Recovery is a fresh engineering cycle, not a documentation update.
Which parts of a harness can never be represented as a document — and why?
Alert quality, because its measured property is precision over a named incident window, which only exists once alerts have fired against real traffic and been adjudicated. Regression coverage, because determinism against a pinned model artefact is a runtime property, not a list of test names. And the release-readiness record, because its evidential value comes from being dated, attributable, and tied to one model version.
How should a buyer word acceptance criteria so a summary artefact cannot be handed over as the harness?
Specify per section: a command the client can run post-handoff, the named model version it was last executed against with the run record retained, an explicit pass/fail criterion that blocks a release, and a named reviewer who signs it. A vendor who cannot answer all four for a section is offering a description of that section rather than the section itself.
Harness, card, deck: which owns runtime decisions
A model card documents design intent; a monitoring harness enforces live boundaries — confusion between the two leaves nobody empowered to pull the brake. The teams that do tend to ship the boring, correct version first.
