The section list of a monitoring harness is portable. The contents are not. Copy an eval suite, a drift dashboard and an alerting policy from a computer-vision inspection model onto a retrieval-backed LLM assistant and you will get a regression suite that passes green every week while answer quality quietly collapses. Copy the LLM harness onto a perception stack and you get sampled spot checks where scenario coverage was the actual requirement.

So the useful question in a scoping conversation is not “do you have a monitoring harness?” It is “which variant of each section did you build, and what failure mode did that choice come from?” The six sections — eval harness, regression suite, drift telemetry, alert-quality tuning, release-readiness review, audit-evidence pack — stay fixed. What fills them is determined by what the workload can actually fail at.

What stays fixed and what always changes

The fixed part is structural: every workload needs an executable eval harness, a regression suite with frozen baselines, drift telemetry with thresholds, a tuned alert policy, a signed release-readiness review, and an evidence pack that can be retrieved later. That list does not bend per workload, and a vendor who drops a section because “it does not apply to LLMs” is usually dropping it because it is hard.

The variable part is four-fold, and it is the same four things every time:

• The grading unit — what one row of the eval set is: a frame, a prompt-response pair, or a scenario.
• The ground truth — pixel or box labels, a rubric plus reference answers, or an annotated temporal sequence.
• The drift signal — which population you watch for shift, and against which baseline window.
• The actionable alert — what an on-call engineer can do at 02:00 when the page fires.

If a proposal describes the six sections but never names these four per workload, it is a reskinned template. That is a claim we would defend in any review: a harness is workload-appropriate only when the grading unit, the ground truth, the drift signal and the alert action are separately specified for each model class in the estate.

What does the harness look like in practice for each workload?

Section	CV inspection model	LLM / retrieval workload	Perception stack
Grading unit	Image or crop	Prompt–response pair, plus retrieved context	Scenario over a temporal sequence
Ground truth	Pixel masks, boxes, defect class	Rubric + reference answers; human adjudication on disagreement	Annotated sequences with track IDs and fused object state
Primary eval metric	Slice-level precision/recall at operating threshold	Rubric pass rate, groundedness, refusal correctness	Scenario pass rate, track continuity, fusion association error
Regression suite	Frozen golden set + known-defect cases, fixed seeds	Frozen prompt corpus with pinned retrieval snapshot; band assertions, not equality	Frozen scenario library covering known long-tail conditions
Drift signal	Input distribution shift: lighting, lens, line speed, product mix	Prompt-population shift and retrieval-index staleness	Scenario-coverage gaps, sensor degradation, geography and weather regime shift
Typical alert	Threshold breach on defect-rate or input-embedding distance	Groundedness drop, refusal-rate spike, retrieval hit-rate collapse	Track fragmentation rate, fusion disagreement between sensors
Human in the loop	Re-label sampled disputed cases	Adjudicate rubric disagreements; owns the reference set	Scenario triage and ODD boundary review

Read the table by column and the point lands: three columns of the same six rows, and almost nothing transfers.

The CV column: shift arrives through the sensor

A vision inspection model degrades because the world in front of the lens changed. New supplier packaging, a cleaned or fogged lens, a line-speed increase that adds motion blur, a lighting retrofit on the night shift. Ground truth exists and is cheap to define — a box or a mask — which is why CV harnesses are the easiest to build well and the most misleading to copy.

Practically, the CV eval harness is a versioned dataset with slices that mirror production conditions, evaluated at the deployed operating threshold rather than at the argmax that flattered the model in training. Drift telemetry watches inputs, not just outputs: embedding-distance on a reference batch, or per-slice defect-rate shift. Tooling is unremarkable and that is fine — OpenCV for the ingest checks, PyTorch for the eval pass, a run ID from the experiment tracker so the number is retrievable a year later.

The alert is genuinely actionable, which is why CV teams get the highest alert precision of the three. When input drift fires, the response is a physical check on the line. That is a closed loop.

The LLM column: there is no single correct answer

This is where the copied harness fails hardest. There is no ground truth to grade against, so a regression suite built on output equality either passes trivially or fails on paraphrase. The workable construction is a frozen prompt corpus, a pinned retrieval snapshot so the index cannot move under the test, and assertions on graded properties rather than strings: is the answer grounded in retrieved context, does it refuse when it should, does it stay within a rubric band. Where graders disagree, a human adjudicates and the adjudication becomes part of the reference set.

Two failure modes are specific to this column and both are invisible on a CV-shaped dashboard. Prompt-population shift: real users start asking a different distribution of questions than the corpus was built from, so the suite is still measuring last quarter’s product. And retrieval staleness: the model is unchanged, the index is stale, and quality drops with no model event to correlate against. Any LLM harness whose drift telemetry watches only model outputs will miss the second one entirely.

We also expect lower alert precision here, and say so up front. Rubric-graded signals are noisier than a threshold on a defect rate, so the honest scoping position is that the LLM drift channel starts with a weekly review cadence and earns paging rights only after its precision has been measured over a named incident period — not on day one.

The perception column: the frame is the wrong unit

A perception stack does not usually fail at the detector. It fails at fusion and tracking — an object detected in three consecutive frames but assigned three different track IDs, a radar return and a camera detection that never associate, a pedestrian held through occlusion by one sensor and dropped by the fused state. Frame-level mAP is blind to all of it, because every one of those failures is composed of individually correct frames.

So the grading unit is a scenario, and coverage is the metric that matters: which operating conditions in the declared domain does the frozen scenario library actually contain, and which known field conditions have no scenario yet. Sampled spot checks — perfectly reasonable for an LLM harness — cannot answer that question, because absence of coverage is not something a sample reveals.

Regression here is expensive and unavoidable: replaying sequences through the fused stack, asserting on track continuity and association error, and treating each new field incident as a new permanent scenario. The production AI reliability practice we run treats an unrepresented field failure as a harness defect rather than a model defect, which changes who owns the fix.

Sequencing harness work across a mixed estate

Buyers with all three workload classes usually ask which to instrument first. The answer is not “the biggest model.”

1. Start where the alert loop closes. Usually CV. A workload whose drift alert has an obvious physical response gives you a working harness pattern and a credible alert-precision baseline before anything harder.
2. Then the workload with the highest consequence of silent failure. Perception, if you have it — scenario coverage takes the longest to build and gets worse the longer it waits.
3. Then the LLM workload, but scope the reference set first. Building the rubric and reference answers is the long pole; eval tooling is not. Teams that buy tooling before agreeing the rubric rebuild both.

Across all three, the single largest avoidable cost we see is the rebuild that follows discovering the first harness cannot be executed against the deployed system at all — the eval set exists but there is no way to replay production inputs through the served model. That check belongs in discovery, alongside the workload-variant decision, not after the first eval run. The parent argument for why the harness is the deliverable — rather than the model or the report — is developed in what a production AI monitoring harness is and what it must contain, and the per-section evidence requirements sit in what evidence belongs behind each harness section.

One thing we still hold loosely: whether the LLM drift channel can reach CV-grade alert precision at all, or whether rubric-based signals are structurally noisier and should stay on a review cadence permanently. We do not have enough measured incident periods across enough estates to call that yet.

Frequently Asked Questions

What does the monitoring harness look like in practice for a CV workload versus an LLM workload versus a perception workload? The six sections are identical; the contents are not. CV grades images against pixel or box labels and watches input distribution shift; LLM workloads grade prompt–response pairs against rubrics and reference answers and watch prompt-population and retrieval shift; perception grades scenarios over temporal sequences and watches coverage gaps at the fusion and tracking layer.

Which harness sections stay identical across workloads, and which are always workload-specific? The section list is fixed: eval harness, regression suite, drift telemetry, alert-quality tuning, release-readiness review, audit-evidence pack. Four things are always workload-specific — the grading unit, the ground truth definition, the drift signal, and what an on-call engineer can act on when an alert fires.

What is the right drift signal for each workload? For CV it is input distribution shift — lighting, lens condition, line speed, product mix. For LLM workloads it is prompt-population shift plus retrieval-index staleness, and the second one moves with no model event to correlate against. For perception it is scenario-coverage gaps together with sensor degradation and regime shift.

How is the regression suite constructed when there is no single ground truth, as with LLM outputs? Freeze the prompt corpus and pin the retrieval snapshot so the index cannot move under the test, then assert on graded properties — groundedness, refusal correctness, rubric band — rather than output equality. Human adjudication resolves grader disagreement, and each adjudication is folded back into the reference set.

Why do frame-level metrics fail for perception stacks, and what replaces them? Perception failures are composed of individually correct frames: consistent detections assigned inconsistent track IDs, or sensor detections that never associate in the fused state. Frame-level mAP cannot see those, so the grading unit becomes a scenario and the metrics become scenario pass rate, track continuity, and fusion association error.

How does alert quality tuning differ per workload? CV drift alerts close onto a physical check on the line, so they tolerate paging earliest. Rubric-graded LLM signals are noisier and should start on a review cadence, earning paging rights only after their precision is measured over a named incident period. Perception alerts key on track fragmentation and cross-sensor disagreement and are triaged into scenario work rather than paged.

How should a buyer with a mixed estate sequence harness work? Start with the workload whose alert loop closes cleanly — usually CV — to establish a pattern and an alert-precision baseline. Take perception next, since scenario coverage has the longest build time and the highest silent-failure cost. Do the LLM workload last, and agree the rubric and reference set before buying any eval tooling.

Why computer vision and LLM harnesses diverge after setup

Expect your monitoring harness to fork: latency percentiles that matter for object detection tell you almost nothing about token-by-token generation stalls.