Two proposals land on the same desk. Both promise to harden your production AI, instrument your models and set up monitoring. Both quote within 15% of each other. Neither names a single artefact that will exist when the invoice is paid — and that omission, not the price or the reference logos, is what decides whether the engagement closes your reliability gap or reproduces it at cost.
The failure mode here is not vendor dishonesty. It is a definitional vacuum. “Hardened” is a word both parties can sign without ever agreeing what it means, so the buyer scores narrative quality and the vendor delivers whatever their default toolkit produces. The gap only becomes visible at the handoff meeting, by which point the budget is gone.
The mechanism: narrative scoring versus section scoring
When a proposal is scored on narrative, the reviewer is comparing prose. Prose is cheap to produce and impossible to fail. A vendor writing “we will establish comprehensive monitoring for your models” has committed to nothing that can be tested at a milestone, and a buyer who accepts that sentence has bought a judgement call they will have to make subjectively at the end.
Scoring on sections works differently. A harness-shape reference is a fixed list of the things production AI reliability work actually produces:
- an eval harness — a versioned dataset and scoring code that can be re-run on demand
- a regression suite — frozen baselines and slice-level assertions with pass/fail gates
- drift telemetry — input, prediction and label drift signals with thresholds tied to decision boundaries
- alert-quality work — precision and actionability tuning, not just threshold placement
- a release-readiness review — a pass/fail record produced before a model version ships
- an audit-evidence pack — the retrievable trail behind every number the other sections report
Hold a proposal against that list and the vagueness collapses immediately. Each section is either committed to, explicitly out of scope, or silently absent. Silently absent is the dangerous state, because both parties will later claim it was covered.
A proposal that names no artefact has not been priced — it has been narrated.
What specific sentences signal that no artefact was defined?
Certain phrasings are reliable tells, and we see them regularly in proposals clients bring to a scoping call. They share a grammatical property: the object of the verb is an activity, not a thing.
| Proposal sentence | What is missing | What a committed version says |
|---|---|---|
| “We will set up monitoring and alerting.” | Which signals, which thresholds, whose staffing model | “Drift telemetry on three named input features plus prediction distribution, thresholds derived from your decision boundary, alert precision measured over a 30-day incident window.” |
| “We will instrument your models for observability.” | Whether anything is executable or re-runnable | “Eval harness with a versioned dataset and a CLI entry point; re-runnable against any future model build.” |
| “We will validate model performance.” | Baselines, slices, pass/fail authority | “Regression suite with frozen baselines on five business-critical slices; gates fail the build.” |
| “We will provide a dashboard and documentation.” | Any artefact that can be failed | “Release-readiness review producing a signed pass/fail record per model version.” |
| “We will ensure the model is production-ready.” | Everything | Nothing — this sentence cannot be made concrete without naming sections. |
The pattern is that unfalsifiable deliverables cluster around dashboards and documents. A Grafana board and a summary deck are real work, and both can be honestly delivered against every sentence in the left-hand column. They just cannot be executed, re-run, or failed — which is the property that distinguishes a harness from a description of one. Our colleagues explore that distinction in more depth in why a monitoring harness is not a model card or a slide deck.
Where identical narratives diverge
The divergence point is not gradual. It is a single meeting.
Up to handoff week, the narrative-scored engagement and the harness-scored engagement look similar from the buyer’s side: standups happen, PyTorch training code gets touched, MLflow or Weights & Biases appears in the stack, dashboards fill with lines that trend in reassuring directions. Then the deliverable arrives. In one case it is a dashboard, a metrics summary and a set of recommendations. In the other it is a repository an engineering reviewer can clone, run, and watch fail on a deliberately regressed model.
The practical test of which one you bought is the next model update. A regression suite with frozen baselines survives a retrain and tells you what moved. A dashboard survives the retrain too — it just keeps showing green, because nothing in it encoded what “correct” meant for the previous version.
Which sections go missing most often? In our experience the ordering is consistent, and it tracks effort rather than importance (observed across TechnoLynx engagements; not a benchmarked ranking). Alert-quality work disappears first, because tuning precision requires a real incident history and nobody wants to bill for waiting. The audit-evidence pack disappears second, because assembling retrievable run IDs and computation windows is unglamorous and invisible in a demo. Regression suites survive most often, since a partial one is easy to gesture at — but they survive in a form that only asserts on aggregate metrics, which is close to useless when a single slice degrades.
Converting the section list into acceptance criteria
The fix is unexciting and takes about a day. Copy the section list into the statement of work, and against each section write three things: what artefact exists at completion, who signs it, and what “fail” looks like.
- Name the artefact per section. Not “monitoring” — “drift telemetry module, in-repo, with baselines committed.”
- Name the signer per section. Different sections need different competence; who signs which is a decision the SOW should settle, and who signs each section of a production AI monitoring harness sets out the reasoning.
- Name the failure condition. A milestone you cannot fail is a milestone you cannot use for acceptance.
- Mark out-of-scope sections explicitly. A vendor who writes “audit-evidence pack: out of scope, client to assemble” is being more useful than one who leaves it unmentioned.
- Re-run the check before signature, not at handoff. The gap check takes days when it happens at proposal review and costs a full second engagement when it happens after.
That last point is where the money is. The expensive pattern is not overpaying a vendor; it is paying twice for the same reliability scope, because the first engagement left no regression suite or evidence pack the second team could inherit. The second team re-derives the baselines, re-argues what a slice assertion contains, and re-negotiates thresholds — from zero. Buyers who name harness sections in the SOW get a concrete acceptance test at each milestone rather than a subjective judgement about whether the AI now feels more reliable.
We take the same list into diagnostic work: the structure of the artefact and the evidence behind each of its sections is set out under production AI reliability, and the reliability-audit engagement exists partly to pin down scope and deliverable before any hardening work starts.
When a dashboard is genuinely the right scope
Sometimes it is. If the model is pre-production, the failure cost is low, and the immediate need is visibility rather than release control, then observability alone is a defensible purchase. The honest proposal says so: “this engagement delivers drift telemetry and dashboards; it does not deliver a regression suite, and no release-readiness decision can be signed off its output.” That sentence loses no work and prevents the entire failure mode described here.
What it also does is name the second engagement. A buyer who knows the harness has six sections and has bought two of them is in a completely different position from one who believes they have bought “hardening” and will discover otherwise at the first retrain.
The open question we have not resolved is how much of the section list a buyer can reasonably enforce against a vendor who genuinely does not work that way. Some teams produce excellent reliability work with idiosyncratic artefacts that do not map cleanly onto six named sections. The comparison frame still helps there — but it becomes a translation exercise rather than a checklist, and translation is where vagueness re-enters.
Frequently Asked Questions
What does “why ‘we’ll harden your AI’ proposals fail without a harness-shape reference” mean in practice? Without a harness-shaped structure, proposals to harden AI systems collapse under their own architectural ambiguity. It means the proposal contains no named artefact, so neither party can test whether the work was done. Both sides sign the word “hardened” while holding different definitions of it, and the difference only surfaces at handoff when the deliverable turns out to be a dashboard rather than something a reviewer can execute and fail.
What specific sentences in a reliability proposal signal that no artefact has been defined? Sentences whose object is an activity rather than a thing: “set up monitoring”, “instrument your models”, “validate performance”, “ensure production readiness”. None of them can be failed at a milestone. A committed version names the signal, the threshold, the baseline, or the repository entry point.
Which harness sections are most often silently dropped, and why? Alert-quality work goes first, because tuning precision needs real incident history that nobody wants to bill for waiting on. The audit-evidence pack goes second, because assembling retrievable run IDs and computation windows is invisible in a demo. Regression suites usually survive, but often only in aggregate-metric form, which misses single-slice degradation.
How can a buyer convert the harness section list into acceptance criteria inside an SOW? Against each of the six sections, write the artefact that will exist, the named signer, and the failure condition — then mark any section that is out of scope explicitly. A milestone with no failure condition cannot be used for acceptance, and an unmentioned section is the one both parties will later claim was covered.
When is a dashboard-only deliverable actually the right scope? When the model is pre-production, the failure cost is low, and visibility matters more than release control. The proposal should say so plainly — that it delivers telemetry and dashboards, not a regression suite, and that no release-readiness decision can be signed off its output.
Will your workflow actually absorb a hardening layer?
Map which teams will use the hardened system day-to-day and confirm their process includes a checkpoint where validation outputs can be reviewed and acted upon. That answer is workload-specific, and it is worth writing down before you build.
