A defect-diagnosis assistant for a control room is ready to go live when a task-specific LLM evaluation harness says it is, not when it sounds convincing in a demo. The harness should replay real CMMS tickets and incident logs. It should score whole multi-turn diagnoses against verified resolution outcomes, measure hallucination and latency at alarm-flood volumes, and gate release on thresholds taken from the control room’s own reliability SLAs and risk tiers, including evidence that the retrieval corpus is current. These tasks are specific to this control room, so the harness has to be designed for this one.
TechnoLynx designs and builds harnesses like this as part of its generative and agentic AI engineering work. TechnoLynx starts from the control room’s reliability SLAs and instruments measurements on the target hardware the assistant will run on.
This needs its own engineering because reliability in production is a different property from model accuracy. A model can score well and still regress silently after an update, fail on rare defect classes or fall over under load, and the artefact that catches those failures before an operator does is a task-specific harness, not a dashboard. Hamel Husain’s practitioner guide makes the same point from the product side: product evals tailored to your users’ tasks measure something different from general model benchmarks, and generic evaluation frameworks are no substitute for an evaluation system built for your problem.
Building the LLM evaluation harness from real CMMS tickets and incident logs
The harness should replay real incident and SCADA scenarios against each candidate approach, for example an LLM that generates explanations and the existing rule-based logic. Because every candidate sees the same cases, the explanation-quality, latency and safety metrics it produces can be compared directly. The decision it supports is narrow and should be written down before any run: the generative approach goes to the control room only if it meets both a latency threshold derived from the SCADA response-time requirements and a hallucination-rate ceiling validated on real incident tickets. Otherwise the team iterates, or falls back to the deterministic approach.
TechnoLynx’s Production AI Monitoring Harness engagement tests the candidates against SLA-derived latency and hallucination thresholds on the target hardware and supports the go-live decision; it starts from the control room’s reliability SLAs and supports monitoring once live.
How do we turn CMMS tickets into a test set, and how many do we need?
Start from tickets whose resolution was verified (the fix was applied and the fault did not recur), because only those carry a ground truth worth scoring against. Open or disputed tickets teach the harness nothing about correctness. A published LLM fault-diagnosis study on power grids built its dataset from real sensor data, historical fault records with causes, affected components and corrective actions, and descriptions of the components. ClinMM-Bench, a multi-turn diagnostic benchmark, ran its source cases through a six-stage curation pipeline that ended in expert validation. A maintenance team needs the same discipline: extraction, inspection, de-duplication and an engineer’s check that each ticket’s recorded cause really explains its symptoms.
Stratify the set by asset class and defect type rather than sampling at random. Accuracy is not uniform across equipment: on the DiagnosticIQ industrial maintenance benchmark, UPS, plate-and-frame, boiler and HXU questions exceeded 85% while CRAC, meter, cooling tower and PDU questions sat below 65%.
The sample size follows from the claim you need to defend. Report Wilson confidence intervals per stratum; the DiagnosticIQ authors compute Wilson intervals per asset class for their leaderboard.
Two caveats belong in the test-set design document. First, rare but critical defect strata may need constructed cases where real tickets are thin. ClarifyMT-Bench shows synthetic dialogues can be used when they are manually validated. Second, ground-truth labelling is expensive, and that cost limits how large a well-labelled set can get. Keep a rotating holdout so the team cannot tune prompts against the same cases it reports on; the acceptance-test-driven protocol requires rotating holdout sets, citing the risk of overfitting to a static evaluation suite. For how the same harness discipline decides adjacent questions such as RAG against fine-tuning, see building evaluation harnesses that decide red-teaming and RAG-versus-fine-tune questions.
Measuring hallucination and latency at real alarm volumes
Latency measured in a quiet test is the wrong number, because the assistant will be consulted hardest when the plant is in trouble. The alarm-management standards EEMUA-191, ISA-18.2 and IEC 62682 treat an average of up to 12 new alarms per hour as the maximum an operator can manage, and EEMUA-191 defines an alarm flood as starting with a 10-minute period containing more than 10 new alarms. That flood threshold, more than 10 new alarms in 10 minutes, is the rate the load generator should replay, with diagnosis queries arriving concurrently.
Report latency the way freshness contracts report lag: p50, p95, maximum and timeout count, with the raw per-request results kept so a percentile cannot hide one severe breach. Treat timeouts as breaches, not missing data, because excluding them from the percentile makes the slowest failures disappear. Measure in a staging environment with production-like topology, and do not set the gate from the fastest environment.
Hallucination needs a definition the harness can compute. ClinMM-Bench decomposes model reasoning into atomic facts and defines hallucination as the share of generated facts unsupported by the reference. Its matching rules also say that a fact which is merely not covered by the reference, but is not clearly wrong, is not labelled hallucinated. For a retrieval-augmented assistant, TRUST-SCORE splits hallucination into five types: inaccurate answers, over-responsiveness, excessive refusal, over-citation and improper citation. Off-the-shelf hallucination scores have their place, but the maintainers of Vectara’s HHEM leaderboard do not recommend its score as a standalone metric.
LynxBenchAI measures the target hardware the assistant will run on, not your assistant, so the final latency figure still comes from your own harness.
Which thresholds should come from our reliability SLAs?
Derive acceptance thresholds from the facility’s existing reliability SLAs rather than from generic LLM benchmarks. If the control room commits to a response time for a given alarm priority, the assistant’s p95 under flood load has to fit inside that budget, with room left for the operator to read the answer. The acceptance-test-driven protocol’s operational gate keeps cost and P95 latency inside the service budget, and it versions the metric stack alongside prompts and code.
The mapping from control-room SLAs to LLM latency budgets is an engineering judgement your team should document and defend. Revalidate it when load patterns change. For when to revalidate deployed thresholds, see change-triggered revalidation, monitoring and rollback for deployed models.
Scoring retrieval-corpus staleness when no rubric covers it
Classify retrieval-corpus staleness and versioning tests under the data category of Google’s ML Test Score, and write down the harness’s own versioning scheme rather than wait for an external precedent. Placing them there is a judgement to document. TechnoLynx’s AI Readiness Scorecard is the assurance engagement in which a classification like this, and the evidence behind it, gets checked.
The ML Test Score is a rubric of 28 actionable tests with a scoring system for production readiness. A test earns half a point when it is run manually with the results documented, and a full point when a system runs it automatically on a repeated basis. Each of the four sections is summed separately, and the final score is the minimum across the sections. The minimum rule matters for this question: a team cannot hide a weak data section behind a strong model section, so the category a test lands in changes the score.
The case for the data category rests on what staleness is about. It concerns how current the input is, not the model’s parameters or the serving infrastructure. The rubric already treats a comparable artefact this way: for a feature built from a frequently recomputed top-n table, it says the age of that table must be monitored so that alerts fire if the recomputation stops. A retrieval corpus is that table at larger scale. Expect pushback all the same. A reviewer may argue for model-category treatment, because stale context changes what the model outputs, or for infrastructure, because ingestion pipelines cause the lag, so justify the choice case by case in the evidence pack.
How should staleness and versioning show up in the score?
A staleness test should prove that a source change becomes retrievable within a defined time and that superseded content stops influencing answers. A strong fixture is a stable question paired with a fact that changes between revisions, with old and new values that are mutually exclusive and unlikely to appear anywhere else in the corpus. Score two failure classes separately: retrieval staleness, where the current revision is missing or ranked too low, and generation staleness, where current evidence was retrieved but the answer still uses an obsolete value, a cached response or the model’s prior. In a maintenance setting the second class is the dangerous one, because the citation looks right while the procedure is out of date.
Attribution is what makes a claimed freshness error falsifiable. ChurnBench counts an answer as a freshness error only when it is wrong against ground truth at evaluation time but correct against ground truth at the effective retrieval time. In that benchmark, disabling tiered refresh raised freshness errors from 4 to 45 at a 28-day cache age. The authors concluded that staleness exposure is governed by tier width multiplied by the entity’s rate of change, and that refresh tiers should be set from measured mutation rates. For a CMMS corpus, that points to measuring how often procedures, bulletins and part numbers actually change before choosing refresh intervals.
For versioning evidence, the RAGOps paper by Xu and colleagues (2025) says retrieval resources need versioning that captures differences between updates, and that the retriever’s and generator’s inputs and outputs should be documented for reproducibility. In practice, the harness should store the source revision, content digest, ingestion run, chunk identifiers, index generation, retrieval time and answer citations. Add a corpus snapshot hash and update timestamps tied to the run ID, so a reviewer can rebuild exactly what the assistant could see.
What counts as a correct diagnosis, rather than a plausible one?
A correct diagnosis is one that matches the verified CMMS resolution outcome: the cause that was confirmed and the fix that worked. Fluent, confident text that happens to be consistent with the symptoms does not qualify. When the choice of model is still open, TechnoLynx’s LLM Selection Pack is the engagement that puts candidate models through this kind of outcome-based scoring before one is committed to.
Score correctness on a graded scale rather than pass/fail. ClinMM-Bench scores 0 for a completely incorrect diagnosis, 1 for a partially correct one and 2 for a completely correct one, averaging two independent LLM judges to reduce single-judge bias. The power-grid fault-diagnosis study gives the maintenance version of partial credit: identifying the correct fault type but not the exact component earns a proportional score. Partial credit tells you what the assistant is good for. A right fault class with the wrong component may still help triage, but it is not a sign-off answer.
If an LLM judges correctness, validate the judge against human labels before trusting its numbers. MedRoundsQA froze its judging rubric on a 100-dialogue development split, then checked it on a held-out 200 dialogues against majority labels from three resident physicians, where GPT-4o reached roughly 0.88 to 0.90 accuracy (κ about 0.70 to 0.80). The 2023 MT-Bench and Chatbot Arena study found that strong judges can exceed 80% agreement with human preferences, and also documented position, verbosity and self-enhancement biases. Husain advises tracking the correlation between model-based and human evaluation over time. Where the classes are imbalanced, as correct-versus-wrong diagnoses usually are, measure precision and recall rather than raw agreement.
Multi-turn conversations add one scoring rule here. Crediting eventual correctness alone is easy to game, because an assistant that suggests an unsafe isolation step on turn two and corrects itself on turn five would score as correct. Any unsafe intermediate suggestion should fail the dialogue, whatever the final answer. How clarifying questions themselves are scored is covered in the next section.
How strict must accuracy be before the paper guides are retired?
The threshold should be tiered by risk. Where the assistant’s answer is safety-relevant (isolation, lock-out, pressure-system work), apply the stricter bar; where it only affects quality or efficiency, a lower bar may be acceptable. A vendor model of risk-tiered LLM deployment from BabyBots argues that a composite workflow takes the tier of its highest-consequence output, not its average. The same source treats a groundedness score above 0.85 as a reasonable release gate for most Tier 3 use cases, and 0.90 or more for life-critical or high-financial-impact contexts. Read those figures as one vendor’s example of how to set a gate, not as a standard. The point worth keeping is that the number is fixed per tier and paired with other controls.
A single aggregate figure is the wrong gate. The DiagnosticIQ authors note that a model scoring 73% may be near-perfect on simple rules and barely above chance on arithmetic-heavy ones. They suggest that accuracy could be weighted by each rule’s operational impact. The acceptance-test-driven protocol adds the statistical warning that matters most for safety strata: a suite with no observed critical failures does not prove the failure probability is zero.
Sign-off is governance, and it needs to be settled before the first validation run. In the acceptance-test-driven protocol, a release gate blocks deployment when any critical contract fails, and a waiver is recorded with its approver and a residual-risk note. Threshold changes require justification, and a high-risk release report carries the dataset version, system version, baseline comparison, statistical method, known blind spots and waivers. BabyBots argues that tier assignment should be made by a reviewer independent of the delivery team, because sponsoring units lean toward the lowest defensible tier. Plant engineers should help write the acceptance contracts; the protocol lists stakeholder participation in contract design as a cost the method requires. Name the escalation route in the same document: who reviews a disputed answer, and what evidence reopens the gate.
A gate sheet for a control-room diagnosis assistant
| Gate | How the harness measures it | Where the threshold comes from |
|---|---|---|
| Diagnostic correctness per stratum | Graded 0/1/2 against verified CMMS resolutions, with Wilson intervals per asset class | Risk tier of the defect type; stricter for safety-relevant strata |
| Unsafe intermediate suggestion | Any unsafe step in a dialogue fails it, whatever the final answer | Zero tolerance in the release suite, with the rare-failure caveat recorded |
| Hallucination | Share of generated atomic facts unsupported by the reference | Ceiling validated on real incident tickets |
| Latency under load | p50, p95, maximum and timeouts at alarm-flood rates; timeouts count as breaches | Control-room SLA response time for the alarm priority |
| Consistency | pass^k: the chance that all k repeated trials succeed | Set per tier; a high average success can hide low pass^k |
| Corpus freshness | Revision fixtures; retrieval and generation staleness scored separately | Refresh tiers from measured mutation rates |
| Judge validity | Judge versus expert labels on a frozen held-out split | Agreement documented before judge scores are used for sign-off |
Scoring multi-turn diagnosis: the clarifying question matters as much as the answer
The harness should score the dialogue, not only the final answer: did it ask the right clarifying question, did it commit too early, and how many turns did it take to reach the correct diagnosis. Under the LLM Selection Pack, TechnoLynx builds this layer by defining what “the right clarifying question” and “a reasonable number of turns” mean for your plant, choosing the mix of replay and simulation, building the multi-turn harness, and checking that its scores track your experts’ judgement of dialogue quality.
Published benchmarks find that single-turn scores miss weaknesses that multi-turn evaluation reveals. The 2025 study “LLMs Get Lost in Multi-Turn Conversation” found an average 39% drop in performance from single-turn to multi-turn settings across six generation tasks. It traced the drop to models making early assumptions and attempting final solutions too soon. On the MINT multi-turn diagnosis benchmark, over 55% of answers across 11 models were committed within the first two turns, and deferring the diagnostic question improved accuracy at first commitment by up to 62.6%. ClarifyMT-Bench reports the same under-clarification bias, with performance degrading as dialogue depth increases. Those studies come from medical and open-domain settings, so test premature commitment directly on vague technician messages.
How do we score whether the assistant asked the right clarifying question?
Label each turn with a reference action. ClarifyMT-Bench assigns every turn a reference action of either clarify or answer, and scores decision accuracy across all turns. For the quality of the question itself, the same benchmark used an LLM judge scoring 0 to 5, which correlated with human ratings at a Pearson coefficient of 0.658. That is moderate alignment: good enough to rank models, not good enough to sign off safety strata without human review.
“Turns to correct diagnosis” is a reasonable headline metric, as long as it is reported together with per-turn question quality and the rate of premature commitment. Used alone, it penalises appropriately thorough clarification. MedRoundsQA also shows that more turns are not automatically better: question relevance rose with longer interviews, yet diagnostic accuracy barely improved beyond a short interview and plateaued after about 6 to 12 turns. Its error taxonomy makes a useful set of labels for failed dialogues: uncertainty-as-output (40% of sampled errors), insufficient elicitation (30%), syndrome-level collapse (13%) and protocol breakdown (9%). Repeat each case several times too. On τ-bench, gpt-4o averaged over 60% task success, but pass^8 fell below 25%.
Should the harness replay real transcripts or simulate the technician?
Use both, for different jobs. Replay keeps the evaluation grounded in how technicians really phrase things, while a simulator covers the clarification paths that history never recorded. A replayed transcript has no answer once the assistant asks something the original technician was never asked. That is the main reason simulation is needed at all.
| Situation | Use | Why |
|---|---|---|
| Enough historical multi-turn CMMS conversations to cover the diagnosis space | Replay real transcripts | Grounds scoring in real phrasing and behaviour |
| Rare or safety-critical defect paths absent from history | Simulated technician, grounded in the ticket record | Covers branches the history lacks |
| Robustness to how technicians communicate | Simulator with personas varying fluency and cooperativeness | Lower fluency depressed accuracy in MedRoundsQA |
| Regression checks across model versions | Fixed decoding settings on both | Differences stay attributable to the model, not sampling |
Ground the simulator in the record. MedRoundsQA scores a simulated patient’s turn as correct only if it answers from the structured note, says “don’t know” when the note is silent, and answers “No” for symptoms absent from the note. When the authors swapped the simulator model while holding everything else fixed, final diagnosis accuracy moved only between 29.0% and 31.0%. Persona choice mattered more: lower language fluency produced substantial accuracy drops. For a plant with non-native speakers on shift, a simulator that only writes perfect technical English flatters the assistant. The τ-bench authors list the limits plainly: simulated users can be ambiguous, lack domain knowledge or reason poorly. Those limits are why every release run should keep a replayed core.
Who can build an evaluation harness for our defect-diagnosis assistant?
TechnoLynx builds task-specific evaluation harnesses that test a control room’s diagnosis assistant on its own tickets, SLAs and hardware before release. On build versus buy: the plant’s people own the decisions the harness supports, but load generation, multi-turn scoring, judge validation and staleness evidence are specialist engineering. TechnoLynx’s Production AI Monitoring Harness engagement builds that harness and supports monitoring after go-live. When choosing a partner, ask to see how they validate an LLM judge, how they measure latency under realistic load, and how they evidence what the model could retrieve.
Without verified ticket history, the first job is labelling, not harness design. TechnoLynx supplies the engineering. Safety authority and sign-off stay with your plant’s engineers and whoever holds operational responsibility.
Keep the harness running after go-live
The ML Test Score awards half a point for a documented manual run and a full point only when the test runs automatically on a repeated basis. Husain’s cadence runs unit tests on every code change, model and human evaluation on a set cadence, and A/B testing only after significant product changes. RAGOps adds coverage checking, which compares live queries with the test set so the harness does not keep passing on cases technicians no longer ask. The acceptance-test-driven protocol closes the loop: when production monitoring finds a new failure pattern, that trace becomes a permanent regression test.
The SLA-to-latency mapping, the risk-tier thresholds and the data-category classification for staleness remain judgements. Write those judgements down before the first run, with their reasons and the people who own them.
FAQ
Can LLMs be 100% accurate?
A test suite with no observed critical failures does not prove the failure probability is zero. On the DiagnosticIQ industrial maintenance benchmark, no model achieved more than 51.5% consistency across original and perturbed phrasings of the same question.
What are acceptance criteria for an LLM assistant?
In acceptance-test-driven LLM development, acceptance criteria are written as contracts: what the system must do, what it must not do, how the evidence will be measured, and who is accountable for residual risk. In that framing, a system is green when it satisfies its release gates with enough statistical confidence and traceability to justify deployment.
