Evaluating machine learning models for a procurement decision depends on one artefact: a task-specific evaluation harness that every candidate, internal or vendor, runs through in exactly the same way. The harness should hold a fixed, versioned case set built from your own labelled history. It should replay real production sequences and score each stratum separately, so a regression in any stratum blocks promotion. Thresholds should be agreed before anyone sees results, and models should be compared with paired statistics over repeated runs. Public benchmarks and single aggregate scores cannot carry this decision, because they answer what a model can do in general, not whether a workflow is fit, reliable and safe on your own data and under your own controls.

TechnoLynx builds these harnesses alongside its generative and agentic AI engineering work. We treat each one as a reliability artefact in its own right, because a harness catches the silent quality drift and rare-case failures that a model’s headline accuracy never shows. The examples below come from very different settings (maintenance tickets, clinical summaries, fraud alerts, engine-room telemetry, client commentary, rubric grading), and we use them because the discipline is the same in each.

Aggregate LLM evaluation scores hide the regressions that matter

A Respan guide to RAG observability (2026) gives the pattern in one line: an aggregate faithfulness of 0.93 can hide 0.98 on FAQ queries and 0.71 on legal ones. Latency rankings also shift with the slice. In a 2026 reproducibility study run on an A100 with vLLM 0.9.1, the model with the lowest tail latency changed with load: Qwen2.5-7B led at five of eight load levels, Mistral-7B at two and Llama-3.1-8B at one. In that study the ranking also changed with context length, with a different model ahead at 8k and at 16k tokens.

Quality scores hide strata in the same way. ASTRA, a 2026 agentic fault-analysis system, reached a mean quality score of 4.13 out of 5 across 987 real telecom tickets. Split by fault type, though, 67.8% of hardware tickets scored 2 or below, against 41.3% for software and 30.8% for configuration faults (Cohen’s d = 0.80 between hardware and configuration). A 2019 study of medical imaging models called this hidden stratification, and found relative performance gaps of over 20% on clinically important subsets that good overall numbers concealed.

An aggregate score has its place in reporting. It should never be what gates a decision.

Turning years of SAP PM tickets into labelled evaluation cases

For maintenance triage, the evaluation set should come out of a scripted pipeline you can rerun. The pipeline should extract SAP PM notification text and photo captions across several years and fault types, normalise the free text, collapse near-duplicates, sample by stratum and send each sample through a written labelling rubric with more than one rater.

TechnoLynx builds this pipeline as part of the LLM Selection Pack, starting by agreeing the extraction scope, what “representative” means for your fleet and the labelling goals with your maintenance engineers. TechnoLynx finishes by handing over a documented pipeline your team can rerun as new tickets come in.

The least manual route is scripted export plus stratified sampling, as long as the history has enough volume and variety. If the fault taxonomy is unstable or the volume is low, one manual curation pass to settle the categories should come before any automation, because automating a moving taxonomy just locks in its confusion. Maintenance work-order text is unstructured, so it needs cleaning before you sample it. Hastings, Sexton, Brundage and Hodkiewicz (PHM Society, 2019) describe experts in the Nestor tagging toolkit mapping “replace”, “replaced” and the misspelling “repalce” to a single alias. Where photo captions are sparse or missing, that limits how much multimodal coverage the set can claim.

Ground-truth triage labels should come from senior maintenance engineers working to a written rubric, with an adjudication step for disagreements. The same 2019 paper raises the weak point of using one annotator at a time: nobody can tell whether a single expert’s classifications are correct. The DKPro Statistics tutorial on inter-rater agreement recommends trained or professional annotators and reporting the full agreement table. It treats agreement above 0.8 as the mark of reliable data, while noting that the right level depends on the task.

Near-duplicates are a further source of bias. Lee et al. (ACL 2022) found that web-scraped language-model training datasets contained 3.04% to 13.63% near-duplicates. They also showed that train-test overlap makes accuracy look higher than it is and pushes model selection towards models that overfit. Clustering near-duplicate tickets and capping each cluster keeps common faults from drowning out rare ones. The similarity threshold is a heuristic, and you should expect to tune it for each fault domain.

A held-out set of de-identified discharge summaries with clinician references

For clinical summarisation, the harness needs a held-out set of de-identified discharge summaries, each paired with a reference summary written by a clinician. The set should be sized by power analysis, checked for de-identification by a qualified reviewer, and frozen under version control so it can serve as the regression benchmark for later model updates.

TechnoLynx designs and builds this harness as part of the LLM Selection Pack, combining automated metrics with a clinician review workflow. TechnoLynx begins by defining the summarisation task, the bar for reference quality and the statistical targets with your clinical leads.

The number of documents depends on the smallest difference you need to detect and on the metric’s variance, so no single count is right for every team. Card et al. (EMNLP 2020) give a reference point: a typical machine translation test set of 2,000 sentences has about 75% power to detect a 1-point BLEU difference. The same paper concludes that the usual human-rating study design is underpowered for small model differences. That matters here, because clinician review is itself a human-rating study, and a small clinical dataset limits power however well it is designed.

De-identification should be verified against a recognised standard, not asserted by the engineering team. HHS Office for Civil Rights guidance sets out the two methods that satisfy the HIPAA Privacy Rule’s de-identification standard, Expert Determination and Safe Harbor. The guidance also covers who counts as an expert and how long an expert determination stays valid for a given dataset. Data governance has to accept any residual re-identification risk explicitly. That sign-off belongs to compliance, not to whoever builds the harness.

The held-out set can be reused for regression testing, on two conditions: it stays representative, and it never leaks into training or tuning. EnterpriseVal (2026) keeps any tuning on a development split that is disjoint from the graded set. If model developers get access to the frozen set, it stops measuring anything, so access control is part of the design. Refresh the set only when you detect a distribution shift, and give each refresh a new version.

How do we split the evaluation by fraud typology so one pattern cannot hide behind the average?

Stratify the offline evaluation by fraud typology (structuring, layering, mule accounts and so on) and by customer segment. Score every candidate in each bucket against a labelled library of known patterns. If any bucket regresses beyond its tolerance against the production baseline, cutover is blocked, whatever the aggregate says. Scenario and rule design practice should define the buckets, so the evaluation’s typologies match the ones investigators already work with.

TechnoLynx builds this stratified evaluation as part of the LLM Selection Pack, scoring every candidate per typology and segment, and starts by framing the decision and its metrics with your team.

A fraud detection study reported per-category results: 76.2% recall on dating fraud, which made up 1.34% of the evaluation set, and 1.3% recall on adult-content fraud, which made up 0.18%. The normal class reached 98.7% recall. Any accuracy figure computed over that table is mostly a measurement of the normal class. A fairness audit of transaction fraud models found that, on the IBMCard dataset, bias in normalised TPR and VDR parity did not show at a global false-positive ratio of 5.0 but did at a group-wise ratio of 5.0, so what you see depends on how you measure.

Replaying one historical alert set also puts competing architectures on the same footing. A single-pass classifier with rules and an agentic multi-step investigator can be scored on the same alerts, which separates the effect of the architecture from the effect of typology. Offline replay will not capture the production latency or operational load of the agentic design, so treat it as evidence on decision quality only.

The count has to come from sample-size analysis for each bucket. In a worked example in the 2026 monograph Engineering Reliable Coding Agents, 20 of 1,000 deployment items are defects, so a natural sample of 100 holds two, and one miss moves detection by 50 percentage points. Rare typologies with too few labels should be reported as unresolved, not as passed. Typologies that have not appeared yet are not in the library, so a passing harness tells you nothing about them.

How do we replay past alerts to check a new model would have caught the same failures?

Run the labelled historical alert stream through both the existing classifier and the candidate RAG assistant. Score each one against the recorded outcome: would it have triggered the same intervention, or a better one? The metrics should be tied to your ticket and failure taxonomy, not to how pleasant the output reads.

TechnoLynx builds this replay harness. TechnoLynx starts by defining the decision-quality metrics against your ticket taxonomy and intervention outcomes, then wires the harness into a regression suite for later architecture and version comparisons. The LLM Selection Pack is the engagement for this decision.

Production logs have to keep the event order and enough context to rebuild the state at each point, and the harness processes events in that order, updating state as the model runs. If the logs leave out context, the replay has to approximate it, and state held in external systems is especially hard to reproduce. Older logs drift away from current conditions, so replay data needs refreshing with recent sequences. The Distilled Patterns write-up gives a perception example: aggregate detection improved after an update, but replaying known difficult scenes showed more missed obstacles in low light.

The metric that should show an assistant adds decision quality is the difference from the current classifier on correct-trigger rate, on precision and recall for known failures, and on lead time before failure. User preference for the interface does not show it. Your labelled failure history can serve as the held-out set if it is representative, and if none of it was used to build the retrieval index or the prompt examples. Informal spot-checks cannot stand in as the baseline for a statistical comparison, and failure types that never happened are not evaluated at all.

The same logic applies to templated versus LLM-written client commentary. Comprehension should be measured directly with human quiz or task-based tests. Each variant should be gated on an automated check of every statement against the account figures, and comprehension, complaint-risk proxies and factual consistency should be reported separately rather than merged into one score. Separate reporting keeps each dimension visible: in ASTRA the other four metrics correlated with accuracy at only 0.15 to 0.32, and near-ceiling presentation scores dominated the composite.

How do we score whether a model asks the right clarifying question, not just the final answer?

Score the whole dialogue, under a configuration that stays fixed. RegretBench (2026) computes all of its metrics with the split, persona, horizon, rollout count, simulator backends, random seed and cost profile held constant. Its final score combines intent success, interaction cost, penalties for unsupported or ineffective clarification, and regret against a reference planner. Two of its diagnostics target clarification behaviour directly. The unsupported-clarification rate counts questions that cannot be mapped to a supported clarification action. The forced-terminal-answer rate counts dialogues where the model reaches the turn limit without committing to an answer.

Final-answer accuracy is not enough here. A 2025 analysis of more than 200,000 simulated conversations found that top LLMs performed 39% worse on average in multi-turn settings than in single-turn ones, across six generation tasks. In RegretBench, K2-Think V2 had a relatively high success rate but a strongly negative reward, because of excess turns and cost.

Reproducibility also needs a consistent simulated user. ClarEval (2026) runs its multi-turn setting with a deterministic, rule-based user simulator built from human-annotated intents, which isolates the model’s communication from noise in the simulation. Its reviewers reached a Fleiss’ kappa of 0.82 when deciding which samples to keep. Repetition matters as well. Claw-Eval (2026) reports Pass@k, where a task counts if it passed at least once, alongside Pass^k, where it must pass on every trial. Under error injection, Pass@3 stayed stable while Pass^3 fell by up to 24 percentage points. For a procurement comparison we would prefer a scripted simulator wherever the domain allows it, and report reliability across trials next to capability. Any automated grader in the loop should be checked against human labels before it gates a release. Validating LLM-as-judge scoring is a separate question, and a page of its own.

Which agreement or exact-match thresholds should block promotion, and who signs them off?

Set a threshold for each gated rubric category before any results are unblinded. Use quadratic-weighted kappa for ordinal categories and Cohen’s kappa for categorical ones. Route sign-off jointly through ML, psychometrics (or whoever owns measurement for the domain) and compliance, and archive the thresholds and approvals as audit evidence.

TechnoLynx sets these per-category promotion gates as part of the LLM Selection Pack, agreeing the thresholds before unblinding, and starts by framing the decision and its metrics with your team.

The metric should match the shape of the category. The Autorubric paper (2026) uses Cohen’s kappa for categorical judgments, because it corrects for chance agreement, and quadratic-weighted kappa for ordinal scales, because it accounts for the size of a disagreement. Report accuracy and kappa together: in Autorubric, criteria met more than 89% of the time produced kappa close to zero even at high accuracy, because a judge could reach 93% by predicting “met” every time. The same paper recommends binary criteria where you need a defensible threshold and ordinal criteria where gradations carry meaning.

No single number is correct, but the sources give two useful reference points. EnterpriseVal lets a metric into a gate decision only if kappa is at least 0.6; below that, the rubric is revised, not the system. The DKPro tutorial treats agreement above 0.8 as the mark of reliable data, depending on the task. Our view is that a promotion gate should sit between those two for most ordinal grading work, with the exact value set by the cost of a wrong grade. Set it too strict and viable models get blocked. Set it too loose and the compliance exposure grows.

On sign-off, EnterpriseVal has each gated metric carry two thresholds, an outer minimum and an inner scaling target, agreed before unblinding by the business owner, the model owner and the second-line risk function. In a grading pipeline, we would seat the psychometrics lead as owner of measurement validity. EnterpriseVal sets thresholds in a facilitated session, then signs and versions the charter, and records any change made after unblinding as a deviation. Writing thresholds first is not bureaucracy: in that pilot it exposed an inverted pair of thresholds, a self-grading step and an unstated baseline. For audit, the Federal Reserve and OCC’s SR 11-7 guidance (2011) expects predetermined thresholds of acceptability that trigger adjustment or redevelopment when outcomes keep falling outside them. It also expects documentation detailed enough that someone unfamiliar with the model can understand how it works and where its limits are.

How many labelled examples per category before a difference between models is real?

There is no fixed count. Work backwards from the smallest difference that would change the engineering decision, measure the variance in a pilot, and let power analysis set the number of items and repeats. The paper Adding Error Bars to Evals (2024) recommends standard errors on every score, clustered standard errors when questions come in related groups, inference on paired question-level differences, and power analysis to check whether an eval can test the hypothesis at all. Its clustered standard errors came out up to three times larger than naive ones. Pairing gives a reduction in variance at no cost whenever two models agree on which questions are hard.

The Engineering Reliable Coding Agents monograph shows why single runs mislead. One configuration led another by +0.054 on a task family. After three reruns per configuration, the difference fell to +0.0035, with a 95% confidence interval from -0.0005 to +0.0074. Across 60,000 SWE-bench-Verified trajectories, single-run pass@1 varied by 2.2 to 6.0 percentage points, enough to swallow 2 to 3 point gains on that class of task. The monograph treats three independent repeats per configuration as a workable minimum. For metrics such as F-score or corpus BLEU, which do not decompose into per-item averages, it uses a paired bootstrap that resamples items as pairs and recomputes the full metric. An experiment that lacked the power to resolve the decisive difference returns no verdict, and the report should say so plainly.

Our companion piece on designing statistically sound benchmarks to pick the right model strategy goes further into cost-quality trade-offs and experiment sizing.

The same discipline tells you whether a technique, such as a prompt pattern or a retrieval step, still earns its place. Run it as a paired, seeded comparison across model families and report the effect size with a confidence interval. Treat “the newer base model no longer needs it” as a separate hypothesis, tested through a technique-by-model-generation interaction rather than a before-and-after score. Any ablation result only holds for the configuration in which the component was removed.

Locking the scenario set and the LLM evaluation harness so every candidate is scored identically

A procurement comparison is only fair if the scenario set and the scoring harness are both frozen before the first candidate runs. Every model then meets the same items, the same serving setup and budgets, and the same scorer.

TechnoLynx’s LLM Selection Pack delivers this harness. It starts by defining the fixed, versioned scenario set and the harness architecture with your team, then builds a scoring pipeline that runs every candidate identically and produces reporting ready for an auditor.

How do we build a fixed benchmark set that every candidate, internal or vendor, runs through the same way?

Version four things: the task, the dataset, the model and the code. The MTEB maintainers (2025) version each task’s evaluation protocol separately, pin each dataset to a specific source revision, point each model at a specific checkpoint or API version, and release the package under semantic versioning. They also fix seeds wherever evaluation involves a random step such as clustering or linear probing. Their own history is a warning: breaking changes that slipped into minor and patch releases confused and frustrated regular users.

For vendor models, pinning the version is harder than it sounds. A 2024 study found that the same GPT-3.5 Turbo model gave statistically different results through the Azure OpenAI API and the OpenAI API. A 2023 paper on evolving LLM APIs notes that these APIs are often updated silently and scheduled for deprecation, which can cause performance regressions. Record the endpoint, the version string and the date for every run. To stop the scenario set drifting silently between rounds, pin every dataset revision and give any edit a new version, as MTEB does, and treat any change to the system configuration as a trigger for re-evaluation, as EnterpriseVal does. MedicalHarness (2026) adds a useful rule: no episode is rerun once its score has been seen.

A locked set goes stale as real conditions change, so versioning has to allow planned refreshes.

What the harness records for each case

Report per item, then aggregate. The Engineering Reliable Coding Agents monograph keeps, for every task, the task identity, both systems’ scores, the nuisance conditions assigned and the run identity, so the pairing survives into the analysis. It records success, reliability, cost, latency, model, harness, prompt, permissions and a pricing snapshot as separate fields. MedicalHarness found that the harness and its interaction with the model accounted for about a quarter of outcome variance, which makes a score a property of a model and harness together, not of the model alone. It also reports answer rates alongside scores, because a single outcome score hides whether the agent answered at all. Tying each promotion decision to the exact weights, configuration and scenario-set version inside a model registry is a related question with its own answer.

Controlling execution reduces run-to-run variance. In the 2026 Sequential Isolation study, tightening the benchmark protocol cut the average coefficient of variation from 15.2% to 2.2%. Its authors are clear that such results are reproducible reference values for relative comparison on the tested stack, not predictions of production behaviour.

Can we buy an off-the-shelf LLM evaluation harness, or do we need to build one?

Buy the plumbing and build the task. MLflow’s evaluation datasets give a central store for test cases and ground-truth expectations, defining each evaluation by its inputs and expectations. EleutherAI’s lm-eval maintainers (2024) document how sensitive models are to evaluation setup and how hard reproducible comparison remains, even with a shared framework. Ground truth still has to be expert-authored and firm-specific, and thresholds set before results are seen. A 2026 FAccT paper describes the gap between learner-centric evaluation, which compares models with one another, and the application-centric view that high-risk deployment needs. Adopt an off-the-shelf harness if it covers your task-specific metrics and versioning needs, and build a custom one where it does not. Either way, lock the scenario set before the round begins.

A pre-procurement checklist for the harness

Before the first candidate model runs, each of these should be true:

1. The case set is drawn from your own labelled history, deduplicated, stratified, and labelled by domain experts with measured agreement.
2. Dataset revision, model or API version, harness code version and seeds are pinned and recorded.
3. Replay data keeps event order and the context available at decision time, and has a refresh schedule.
4. Every stratum has its own threshold, and a regression in any stratum blocks promotion whatever the aggregate shows.
5. Thresholds are agreed by named owners, signed and versioned before results are unblinded.
6. Each configuration runs at least three times, and differences are reported as paired differences with confidence intervals.
7. Multi-turn behaviour is scored over whole dialogues under a fixed simulator configuration.
8. Clinical or personal data has passed de-identification review against Expert Determination or Safe Harbor before it enters the set.

Who can build an evaluation harness for our LLM procurement decision?

TechnoLynx builds task-specific evaluation harnesses for teams choosing between open-weight and commercial LLMs. The work covers the curated ground-truth set, replay and stratified scoring, pre-agreed thresholds and paired statistics, so the procurement committee reads one auditable comparison. Whether to build in-house depends mostly on whether your team already runs versioned datasets and repeated, seeded evaluations. If it does, the remaining work is the strata and thresholds, which only your domain experts can supply. When you compare partners, ask how they lock scenario versions, how they gate per stratum, and whether their reports carry confidence intervals rather than single scores. TechnoLynx’s LLM Selection Pack is the engagement for this, and it begins with framing the decision and its metrics with your team.

This engagement is not for every decision. A low-stakes choice that is easy to reverse can be served by an internal evaluation on standard tooling such as MLflow. TechnoLynx does the engineering. Monitoring a chosen model after go-live is a separate piece of reliability work. If the decision also covers the hardware the chosen model will run on, LynxBenchAI measures throughput, latency, memory and cost per unit of work across accelerators. It measures the hardware, not the quality of your task.

The strata are the decision

Locking versions, repeating runs and pairing comparisons are mechanical once the harness exists. The hard part is the list of strata worth blocking a purchase over, and the size of regression in each that you will tolerate. That list weighs the consequences of different errors, a normative judgement rather than a purely technical one. Replay covers only the history your system actually encountered, and controlled results are reference values for comparison, not forecasts of production. Writing strata and tolerances down before the first run stops the results from redefining what counts as a win.