You can trust an LLM-as-judge evaluation harness to gate claims-triage releases only after you have validated it at two levels of detail. At the aggregate level, check its go/no-go calls against human-labelled gold cases and against regressions that actually reached production, and report agreement as a chance-corrected statistic such as Cohen’s kappa rather than percent agreement. At the category level, reasoning, refusal rate, factuality and formatting have to move as separate signals, and known drift and adversarial test sets have to come out labelled as what they are, not as a capability drop.

TechnoLynx builds this validation into the evaluation and release gate as part of its Production AI Monitoring Harness. The work covers defining what a production-visible regression means for your triage task, assembling human gold labels, measuring judge agreement and testing the per-category breakdown against labelled drift and adversarial sets. The first step is framing: agreeing with your claims and engineering leads which triage errors count as regressions, and how many gold cases you need to measure them.

Does the go/no-go call predict regressions customers see, or just stylistic drift?

The answer comes from replaying releases whose production outcome you already know, not from how persuasive the judge’s reasoning reads.

The trap is that a judge can have moderate global agreement with humans and still be weak at the decision you actually use it for. In a 2026 study of best-of-n selection on 5,000 Chatbot Arena prompts, a judge with a moderate global correlation of r = 0.47 captured only 21.0% of the improvement that perfect selection would have achieved over random choice. The within-prompt correlation was only 0.27, because global agreement was driven largely by prompt-level baseline effects. Applied to claims triage, a judge can agree with your reviewers that complex multi-party claims are hard without reliably telling a correct routing from a wrong one on the same claim. The same study recommends auditing in the hard regime that matches deployment, near-neighbour candidates, because trivially distinguishable pairs inflate every metric.

Published incidents show that passing evaluations is not proof. OpenAI wrote that its offline evaluations of a GPT-4o update “generally looked good” and its A/B tests suggested users liked the model, yet neither caught the sycophantic behaviour that followed. Anthropic’s Claude Code postmortem describes system-prompt ablations, run on a broader set of evaluations during the investigation after initial evals missed the issue, in which one evaluation showed a 3% drop for Opus 4.6 and 4.7; the prompt was reverted. Anthropic now adds soak periods, a broader eval suite and gradual rollouts for changes that could trade off against intelligence.

Agreement among evaluators is not validity either. In a 2026 study of LLM judges predicting real A/B-test outcomes, judges that differed in model or prompt agreed with each other at κ = 0.74–0.88 but agreed with actual outcomes at only about 0.2. A panel of 15 human experts in that study showed inter-rater κ of 0.53, at chance against the real results. For a release gate, the check that matters is whether the harness would have said no-go on the releases that hurt you. A clean scorecard on a curated benchmark does not answer that.

Measure agreement with Cohen’s kappa, not percent agreement

Cohen’s kappa compares observed agreement with the agreement expected by chance from the two raters’ label distributions, so a judge cannot score well just by echoing the majority class. A judge that waves almost everything through can post high raw agreement while adding no decision value. The A/B-test study puts a number on the effect: on curated catalogues, the rule “always pick the variant” scores roughly 75–85% accuracy and contributes nothing.

The gap between the two metrics runs to tens of percentage points. A 2026 evaluation of 21 judge models across three benchmarks found that raw exact-match agreement overstated chance-corrected agreement by 33–41 percentage points on MT-Bench, for every model tested. The same study found that judge rankings shifted by up to 14 positions between benchmarks, so a ranking on one benchmark is a poor estimator of rankings on others. A 2024 study of judges on a question-answering task reported kappa of 79 and 84 for Llama-3 70B and GPT-4 Turbo, well below the human-to-human figure of 96. It also found that similar kappa could hide score deviations of up to 20 points when judges were compared by percent agreement.

You will often see the claim that strong judges match human preferences more than 80% of the time, the same level at which humans agree with each other (Zheng et al., 2023). That figure is raw agreement, not chance-corrected: MT-Bench is among the benchmarks that the 2026 study says rely on raw agreement as their headline validation metric. For the general case, see our page on validating LLM-as-judge scoring for production evaluation. This page applies it to a claims-triage gate.

What’s a defensible way to calibrate judge thresholds against human-labelled gold cases?

A threshold is defensible when you tune it on a gold set that contains enough real failures to measure the judge’s true-negative rate, report it with error bars, and check it again whenever the judge or the traffic changes. Start with the true-negative rate, where judges in published studies are weakest. A 2025 study of 14 LLM judges on programming feedback found true-positive rates usually above 96% and true-negative rates typically below 25%, so the judges systematically overestimated precision. For a triage gate, this asymmetry means a misrouted claim is far more likely to be passed than a correct one is to be blocked.

Three techniques make thresholds sturdier:

• Bias-corrected estimation. The same 2025 study modelled each judge’s true-positive and true-negative rates from a small human-annotated set. With five annotated datasets, maximum estimation error fell to 1.2%, assuming those rates stay stable across generators.
• Selective judging. The Trust or Escalate framework (2024) has the judge abstain when its confidence is low and passes those cases to a stronger model or a human. On a Chatbot Arena subset, its calibrated statistical guarantee was over 80% human agreement with almost 80% coverage.
• Error bars. Anthropic’s 2024 guidance on eval statistics recommends reporting standard errors, clustering them when questions come in related groups (the adjustment reached 3x in its example) and comparing models on paired question-level differences.

The gold set is a measurement instrument too, and it can be wrong. In the A/B-test study, 44% of the “ground-truth” labels came from tests that never reached statistical significance, and the judge agreed more with those unreliable labels than with reliable ones. The authors recommend stratifying judge accuracy by label quality as standard practice. Claims-triage gold labels from a single reviewer, or from disputed cases, call for the same split.

The judges themselves also change over time. Chen, Zaharia and Zou (2023) found that GPT-4’s accuracy at identifying prime numbers fell from 97.6% in March 2023 to 2.4% in June 2023 on the same questions. A hosted judge can change underneath a fixed threshold in the same way, so periodic recalibration against freshly labelled samples belongs to production AI reliability, not one-off setup. The recalibration cadence is a judgement call: tie it to judge-model updates and to shifts in claim mix rather than to the calendar alone.

Should we use a panel of judges or a single strong judge model?

Use a single strong judge when its kappa against your gold set is high and stays stable across recalibrations. Move to a panel when agreement is low, when a wrong pass is expensive, or when the judge shares a model family with the system under test. The self-preference risk is documented: in the 2024 Panel of LLm evaluators (PoLL) study, a GPT-4 judge ranked a GPT-4 variant second when its actual position was fourth. The same study found that a panel of smaller models from disjoint families outperformed a single large judge across six datasets, showed less intra-model bias and cost over seven times less.

Panels have their own failure mode: the votes are correlated. The A/B-test study measured a one-way intraclass correlation of 0.34–0.51 on panel votes, so a 16-vote panel carried only 1.9–2.6 effective independent votes. Unanimity there did not mean accumulating evidence. When you measure a panel, report three numbers: pairwise kappa (or Krippendorff’s alpha) between judges, each judge’s kappa against humans, and the effective number of independent votes.

How you combine the votes matters too. In the 2025 agreeableness study, majority consensus reached a true-negative rate of only 19.2%, while a minority-veto rule reached 95.5% true positives and 30.9% true negatives. For a release gate where a missed failure costs more than a false alarm, veto-style aggregation fits the asymmetry better than a simple majority.

Can the suite report reasoning, refusal rate, factuality and formatting as separate signals?

It can, but only if you design the categories to be independent and then test that they really are. The reason to bother is that models change unevenly across tasks. Between March and June 2023, GPT-4 became much worse at identifying prime numbers, less willing to answer sensitive questions, and more prone to formatting mistakes in generated code. Averaged into one score, those movements can partly cancel, and the release review sees a small dip with no clear cause.

Introduce a change that should affect only one category, such as a formatting-only perturbation of correct answers, and confirm that the factuality and reasoning scores do not move. Score each category with its own standard error, clustered where questions come in related groups. The wider case is made in stratified evaluation, and why one blended metric isn’t enough.

How do we test that a synthetic prompt shift is labelled drift, not regression?

Hold the model fixed, shift the inputs, and check that the drift signal fires while the regression signal on familiar inputs stays flat. Then reverse the experiment. Distribution-shift decomposition gives a way to draw the distinction. The DISDE method (2023) splits a performance drop into three parts: an increase in harder examples already seen in training, a change in the relationship between inputs and outcomes on shared inputs, and poor performance on inputs that were rare or unseen. Prompt drift shows up mainly in the first and third parts.

A workable procedure for a triage harness:

1. Freeze the candidate model and the judge configuration.
2. Build a shifted prompt set: new claim types, different phrasing, a different customer population.
3. Run a shift detector on the score stream. In a 2026 study, a sliding-window Kolmogorov–Smirnov monitor detected 86.8% of constructed shifts under control validation, though the authors note this measures a change in the score distribution, not a drop in accuracy. The “Failing Loudly” study found two-sample testing on representations from pre-trained classifiers worked best across the shifts it examined.
4. Confirm that the regression signal on the shared-input slice stays within its error bars.
5. Reverse the test: degrade the model on unchanged prompts and confirm that the regression signal fires while the drift monitor stays quiet.

Calibrate the detector on traffic that looks like production. In the same 2026 study, the KS false-alarm rate rose from 4.3% to 23.4% on heterogeneous traffic and recovered to 2.6% after recalibration on a representative production sample. Static calibration also breaks under prompt drift: the PromptShift-CRC synthetic pilot found that fixed conformal risk control failed after the shift, while drift-aware methods recovered much of the target coverage. A harness that labels synthetic drift correctly has passed a necessary test, not a sufficient one, because no synthetic set spans every way real claims traffic can move.

Is a known adversarial-prompt set flagged distinctly from a capability drop?

It should be, and the cleanest way to show it is a disagreement canary: a second scorer whose divergence from the primary judge raises its own alarm, separate from the capability categories. In the 2026 safety-classifier study, a score-disagreement canary flagged 75.5% of GCG-attacked inputs against 12.2% of random-token controls, and the best single canary reached 93.9%. The authors’ deployment recipe is to choose a canary that the attacker has not optimised against, estimate benign disagreement on held-out traffic, set a threshold for the false-positive rate you can accept, and alert on sustained disagreement. They treat each alarm as a triage signal for human review.

Your adversarial set should include attacks on the judge itself, not just on the model. The 2024 study of judges found that some could be fooled by dummy answers such as “Yes” and “Sure”. The silent failure is worse still. Reasoning-model judges given tight token budgets can return empty responses that a naive parser turns into a default score. If that default is “safe”, the pipeline would pass every adversarial input undetected. Invalid or empty judge output needs its own reported category, never a quiet default to pass. The expected signature is clear: the adversarial set trips the canary, while the capability scores on the clean set do not move.

Judge biases that distort per-category scores

Several documented judge biases matter for a per-category breakdown, because they do not act evenly across answers. Position bias is well documented. Wang et al. (2023) showed that swapping the order of two responses was enough to let Vicuna-13B beat ChatGPT on 66 of 80 queries with ChatGPT as the evaluator, and with GPT-4 as evaluator, position mattered most when the two responses’ scores were close. Balanced position calibration, which aggregates verdicts across orderings, is one mitigation the authors propose.

High repeatability does not rule out bias. The 2026 study of 21 judges found test-retest reliability above 0.95 alongside position bias above 0.10 in two production-deployed judges. Its protocol treats that combination as a failure mode, not a strength. Verbosity bias was small in that cohort (below 0.011 under a single pairwise rubric), which differs from earlier reports.

Other weaknesses fall on specific kinds of answer. Judges tend to be lenient on under-specified answers and are better at catching completely wrong answers than partially wrong ones. Low true-negative rates may vary with error type and difficulty. The criteria themselves move as well. The EvalGen study (2024) described “criteria drift”, where grading outputs changes what reviewers think the criteria should be. That is a reason to version the rubric alongside the gold set.

A validation checklist for an LLM evaluation harness

Check	What to measure	Pass looks like
Chance-corrected agreement	Cohen’s κ or Krippendorff’s α against human gold labels, alongside exact match	κ is the headline number
Position bias	Paired AB and BA runs, reporting abs(P(A wins) − 0.5)	Bias below 0.10, especially when test-retest exceeds 0.95
Repeatability	Test-retest over at least three runs at temperature 0, caching off	Stable verdicts across runs
True-negative rate	Judge catch rate on known-bad outputs	Measured on enough negatives to carry error bars
Decision validity	Replay of past releases with known production outcomes	No-go on the releases that hurt users
Panel independence	Inter-judge κ and effective independent votes	Effective independent votes are known, and unanimity is not read as accumulating evidence
Drift labelling	Shifted prompts with the model frozen	Score-distribution shift monitor fires under control validation
Adversarial labelling	Known attack set plus dummy answers	Canary fires, capability scores unchanged
Threshold stability	Recalibration against fresh human samples	Thresholds rechecked whenever the judge or traffic changes

Who can validate our LLM-judge harness before it gates claims-triage releases?

TechnoLynx builds eval, regression, drift and slice-metric harnesses for production AI systems such as claims triage. TechnoLynx tests whether the go/no-go calls track real regressions and whether the category scores separate regression, drift and adversarial failure. The Production AI Monitoring Harness engagement frames what counts as a regression and the gold labels it needs. It then picks the agreement statistics and the single-judge or panel design, builds the calibration and human-agreement study against your harness, runs labelled drift and adversarial sets through it, and wires calibrated thresholds and per-category dashboards into release gating with scheduled recalibration. A specialist helps most when the gate is already blocking or passing releases and nobody can yet say whether it is right. When choosing a partner, ask how they would test drift labelling and true-negative rate. Benchmark leaderboard positions are a weaker signal.

What the evidence still leaves open

The studies behind this page measure judges on benchmarks, A/B tests and safety classifiers, not on claims triage, so every threshold you adopt has to be re-earned on your own gold set. We found no published measurement of how fast judge-human agreement decays as hosted judges update, and none that shows reasoning, refusal, factuality and formatting signals staying statistically independent in practice. Both are properties your harness has to demonstrate, not assume. The open question for any team is how far its synthetic drift and attack sets are from the claims that will actually arrive next quarter. Answering it means going back to production outcomes, release after release.