To evaluate an LLM on your own task before committing to it, take the test prompts from your own workload, score the outputs against a written rubric whose automated judge has been checked against human labels, run each item more than once so variance shows, and record cost and latency at your real concurrency next to the quality score. LLM evaluations built this way give a non-engineer evidence they can read, defend and sign against. That evidence is itself an engineering output, not a marketing summary or a policy memo.
Why a leaderboard ranking cannot carry a committee decision
Public leaderboards present aggregate scores that blend tasks from undergraduate trivia to competitive programming. A 2026 meta-benchmarking study of financial-services model selection calls those averages insufficient for an organisation choosing a model for a specific operational purpose. The same study positions its own domain scores as a screening step that narrows the field before internal evaluation against production task distributions. It does not offer them as a replacement for that evaluation.
Rankings are also highly sensitive to small details. Alzahrani et al. (2024) found that small changes to MMLU, such as reordering the answer choices or changing how the answer is selected, moved model rankings by up to eight positions. Sclar et al. (2023) measured accuracy gaps of up to 76 points on LLaMA-2-13B from prompt formatting alone. They also found that which formats suit a model correlates only weakly from one model to the next. Contamination makes it worse: the 2026 paper ‘Hidden Measurement Error in LLM Pipelines’ found documented training-data leakage in all six benchmarks in its survey that publish their test sets.
In a 2024 interview study of benchmark use, practitioners said a low score can stop a model from being deployed, but a high score does not guarantee deployment. We think that asymmetry is the right reading: a leaderboard is for shortlisting, and the committee decision needs evidence the leaderboard cannot hold. We cover the committee side in why leaderboard rankings stall committees, and individual public numbers in separate pages: what the Chatbot Arena leaderboard measures and where it stops and what local Ollama benchmark numbers mean for procurement. For the general contrast, see public leaderboards versus task-specific evals.
What an LLM evaluation report needs to contain for a model-selection decision
The checklist below pairs each record in the pack with the check that makes it defensible. Two published rubrics ask for the same kinds of checks. NIST AI RMF 1.0’s MEASURE function asks for rigorous testing with measures of uncertainty, comparison against benchmarks, and formal documentation of results. Its subcategories state that test sets, metrics and tools are documented, and that performance is demonstrated under conditions similar to deployment. Google’s ML Test Score rubric gives one point for running a test by hand and documenting the result, and a second for automating it so it runs repeatedly. It also asks for model quality to be tested on important data slices, so that a single global metric cannot hide fine-grained problems. A full crosswalk from NIST AI RMF functions to domain-specific evidence sits on its own page.
Checklist: what the pack records and what makes each record defensible
| What the pack records | The check that makes it defensible | What goes wrong without it |
|---|---|---|
| Quality on your own prompt distribution | Test set built from production-like traffic, including typical, edge and adversarial cases | A score that describes someone else’s workload |
| Failure modes by risk category | Results split by risk class, each judged against its own threshold | An aggregate score that hides a risk-critical failure |
| Judge agreement | Chance-corrected agreement with human labels on a representative sample, rechecked when the judge changes | Scores that describe the judge rather than the model |
| Variance and significance | Repeated runs, paired tests, confidence intervals | A gap between candidates that is run-to-run noise |
| Operational noise | Per-request truncation flag (finish_reason = length) tracked as a metric |
Quota-truncated answers scored as model failures |
| Cost and latency at load | p95 and p99 at target concurrency, cost per quality-qualified success | A model that is cheap per call and expensive per usable answer |
| Leakage | Training cutoff recorded per candidate, plus post-cutoff or private test items | Memorised test items inflating a score |
| The tested artefact | Precision and configuration of the exact build evaluated | Approving a model you will not ship |
| Re-run triggers | Frozen suite re-run on any change to model, prompt or provider | Silent drift after approval |
The sibling pages go deeper on each part. One covers what the evidence pack contains beyond a leaderboard. Another covers structuring the pack around committee questions, and a third gives the evidence checklist per section. Procurement teams can use the checklist for turning an evaluation into sign-off-grade evidence, and there is a separate guide on scoring results for a non-technical procurement reviewer.
How a failure-mode catalogue reflects the buyer’s risk tolerance
A failure-mode catalogue reports results per risk category rather than as one global score. One accuracy figure cannot do that. The 2026 financial-services study warns that a high average domain score can sit alongside an unacceptable score on a risk-critical activity, and it recommends threshold or red-flag checks before deployment. A 2026 study of quantised clinical models shows the same gap with real numbers: one model lost 3.0% on HealthBench overall but 26.8% on emergency scenarios. NIST AI RMF 1.0 frames the bar in risk terms: a system’s residual negative risk must not exceed the risk tolerance.
In practice, the pack groups test items into risk categories, writes an explicit threshold for each, and reports every category on its own so an improvement in one cannot hide a regression in another. Anthropic’s guidance on success criteria builds severity into the target itself, for example requiring that 90% of errors cause inconvenience rather than an egregious mistake. For the risk-critical slice, see reading recall in an LLM evaluation. For why one blended metric fails across conditions and sites, see stratified validation beyond aggregate accuracy.
Cost and latency under real load, beside quality
Cost and latency only compare like for like when every candidate is measured on the same workload, at the same concurrency, against the same definition of a successful answer. A 2026 testing guide from QASkills states the stakes plainly: a prompt change that raises accuracy by 1% but doubles output tokens, or pushes p95 latency past the service-level target, is a regression. The same guide notes that a mean latency of 1.2 seconds can coexist with a p99 of 9 seconds. Percentiles at target concurrency go in the pack, and averages do not. Concurrency effects are not linear either: a 2026 TianPan analysis of load testing describes 40 concurrent users being fine and 45 being catastrophic once the GPU’s KV cache fills.
Cost belongs on a per-successful-task basis. A QAJobFit guide (2026) points out that a 200 response with empty text, malformed JSON or a refusal of an allowed request may not count as a successful task. It recommends computing cost per quality-qualified task, not only per API call, from rates with an effective date. MLflow’s 2026 provider-benchmarking guide makes the same point with an example: a cheaper provider that fails 15% of tasks and forces retries can cost more. Size cuts both ways. The CEBench authors (2024) report that Llama3 8B reaches 95.8% of the 70B model’s BoolQ score with 11.75% of the memory.
Which ASR model is best? Only your own clips can say
The reliable answer comes from running each candidate on a representative sample of your own catalogue under one locked scoring method. TechnoLynx can frame that evaluation: defining the content strata and scoring protocol, building a word error rate and latency pipeline on your infrastructure, running vendor engines and open Whisper-class models on a stratified reel, checking that word error rate differences are statistically reliable, and delivering a comparison report plus a reusable pipeline. The first step is framing the problem: agreeing the content strata and scoring protocol.
LynxBenchAI publishes results showing how candidate devices run the public PyTorch stack, which is broadly indicative for the hardware choice where an open Whisper-class model will run on your own GPUs; the first step is reading its published results for the devices you are considering, though it does not run or score your speech workload.
Accuracy on your content type
Word error rate counts substitutions, deletions and insertions against a reference transcript, divided by the number of reference words. Text normalisation rules (casing, punctuation, how numbers are written) change word error rate, so they are fixed before any vendor runs. Otherwise three vendors’ numbers describe three different tests. Word error rate does not score who said what, so a diarisation failure on overlapping speech can hide behind a respectable number. We would give overlap and music clips a qualitative review alongside the metric. NIST AI RMF 1.0 makes the general point: accuracy figures should always be paired with test sets representative of expected use, plus details of the test method.
A genre-stratified harness before the contract
Stratifying the test set by genre (documentary versus reality TV, for example) and by acoustic condition stops a reel heavy with clean narration from flattering a vendor that struggles with crosstalk. We would score two or three vendors on the same audio, reference transcripts and scoring tool. If no reference transcripts exist yet, building them comes before inviting any vendor pilot. A small reel can still miss rare languages or heavy overlap, and a vendor model updated after the pilot invalidates the comparison, so the report states both limits.
Is INT4 better than INT8? Comparing quantised candidates fairly
A quantised candidate’s raw score mixes the model’s quality with the effect of its quantisation, which differs by model. A procurement evaluation reports them separately. LynxBenchAI covers the hardware side of the precision decision. Its published results show how each device runs the public PyTorch stack your own models run on, which is broadly indicative for custom PyTorch models, though it does not run or measure your workload.
A 2026 clinical study argues against a simple rule of thumb by bit-width. The 2024 study ‘Give Me BF16 or Give Me Death’ ran over 500,000 evaluations on the Llama-3.1 family. It found FP8 effectively lossless, well-tuned INT8 losing 1 to 3%, and INT4 weight-only quantisation rivalling 8-bit. The 2026 clinical study found INT8 GPTQ safe everywhere it tested, with a worst case of -1.9%. Under INT4, though, BioMistral-7B lost 19.7% on MedMCQA while Qwen2.5-7B lost 1.7%. The same study found that Qwen2.5-7B, the model most robust to INT4 on accuracy, suffered the largest INT4 safety degradation: a 17.8% drop in refusal rate.
Accuracy also hides changes in behaviour. Dutta et al. (Microsoft Research, NeurIPS 2024) saw quantised models stay within 0 to 2% of baseline accuracy while up to 13.6% of individual answers flipped between correct and incorrect. They propose reporting flips and KL divergence against the baseline. A 2025 ACL study found higher calibration error after quantisation in 41 of 48 measurements, which matters if your workflow relies on reading confidence scores in an LLM evaluation. Work on vision models found that stable accuracy under quantisation does not guarantee stable reasoning, the concern behind explainability in a procurement evidence pack.
The decision logic follows from that evidence. If every candidate runs at one fixed precision, we would compare raw scores directly. Then we would report two numbers: a model-quality estimate, and a quantisation-noise estimate taken from the spread across repeated quantisation runs. Ranking goes on the model-quality estimate, unless you will actually deploy at the noisy precision, in which case both are reported. That separation is a statistical estimate, not a guarantee, and it is only as good as the number of repeated runs behind it. Whatever the result, the evaluation records the precision and configuration tested. Tech Policy Press reports that nominal precision labels have understated actual compression by a factor of three to 14. Distillation raises its own questions, covered in what distillation training means for LLM procurement.
Test-set leakage: recording each candidate’s training cutoff
Score gaps between candidates can reflect what each one saw in training rather than how well it reasons. We would therefore record each candidate’s stated training cutoff, check the evaluation items against it, and report any overlap as a caveat. Leakage here means a spurious relationship created by how the data was collected or processed. It does not exist in the population the claim is about, so it inflates measured performance.
One check is a date comparison against the training cutoff. Roberts et al. (2023) found that GitHub popularity predicted GPT-4 pass rates only on problems released before its September 2021 cutoff. After the cutoff the association disappeared. LiveCodeBench (2024) evaluates newer models only on problems released after their cutoff, and saw clear performance drops at those dates for several models. On GSM1k, a freshly written mirror of a public maths benchmark, accuracy fell by up to 8% for some models. Proprietary test sets built from internal data, or items dated after the cutoff, reduce the risk further. US federal guidance in OMB M-25-22 asks agencies to evaluate with their own data, kept out of the vendor’s reach and as close as possible to deployment data.
We see retraining every candidate on a common cutoff as out of reach for a buyer comparing closed API models. Even where it is possible, it can penalise a model that was never designed for that data version. The posture we would take is a leakage audit with the overlap stated. A stated cutoff is the vendor’s claim, and date checks miss subtler leakage: paraphrased or translated test items can evade standard decontamination and still inflate scores.
Can the committee trust the scores? Judge and rater agreement
What is a high kappa score, and when is kappa misleading?
An LLM-as-judge score should be validated against human labels before it supports a quality claim. That starts with a written rubric that has labelled gold examples at each severity tier, before any agreement is measured. TechnoLynx builds this as an LLM-as-judge evaluation harness validated against human agreement, wired into CI as a pre-release quality gate.
No single kappa value is universally acceptable. A practical guide from Mangold International describes roughly 0.60 as a common practical minimum for Cohen’s kappa, 0.70 or higher as solid and 0.81 and above as almost perfect. It stresses that the value depends on the number of categories and their base rates. For release decisions, one practitioner guide recommends validating the judge on at least 100 human-labelled cases, with chance-corrected agreement at or above 0.80. Kappa also has a known failure on imbalanced labels, the kappa paradox. In a 2023 annotation study, raters agreed on 95.05% of items and still scored a kappa of only 0.495, while Gwet’s AC1 came out at 0.945. So kappa should be reported per category, alongside raw agreement and a paradox-resistant measure. A second rater double-codes a representative subset, commonly 15 to 20% of the material.
Once an LLM judge is faster, cheaper and consistently agrees with human labels, it can scale up the scoring. The MT-Bench study (2023) found strong judges such as GPT-4 agreed with human preferences more than 80% of the time, about the same rate at which humans agree with each other. The same work names position, verbosity and self-enhancement biases. A 2025 study of self-bias found every model ranked its own outputs first, which is why the judge should not be one of the candidate models, and any shared model family should be documented. Swapping answer order and discarding judgements that flip is the simplest position-bias check. The judge also has to be revalidated whenever its provider updates the model. Our view is that the judge works as a monitor between formal recalibration cycles, while human agreement stays the ground truth. Agreement figures on small samples or rare severity tiers are unstable, and the report should say so where it applies.
Agreement between human raters is checked periodically through the work, with reliability reported per category rather than as one pooled figure. Humans should stay involved for calibration, borderline cases and dimensions where agreement remains unstable.
Before blaming the model: rate limits, truncation and timeouts
We would attribute a score drop to the model only after excluding requests with timeouts, truncation or rate-limit events and seeing the clean requests still show the drop. That depends on per-request telemetry (HTTP status, retry count, truncation flag, latency, queue time) logged alongside every quality score. The telemetry has to be in place before the regression happens, because the root cause cannot be rebuilt afterwards.
Some failing requests look successful. TianPan’s 2026 analysis of LLM rate limits describes responses under token-quota pressure that return HTTP 200 and pass JSON schema validation, but are cut off mid-sentence. Starvation and priority inversion under shared quota can degrade high-priority traffic without producing a single 429 error. The analysis recommends tracking finish_reason equal to length as a production metric with an alert threshold. We would keep truncated requests visible in the pack rather than silently dropping them: truncation that shortens a model’s reasoning still lowers the quality users see.
Is the gap between two candidates real, or run-to-run variance?
A single output per prompt cannot capture run-to-run variability. The pack runs each item several times, estimates the variance, and gates the decision on a confidence interval or a paired significance test rather than on the difference between mean scores. A taxonomy of LLM failure modes cites reports of output divergence above 20 to 30% across repeated runs on multi-step reasoning tasks.
The 2024 paper ‘Adding Error Bars to Evals’ sets out the core practice: standard errors of the mean, clustered standard errors when questions come in related groups, analysis of paired per-question differences when comparing two models, and power analysis before running anything. The clustering adjustment can be large: its authors found up to a threefold difference in their example. Their worked power calculation shows that detecting a three-point difference 80% of the time, at a 5% false-positive rate, needs about 969 independent questions. Resampling helps too: raising repeats per question from one to ten cut the minimum detectable effect from 13.2% to 7.5% in their example. The same authors advise against lowering sampling temperature just to reduce variance. We read that as evaluating at the temperature you will deploy, since a cooler setting changes the behaviour being measured. When both candidates are scored on the same items, McNemar’s test applies, and an inconclusive result means the test set cannot resolve a difference that small. It does not mean the models are identical.
Where the variance actually comes from
Standard confidence intervals leave out variability from the choice of judge model, temperature and prompt wording. ‘Hidden Measurement Error in LLM Pipelines’ (2026) found naive standard errors 40 to 60% smaller than corrected ones. In its safety demonstration, the choice of judge panel accounted for 43.8% of the variance in the final estimate, the item-by-judge interaction for another 16.7%, and prompt wording for less than 0.5%. Committing to a single judge in that design increased the variance by 106%. The authors propose a small pilot to find the dominant source before spending the budget: at least 30 items, three prompt variants, two judges and three replications, roughly 540 API calls for one scoring function. Whichever component dominates tells you where to spend: more judges, more items, more prompt variants or more replications. We would re-estimate the noise band periodically rather than fixing it once. For combining new runs with earlier evidence, see Bayesian updating of eval results.
A fine-tuned small model against a prompted frontier model
The comparison needs a custom harness that borrows MLPerf’s discipline (fixed task, fixed hardware, accuracy reported together with latency and memory) rather than MLPerf’s benchmarks. TechnoLynx runs this comparison as part of the LLM Selection Pack. TechnoLynx frames it as a controlled experiment with a shared test set, an identical hardware target and matched decoding settings. TechnoLynx then instruments both systems with the same accuracy, latency and memory harness, and validates the deployed quantised model against the baseline before sign-off. For choosing the hardware target, LynxBenchAI’s published results from a disclosed benchmark harness are the measuring tool, broadly indicative for PyTorch models without running your workload.
We would report two numbers, not one: intent accuracy and slot exact match (or slot F1), separately. Exact-match scoring needs normalisation, such as of whitespace and case, before matching. The GSM1k authors found automatic extraction marking an answer of ‘24.0’ wrong against a reference of 24. If the frontier model is served remotely on hardware you cannot see, its latency is not comparable until it is proxied locally or the network share is separated out. An enterprise benchmark study found a 13B model working as well as a 70B model on legal classification and QA, making it the more cost-effective choice. A small intent set can still miss the tail cases where the two approaches diverge most, and the report should state that limit.
Who can build an LLM evaluation pack our approval committee will sign?
TechnoLynx builds the procurement-grade evaluation pack behind a model choice. That means an LLM eval suite scored on your own tasks and a risk-and-comparison report your approval committee can re-run on every model swap. It is engineering work, not policy advice or certification. Building in-house is realistic if you already have evaluation engineers with statistical depth and time before the committee date. A specialist earns its place when the deadline is close, or when the pack has to survive a challenge from model-risk or audit reviewers. When choosing a partner, ask whether you get the harness, raw records and frozen suite to keep, and whether every number can be re-run. TechnoLynx delivers it through the LLM Selection Pack, and the first step is framing the comparison as a controlled experiment.
The pack does not publish leaderboards or score models for general consumption. It suits buyers who must defend the model choice internally. Each milestone leaves an artefact showing what was known, what was scoped and how the decision was made. It does not suit a team that only wants the cheapest build or a single vendor ranking. Those teams can reasonably rely on an internal platform team running an open-source harness, or on the model vendor’s own published evaluations (which are vendor claims). Our wider work for AI infrastructure and production SaaS teams covers the serving and cost side once a model is chosen.
When should an LLM evaluation be re-run after a vendor ships a new version?
Re-run the frozen suite whenever the model, prompt templates, retrieval logic or provider changes, and also on a schedule, to catch updates nobody announced. The classic evidence comes from a 2023 study: GPT-4’s accuracy at identifying prime numbers fell from 84% in March to 51% in June. A 2026 study of hosted-model updates saw a backend SQL function pass every test one day and fail a safe-encoding test the next, which suggests silent infrastructure changes. Anthropic commits to at least 60 days’ notice before retiring a publicly released model and recommends testing replacements well before that date. Not every trigger needs a full re-run. A materiality check decides whether the change alters the risk profile the original assessment covered. OMB M-25-22 encourages agencies to require vendors to meet performance standards before deploying a new version, or to roll back if it fails. We go further into re-validating when the vendor ships a new version and into regression testing to catch model drift before release.
Where the evidence pack ends and benchmark methodology begins
The evidence pack applies measurement to one buyer’s decision. It does not define benchmark methodology, publish leaderboards or score models for general consumption, which is LynxBenchAI’s work, along with measuring the hardware candidates run on. The page on where benchmark methodology begins traces that hand-off, and LynxBenchAI is where the methodology and published results live.
Residual risk is the committee’s call, not the pack’s
No test strategy guarantees complete coverage of how an LLM will behave. A 2026 framework for testing LLM features in regulated software sets the realistic goal: reduce risk to an acceptable level, document the residual uncertainty, and put monitoring and incident response in place. A good pack therefore ends with what it could not establish: the risk categories with thin samples, leakage it could not rule out, and the precision it did not test. That list is what the committee actually signs against. For how regulation changes that list, see regulated vs unregulated evidence. For the whole sequence end to end, see the worked one-round approval example.
