A cheaper open-weight model is good enough for your work when it holds up on your own documents. That means scoring it against a gold set your subject-matter experts built, putting a confidence interval on the difference from the frontier model, breaking errors down by case type, measuring the current manual process on the same scale and pricing it per document at your real volume. A leaderboard rank answers none of this. Leaderboards tend to show scores and rankings with no indication of whether the differences between them are statistically significant. And when benchmark data has leaked into a model’s training data, its evaluation results are distorted.
The LLM evaluation framework below runs in the order we would work through it. It starts with the baseline you already have, then covers the metrics, the gold set, a paired comparison with cost, the statistics, where the test data comes from, and the case types where a single average can hide misses.
Why measure today’s manual process with the model’s metrics?
A model should be compared with the process it replaces as that process actually performs, not with an assumed error rate of zero. In practice, that means taking a recent window (the last quarter, say), pulling the logged outcomes of the manual triage or review and computing its error or mis-routing rate. Use the same definition of an error, the same gold standard and the same period you will use for the model. If the two measurements differ in any of these, the funding decision compares two numbers that describe different things.
In the LLM Selection Pack, TechnoLynx holds both candidate models to the same success metrics and SME-adjudicated gold set, so the comparison with your measured baseline is like for like; the first step is agreeing the representative document sample, the gold-set requirements and the success metrics with TechnoLynx.
There is evidence that manual work carries real error rates. A 2009 study of spreadsheet error estimation cites Panko’s 2005 review of 13 field studies: an average cell error rate of 5.2% across 43 operational spreadsheets, and at least one error in 94% of 88 spreadsheets. The same paper reports a later study with a much lower rate of 1.79% over 270,722 formulas, falling to 0.87% when only wrong results are counted. That range is the point. The rate depends on what you count as an error, so the definition has to be fixed before anyone counts.
A digital-forensics study of preliminary analysis sent every device for a full investigation, which served as the gold standard. Every examiner scored a precision of 1, but recall ranged from 0.17 to 0.91 across five examiners. One review makes the general case: existing human baselines in model evaluations are neither rigorous nor well enough documented to measure performance differences robustly. The authors back this with a checklist review of 115 baseline studies.
Three caveats decide how much weight the baseline can carry. Historical logs may be incomplete or never audited. A process change mid-quarter splits the window into two regimes. And reviewers may disagree on what counts as a mis-route, so that label needs a written definition before anyone computes a rate. When only part of the history can be re-reviewed, a Bayesian approach can combine expert prior belief with partial test results to estimate the error rate. It also gives a principled point at which to stop checking.
Judging finance summaries beyond “looks fine”
For finance summaries, the rubric needs layers. First, a claim-level check that traces every number and factual statement in the summary back to the source document. Second, an omission check against a list of facts each summary must contain. Third, a blind analyst preference test as a separate, secondary signal. Generic overlap scores alone are not enough for this job. OpenAI’s evaluation guidance lists relying only on academic metrics like perplexity or BLEU as an anti-pattern. A Practical Guide for Evaluating LLMs and LLM-Reliant Systems (2025) argues that no single metric is enough and that term-overlap metrics like ROUGE should be complemented with semantic similarity, natural language inference (NLI) and autorater checks.
For summaries, TechnoLynx uses the LLM Selection Pack to define the success metrics with you and validate both candidate models’ output against your SME-adjudicated gold set.
The guide presents NLI as valuable for evaluating factuality. It tests whether a statement follows from a premise, is unrelated to it or contradicts it. Numbers need an explicit tolerance rule. One 2026 financial extraction benchmark, for example, counted numerical fields as correct within ±2%. For figures an analyst will repeat to a client, set that tolerance at exact match and write it into the rubric. Omissions are harder to automate. The practical route is to have experts list the key facts per document in the gold set and score each summary’s recall of that list.
LLM judges can scale this work, but they bring their own biases towards verbosity, position, self-enhancement, lists and particular styles. OpenAI’s guidance is to scale an LLM judge only once it consistently agrees with human annotations. For the human signal, the same guidance describes a randomised, blinded test in which reviewers rank or grade outputs without knowing which system produced them. It recommends pairwise comparison or pass/fail for reliability.
The decision logic is simple. If factual and numeric errors are the main risk, claim-level verification is the gate and the preference test builds trust. Finance needs both, because analysts have to accept the tool and the numbers have to be right. Budget for analyst time, though. Card et al. (EMNLP 2020) found that typical human rating study designs are underpowered to detect small model differences. The rubric needs re-validating whenever the model or the document types change.
Proving an open-weight model extracts our clauses as reliably as the API
The evidence that settles clause extraction is a documented benchmark. Run both models on a representative sample of your own contract types and score each clause type against an SME-adjudicated gold set. Then compare the error-rate difference with a tolerance your legal risk owners have signed off. Public results tell you a gap may exist, not how large it will be on your contracts. On the CUAD contract dataset, the ContractEval benchmark found GPT-4.1 reaching an F1 of 0.641. The best open-source model, Qwen3 8B in thinking mode, reached 0.540, about 16% lower. The same study found that open-source models more often answered “no related clause” when a relevant clause was present, and that quantisation sped up inference at a cost in performance. Correctness was significantly lower on less common or longer clauses.
For clause extraction, TechnoLynx uses the LLM Selection Pack to run both candidate models on a representative sample of your contracts and validate the results against your SME-adjudicated gold set.
Those findings tell you which metrics to use. ContractEval scored correctness with F1 and F2 because positive and negative cases are imbalanced. It counted a false negative whenever the model said “no related clause” or failed to cover the labelled span. It also tracked the false “no related clause” rate on its own, which was 0.07 for both GPT-4.1 variants and over 30% for Qwen3 8B AWQ in non-thinking mode. We would report that rate per clause type alongside F1. A model that quietly skips indemnity clauses fails in a different way from one that extracts them imprecisely.
Building the SME gold set, and who owns it
A benchmark on 10,000 SEC filings shows what a credible gold set involves. Automated pre-annotation from XBRL tags covered about 60% of financial metric fields. Twelve CFA- or CPA-credentialed annotators then reached inter-annotator agreement of Cohen’s κ = 0.91, and a senior analyst adjudicated the disagreements. Agreement should be reported per annotation, not just in aggregate. That lets you ask whether the model fails on the same items the experts struggled to agree on. A gold standard is only as good as the experts who create it.
Ownership should sit with the function that carries the risk (legal, for contracts), with the engineering team maintaining the tooling around it. The tolerance itself is a legal-risk judgement, not a statistical output. It needs an explicit sign-off, together with the caveat that a sample result says nothing about contract types or edge cases the sample did not contain. If the open-weight model falls outside tolerance, there are two options: keep the API model, or add a review step for the open-weight output.
Cost per document at your volume, not price per token
To find out how much accuracy a switch costs, run the frontier API model and the candidate open-weight model on the same representative sample. Score both for precision and recall against the gold set, then compute cost per document at full volume, 400,000 documents or whatever yours is. Google’s machine learning guidance recommends recall when false negatives cost more than false positives, and F1 over accuracy when classes are imbalanced. Score both models on the same documents and analyse the paired differences. Scores on the same questions tend to correlate, so pairing reduces the standard error for free.
Through the LLM Selection Pack, TechnoLynx scores both candidate models for precision, recall and cost per document on the same sample and reports the cost-quality trade-off.
Cost belongs in the same table as accuracy. Kapoor et al. (2024) argue that what a downstream buyer needs is dollar cost relative to accuracy, not proxies such as active parameter count. They also argue that evaluations should report input and output token counts, so the cost can be recalculated when prices change. The SEC-filing benchmark, which compared orchestration architectures rather than a model swap, shows the shape of the result. Its most accurate configuration reached an F1 of 0.943 at $0.430 per document. An optimised hierarchical configuration reached 0.924 at $0.148. The authors note their prices date from January 2025 and the cost ratios will shift.
The choice is rarely binary. In the same study, routing between models cut cost by 51.3% against using one frontier model for everything, while keeping 98.2% of the F1. FrugalGPT reported matching the best individual model with up to 98% lower cost through a learned cascade of models. Before treating a model swap as the only cost lever, look at the serving path of your AI infrastructure. Batching, caching, routing, quantisation and serving configuration can often bring cost and latency down without replacing the model at all.
Worked example: normalising self-hosted cost against an API
LeanLM’s 2026 analysis gives the formula: effective cost per million tokens equals the hourly GPU rate times an ops factor, divided by saturated throughput times 3,600 times utilisation, times one million. Its inputs are an H100 at $2.89 an hour serving about 8,000 tokens per second on a mixed workload, and a hosted price of $0.60 per million output tokens for the same open model.
- At full utilisation with no ops overhead: $2.89 ÷ (8,000 × 3,600 × 1.0) × 1,000,000 ≈ $0.10 per million tokens.
Once LeanLM’s 3–5× ops factor is included, its break-even against that hosted price is roughly 50 to 80% sustained utilisation. Use aggregate measured throughput, not a demo figure. The same analysis reports a 13 to 16× gap between batched throughput and what a single interactive user sees. Multiply the per-token result by your measured tokens per document to get the number procurement needs. Where the open-weight route means renting or buying accelerators, LynxBenchAI gives measured hardware results rather than spec-sheet peaks. It measures the hardware, not your pipeline or your documents.
Is 200 contracts enough to tell a real recall gain from noise?
Whether 200 contracts are enough depends on the size of the gain and the variance, not on a rule of thumb. Compute the confidence interval on the paired recall delta from the results you already have, and run a power analysis before deciding whether to grow the set. For scale, the DISCO evaluation suite (2026) chose 500 samples because that gives 95% intervals of roughly ±4 percentage points and detects gaps of 7 to 9 points at 80% power. A 200-item set gives wider intervals than that, so a gain of a few points will usually sit inside the noise.
Smaller gains need much larger sets. Miller’s 2024 analysis works an example with assumed parameters in which detecting a 3-point difference needs about 969 independent questions. Its general advice is that evals should have at least 1,000 questions to give a reliable signal. Contracts add a further complication: several clauses come from each document, so the items are not independent. Clustered standard errors account for that, and in Miller’s real-world example the adjustment widened them by up to three times.
Recall from a sample deserves its own care. Webber’s work on recall confidence intervals recommends a beta-binomial method with a half prior. It shows the plain recall estimator is biased when prevalence is low and samples are small. It also argues that wide bounds must be reported, not hidden. Where an LLM judge does the scoring, a 2026 study of hidden measurement error in LLM pipelines found naive standard errors 40 to 60% smaller than ones that account for judge and prompt variance. Report an effect size alongside any p-value, because a low p-value says nothing about how large the difference is.
Fine-tuning, RAG or a prompted frontier model on the same task
Compare all three approaches the same way: the same gold set, the same metrics, and a cost line that separates one-off cost from per-document cost. The quality ordering is not fixed in advance. In a 2024 Microsoft study on agricultural question answering, fine-tuning raised accuracy by over 6 percentage points and RAG added about 5 more on top. The authors flag the high upfront cost of fine-tuning on new data as a significant consideration. They describe RAG as cheap to set up but inclined to enlarge prompts and produce wordier output. Kapoor et al. found RAG and full long-context input almost level on accuracy (67.89 against 67.81) at $52.8 against $99.8 in total cost. They also showed how the benchmark’s own setup made RAG look worse than it would in real use.
That distinction between fixed and variable cost decides many of these comparisons. The more a system is used, the more the per-run variable cost dominates the one-off optimisation cost. A RAG option also needs its retrieval scored separately: hit rate, MRR and NDCG for the retriever, and context relevance, faithfulness and answer relevance for the generated output. The RAG survey warns that these metrics are not yet standardised. For the vendor and contract side of this comparison, our piece on balancing cost and quality across model choices and vendor strategies takes the decision further.
Can we test on our own tickets and documents instead of trusting published benchmarks?
Yes. An evaluation harness built from your own customer-service tickets and documents measures the models on the inputs they will actually see. Public leaderboards then become a sanity check, not the basis for the decision. One reason is contamination. When benchmark content leaks into training data, traditional benchmark testing overestimates performance. One survey reports contamination ranging from 1% to 45% across more than 15 LLMs and six QA benchmarks. When researchers built GSM1k, a new set of problems that mirrors a popular maths benchmark, some models dropped by up to 13% in accuracy. Frontier models showed minimal signs of overfitting, so the effect is not universal. Version numbers are no safer a guide. DISCO found a newer OCR model version consistently worse than its predecessor and concluded that deployment decisions should rest on results from representative data.
TechnoLynx builds this harness in the LLM Selection Pack, running both candidate models on a sample of your own documents and handing it over for your team to run or extend.
Nobody can tell you in advance how much accuracy will change when you move from generic benchmarks to your own use case. That gap is specific to you, and the harness exists to measure it. A 2025 practical guide sets five principles for the dataset: a defined scope, demonstrative of production use, diverse, decontaminated from training data, and dynamic as the real world changes. It adds a useful warning sign: high metric scores alongside low user satisfaction mean the dataset is not representative. Private datasets lower the chance that test items appear in pre-training data.
Ownership follows from how the harness is built. If it uses your data, your gold labels and a documented methodology, it stays yours, independent of any vendor’s proprietary benchmark. You can then re-run it on the next model release. That ownership costs effort to keep up. The dataset has to evolve as real-world ticket patterns shift.
Will the cheaper model miss the urgent cases the expensive one catches?
To find out, stratify the test set by case type and urgency, then compare false-negative rates on the high-risk strata directly. The tolerable miss rate on urgent flags should then be set by the programme or legal stakeholders, not by the vendor. A single accuracy figure can hide exactly this failure. Oakden-Rayner et al. call it hidden stratification: a model with high overall performance that consistently misses a rare but important subtype. They found relative performance differences of over 20% on clinically important subsets in medical imaging. The domain is different, but the mechanism carries over to urgent eviction or safety-risk cases that make up a small share of intake.
In the LLM Selection Pack, TechnoLynx validates both candidate models against your SME-adjudicated gold set, so the comparison rests on your own cases.
Statistical comparisons can be run separately for each category, because two systems may differ only on the hard instances. Failure analysis by category also shows why a system fails, not just how often. The SEC-filing benchmark, for instance, attributed 39.3% of the reflexive architecture’s errors to ambiguous disclosure resolution. Barocas et al. (2021) set out the design choices a disaggregated evaluation involves and the trade-offs between them. Our article on why one blended metric isn’t enough covers stratified validation across conditions in more depth.
The threshold itself is a policy choice. In the forensics triage study, the preliminary process could accept false positives but required a false-negative rate of zero for the decision to send a device for full analysis. If the cheaper model’s urgent-case false-negative rate exceeds the threshold, it should not run triage alone. Keep a human in the loop, or route urgent-looking cases to the stronger model. Two caveats apply. Rare strata hold few examples, so their rates need the same interval treatment as the headline number.
A sign-off checklist before switching models
| Check | What it catches |
|---|---|
| Manual process measured with the same metric, gold set and time window | Comparing the model with an imaginary error-free human |
| Gold set built by SMEs, with agreement reported per annotation and an adjudication step | Labels that are themselves ambiguous or wrong |
| Metrics chosen by error cost: recall, F1, false “no clause” rate, numeric exactness | A score that rewards the wrong behaviour |
| Paired comparison with a confidence interval and a power analysis | Treating noise as a real improvement |
| False-negative rates per case type and urgency stratum | A good average hiding misses on rare, high-stakes cases |
| Test data drawn from your own production, kept private | Scores inflated by benchmark contamination |
| Cost per document at volume, including utilisation and ops for self-hosting | A per-token saving that disappears in production |
| Tolerance signed off by the risk owner | An engineering team making a legal or policy call |
Who can run this evaluation for us without a data science team?
TechnoLynx runs this cost-quality evaluation as a scoped engagement for teams that need a defensible model decision without hiring data scientists. With you, TechnoLynx defines the representative document sample, the gold-set requirements and the success metrics. TechnoLynx chooses metrics such as precision, recall and cost per document, builds a harness that runs both candidate models on the same sample, validates the results against your SME-adjudicated gold set and reports the cost-quality trade-off. The harness and findings are handed over for your team to run or extend. TechnoLynx packages it as the LLM Selection Pack. Whether a platform you already use offers its own model benchmarking is worth asking your vendor, along with whether you keep the gold set and harness afterwards.
The engagement covers the evaluation itself. If you expect to iterate on models continuously, building internal evaluation capability may pay off better in the long run, and open-source evaluation frameworks are a reasonable starting point for a team with the skills to use them.
What the numbers still leave to you
A rigorous harness narrows the decision. It doesn’t make it. You end up with a measured gap, an interval around it, per-stratum miss rates and a cost per document. Whether a 3-point recall loss on termination clauses is acceptable for a given saving is still a judgement about risk, made by whoever answers for the miss. The harness also ages. Prices move, models get new versions and your document mix drifts, so the most useful thing to have at the end of a procurement decision is a test you can run again next quarter, not a single verdict.
