A public leaderboard ranking is a claim about someone else’s task. An evidence pack is a claim the buyer can trace back to their own data, their own thresholds, and a dated run. That difference is the whole anatomy — and it is why a ranking table, however current, cannot be swapped in for a pack section.
This article is a parts list. Not how to run the evaluation, not how to structure the pack around the committee’s agenda, and not how to score models — just what each named section of a procurement-grade LLM evaluation evidence pack contains, what evidence sits behind it, and which sections have no leaderboard equivalent at all.
What does a procurement-grade LLM evaluation evidence pack contain beyond a public leaderboard?
Five sections, in the shape we see hold up under approval scrutiny:
| Section | What it contains | Leaderboard equivalent |
|---|---|---|
| Task-specific accuracy | Scored results on the buyer’s own prompt distribution, with sample counts, labelling provenance and a dated run | Partial — the metric idea travels, the task does not |
| Failure-mode catalogue | Named failure modes, reproducible triggers, and severity scored against the buyer’s risk tolerance | None |
| Cost-per-decision | Unit economics under the buyer’s real load profile, not per-token list price | None |
| Drift posture | The stated position on what happens when the model vendor ships a new version, and what gets re-run | None |
| Provenance and sign-off | Who produced each result, on what date, against which threshold, and who accepted it | None |
Four of the five have no leaderboard counterpart. A leaderboard is a single ranked axis; the pack is four independent axes plus the paper trail that makes them auditable. Substituting the first for the whole is the failure this document exists to prevent.
The task-specific accuracy section
This is the only section a leaderboard even gestures at, and it is where the substitution feels most defensible — both are accuracy numbers, after all. The divergence is the prompt distribution. A public benchmark measures a fixed public task set; the section here measures the prompts the buyer will actually send, sampled from real traffic or from a constructed distribution that names how it was constructed.
What sits behind the section, in practice:
- The prompt set itself, with its sampling basis stated — production logs over a named window, expert-authored cases, or a documented mix.
- Sample counts per intent or category, so a reader can see which conclusions rest on forty examples and which rest on four.
- Labelling provenance: who scored the outputs, against what rubric, and whether inter-rater agreement was measured.
- The run date and the exact model identifier, including version or endpoint, because both move.
- The threshold the number is being compared against — an accuracy figure with no accept/reject line attached is a fact, not evidence.
Strip any one of these and the section reverts to an assertion. We spend more review time on sampling basis than on any other line, because it is the one a committee member with a statistics background will always find.
Two sections that exist only because the leaderboard cannot carry them
The failure-mode catalogue and the cost-per-decision section are the pack’s real load-bearing content, and neither has a public analogue.
The failure-mode catalogue is scoped to risk tolerance, not to a generic harms list. A generic list is portable and therefore useless: it enumerates hallucination, prompt injection, refusal, and toxicity for every buyer identically. A scoped catalogue starts from the buyer’s decision consequences and works backwards. If the model drafts a customer reply that a human sends, a confident fabrication is a severe mode and a refusal is a mild one. If the model routes a ticket autonomously, silent misrouting outranks both. Each entry names a reproducible trigger, an observed rate on the buyer’s own prompt set, a severity grade tied to the consequence, and either a control or an explicit acceptance.
Cost-per-decision is a unit-economics claim under the buyer’s load profile. Per-token list price is an input, not the section. The number that matters is the fully-loaded cost of one decision the business cares about — including retries, retrieval calls, context overhead, and whatever fraction of cases escalate to a human. A model that is cheaper per token and needs two rounds of prompting plus a 15% human review rate is not cheaper per decision. That arithmetic depends entirely on the buyer’s volume and escalation policy, which is precisely why no public source can supply it.
Drift posture: the section buyers skip
The pack is dated the day it is signed. The model is not. A drift posture states, before approval, what the organisation will do when the vendor ships a new version — which triggers count as material, which sections get re-run, and who decides. Recording it inside the pack is what makes a later re-evaluation a re-run of named sections against a comparable baseline rather than a fresh project with fresh arguments about methodology.
The same structural property that makes the pack committee-legible makes it re-runnable: each section is a bounded claim with its evidence attached, so a version bump touches some sections and provably not others. A ranking table has no sections, so a new leaderboard snapshot invalidates all of it or none of it, and nobody can say which.
Where this pack stops and benchmark methodology begins
Two different deliverables get conflated. Benchmark methodology defines the measurement construct — the task taxonomy, the scoring definitions, the reproducibility conditions under which a number means anything. It is maintained independently and applies across many buyers; that work sits with LynxBenchAI. The evidence pack applies a methodology to one buyer’s task and defends one buying decision to one committee. It publishes no methodology and scores no models for public consumption.
Practically, the boundary shows up in a citation. When the accuracy section says the outputs were scored a particular way, the definition of “scored that way” lives in the methodology layer and the pack cites it. When the failure-mode catalogue grades severity, the grading scale is the buyer’s, not the methodology’s. Keeping the two separate is what lets a methodological objection be answered without reopening the procurement decision.
The pack anatomy described here is the artefact side of the broader question of how model evidence earns approval — we develop the underlying evidence discipline in our work on AI governance and trust, and the committee-facing ordering of these same sections is covered in structuring an evaluation pack around approval committee questions.
One open question we have not settled: how thin the task-specific accuracy section can get before a committee should reject it outright. Sample counts in the low tens are common under time pressure, and we have not found a defensible floor that holds across risk tiers.
Frequently Asked Questions
What does a procurement-grade LLM evaluation evidence pack contain beyond a public leaderboard, in practice?
For Inside an LLM Evaluation Evidence Pack, five named sections: task-specific accuracy on the buyer’s own prompt distribution, a failure-mode catalogue scoped to the buyer’s risk tolerance, cost-per-decision under the buyer’s real load, a stated drift posture, and provenance with sign-off. Four of those five have no leaderboard counterpart at all., task-specific accuracy answers “is it accurate enough on our task”; the failure-mode catalogue answers “what happens when it fails on our highest-risk case”; cost-per-decision answers “what does this cost at our volume”; drift posture answers “what happens when the vendor updates”; provenance answers “who says so, and when”.
What evidence sits behind the task-specific accuracy section? The prompt set with its sampling basis stated, sample counts per intent, labelling provenance including the scoring rubric and any agreement measurement, the run date and exact model version, and the accept/reject threshold the number is compared against. Missing any of these turns the section back into an assertion.
Which sections have no public-leaderboard equivalent at all, and why can a ranking table not stand in for them? The failure-mode catalogue, cost-per-decision, drift posture, and provenance. A ranking table is a single ordered axis over a fixed public task set; it carries no severity scoping to the buyer’s consequences, no unit economics under the buyer’s load, and no dated traceability.
How is the failure-mode catalogue scoped to the buyer’s risk tolerance rather than to a generic harms list? It starts from the consequence of each decision the model participates in and works backwards to the modes that matter there, so severity grades differ between a human-reviewed draft and an autonomous action. Each entry carries a reproducible trigger, an observed rate on the buyer’s prompt set, and either a control or an explicit acceptance.
How does the pack record cost-per-decision and drift posture so a later model version can be compared like-for-like? Both are recorded as bounded claims with their inputs named — load profile, retry and escalation rates for cost; trigger list and re-run scope for drift. Because each section is separable, a vendor version bump re-runs the affected sections against the same stated inputs instead of restarting the evaluation.
Where does the evidence pack stop and benchmark methodology (LynxBenchAI) begin? Methodology defines the measurement construct — task taxonomy, scoring definitions, reproducibility conditions — and is maintained independently of any one buyer; that layer is LynxBenchAI’s. The pack applies a methodology to one buyer’s task to defend one decision, cites the methodology rather than restating it, and publishes no scores.
Putting Inside LLM Evaluation Evidence to work
Treat Inside LLM Evaluation Evidence as an engineering problem with a measurable answer, not a positioning question. The teams that do tend to ship the boring, correct version first.
