A section of an LLM procurement evaluation pack is defensible when its central sentence can be traced to a run someone else could re-execute. That is the whole test. Everything else here is the per-section form that test takes.
The failure mode is easy to describe because it is so common. The pack gets written as narrative — one section per committee concern, each closing with a confident sentence about accuracy, cost or risk. It reads well. Then a committee member asks what sits behind the sentence: which prompts, how many, sampled from where, scored by whom, against what threshold. The section collapses into opinion, and the decision defers by a cycle.
The fix is not better writing. It is fixing the evidence requirement per section before anything gets drafted, so that each section is assembled against a specification rather than polished after the fact. In our procurement work we treat this as a checklist step with a hard gate: if the required evidence for a section does not exist yet, the section does not get written — the run gets scheduled.
What evidence does each section need?
| Section | Minimum evidence | Recorded as | Fails review when |
|---|---|---|---|
| Task accuracy | Named prompt distribution, sample count per intent, scoring rubric, scorer identity | Prompt set file + rubric + per-intent score table, run-dated | Score reported without sample size or scoring method |
| Failure-mode catalogue | Reproducible triggering input per mode, observed frequency, position against the buyer’s stated risk tolerance | One row per mode: trigger, frequency, severity, tolerance verdict | Modes listed qualitatively, no trigger, no frequency |
| Cost per decision | Measured token and latency profile under the buyer’s load profile | Tokens in/out per decision type × measured price × concurrency assumption | Cost derived from vendor list pricing alone |
| Drift posture | Dated baseline run on a pinned model version | Baseline artefact ID, version string, date, re-run scope | “We will re-run when the vendor updates” with no baseline |
| Sign-off record | Named owner per section, date, scope of what was approved | Signature table with section-level attribution | Single global approval line |
That table is the article. The rest explains why each row is drawn where it is.
Task accuracy: three numbers, not one
The accuracy section is where narrative creeps in fastest, because a single percentage feels like evidence. It is not, unless three things travel with it. First, the prompt distribution — named, versioned, and described in terms of the buyer’s intents rather than “a representative sample”. Second, the sample count per intent, not just in aggregate; an aggregate of 400 prompts that contains nine examples of the highest-risk intent tells the committee nothing about that intent. Third, the rubric and the scorer: human graders with an inter-rater check, a model-as-judge with its own calibration note, or exact-match against a labelled set — each is acceptable, and each must be stated.
A task-accuracy claim without a sample count, a rubric and a named scorer is not evidence; it is a summary of someone’s impression. Committees discount it correctly.
Failure modes: the trigger is the evidence
A failure-mode entry earns its place when someone can re-trigger it. That means the pack carries the actual input — prompt text, parameters, model version — not a description of the category of input. Alongside it: observed frequency on the buyer’s distribution, expressed as a rate over a stated sample, and an explicit verdict against the risk tolerance the buyer has already written down.
The third element is the one usually missing. A hallucinated citation at 1.2% is tolerable in a drafting assistant and disqualifying in a regulatory filing pipeline; the frequency alone does not tell the committee which situation it is in. The section is only closed when each mode is positioned against tolerance, either as accepted with a named control or as blocking.
Cost per decision: measured, not quoted
Vendor list pricing is an input to a cost model, not the cost section. What the section needs is a measured token profile per decision type — input tokens, output tokens, retry rate, and any tool-call or retrieval overhead — captured under the buyer’s real load and concurrency, with observed latency at the percentiles the product actually cares about. Prompt caching, batching and reasoning-token behaviour all move the number substantially, and none of them show up in a price-per-million-tokens table.
Where the load is served locally or on rented accelerators, throughput and hardware-fit measurement is a benchmarking discipline in its own right, and it belongs to LynxBenchAI’s measurement layer rather than to the procurement pack. The pack consumes those numbers; it does not define how they are produced.
Drift posture: a baseline is an artefact, not an intention
Most drift sections state an intention. The evidence requirement is narrower: a dated baseline run against a pinned model version string, stored so a later run can be compared like-for-like. Without it, the re-validation the pack promises has nothing to compare against, and the committee is being asked to approve a control that cannot be exercised. The re-run scope — which sections a given trigger reopens — is a separate design question we treat as its own step in re-validating an evaluation pack when the vendor ships a new version.
The pre-committee read-through
Before the pack leaves the team, read each section’s closing sentence and ask one question of it: what run does this trace to? If the answer is an artefact ID, a date and a sample set, the section is evidenced. If the answer is a paragraph of explanation, it is asserted, and it will be found. This takes twenty minutes on a five-section pack and is the cheapest defect-detection step available.
Two practical signals we look for during that read-through:
- A number with no denominator anywhere near it.
- A qualitative adjective doing load-bearing work — “robust”, “acceptable”, “comparable” — where a threshold should be.
Both are rewritable in place if the underlying run exists. If it does not, the section is a schedule problem, not an editing problem, and it is far cheaper to discover that internally than at the committee table.
Where this per-section discipline sits inside the wider question of what belongs in a procurement pack at all — provenance, structure, and the questions the pack exists to close — is developed in our work on LLM evaluation evidence packs. The governance framing that sits above both, including how evidence artefacts become part of an organisation’s defensible record, is covered on our AI governance and trust page.
One thing this checklist does not settle: how much evidence is enough for a section whose risk is genuinely hard to quantify — novel agentic workflows with open-ended action spaces, for instance. Sample counts and frequencies work when the failure surface can be enumerated. When it cannot, the honest move is to say so in the section itself and let the committee price the uncertainty, rather than manufacture a number that looks like the others.
Frequently Asked Questions
What does “what evidence each section needs to defend the model choice” mean in practice?
For Evidence Behind Each Section of an, it means deciding, before drafting, what run each section’s central claim will trace to. In practice that is a short specification per section — sample set, method, scorer, date — that the section is then assembled against rather than written first and justified later., all four. A named and versioned prompt distribution described by the buyer’s intents, per-intent sample counts rather than an aggregate only, the scoring rubric in full, and the identity of the scorer with a calibration or inter-rater note where relevant.
What makes a failure-mode entry defensible: reproducible triggering input, observed frequency, and its position against the buyer’s stated risk tolerance? Yes, and the third element is the one most often skipped. The entry needs the actual triggering input rather than a description of it, a frequency over a stated sample, and an explicit verdict — accepted with a named control, or blocking — against the tolerance the buyer already wrote down.
How is the cost-per-decision section evidenced from measured load rather than vendor list pricing? By measuring the token profile per decision type under the buyer’s own load and concurrency: input and output tokens, retry rate, tool-call overhead, and latency at the percentiles the product cares about. List pricing is one multiplier in that model, not the evidence.
What baseline evidence does the drift-posture section need so a later vendor version can be compared like-for-like? A dated baseline run against a pinned model version string, stored as a retrievable artefact with its prompt set and rubric attached. Without that artefact, the promised re-validation has nothing to compare against.
How do you tell a section that is evidenced from one that is merely asserted, before it reaches the committee? Read each section’s closing sentence and ask what run it traces to. An artefact ID with a date and sample set means evidenced; a paragraph of explanation means asserted. Numbers without denominators and load-bearing adjectives in place of thresholds are the two reliable tells.
Where does per-section evidence assembly stop and benchmark methodology (LynxBenchAI) begin? Methodology defines the measurement construct — task taxonomy, scoring design, reproducibility rules — and belongs to LynxBenchAI. The evidence pack consumes those measurements to defend one buying decision to one committee. We build the pack; the methodology layer defines how the numbers inside it are produced.
Three categories of model-choice evidence
Strong section LLM justification rests on architectural rationale, comparative benchmarks, and failure-mode analysis working together. The teams that do tend to ship the boring, correct version first.
