An evidence pack is not a results dump with a cover page. It is a fixed-shape document built around the questions a reviewer will ask, and its completeness — not the eval’s sophistication — is what sets time-to-approval.
We see the same last-mile failure regularly. A team does the hard part well: they scope the eval to a real workflow, build a stratified eval set from production traffic, agree a rubric before anyone sees a score. Then they ship the notebook. Accuracy tables, three transcripts that illustrate the point nicely, and the vendor’s datasheet stapled to the back. The committee reads it, asks how the hard cases were selected, and the team goes back for another eval cycle — not because the eval was wrong, but because the document could not answer a question the eval already had the answer to.
This article is about the deliverable, not the method. If you are still designing the run, the sequencing question is covered in our checklist for building a task-specific LLM evaluation.
What does a procurement LLM-eval evidence pack contain in practice?
Six sections, in this order. The order is not cosmetic — it is the order in which a reviewer’s trust is either established or lost.
| # | Section | What it states | Reviewer question it absorbs |
|---|---|---|---|
| 1 | Workflow definition and decision scope | The specific workflow evaluated, the inputs it receives, the decision the pack supports, and what is explicitly out of scope | “What exactly did you test, and is that what we are buying it for?” |
| 2 | Dataset provenance and sampling rationale | Where examples came from, the date range, the strata and why each exists, how hard cases were selected, what was excluded and why | “How were the hard cases selected?” |
| 3 | Scoring rubric and its owner | Each dimension, its pass/fail or weighted status, the threshold in workflow terms, who wrote the rubric and who approved it before the run | “Who signed the rubric?” |
| 4 | Per-candidate results with failure taxonomy | Results per candidate against the rubric, plus every failure sorted into named classes with counts and representative examples | “What does it get wrong, and does that matter here?” |
| 5 | Validity conditions | The load, context length, prompt version, model version and data distribution under which these results hold — and where they stop holding | “What happens at 3× the tested load?” |
| 6 | Residual risks and monitoring handoff | Risks the eval could not close, each with a named monitoring signal, threshold and owner | “What are we accepting, and who watches it?” |
A pack built around anticipated reviewer questions holds under interrogation; a results dump built around what the team measured sends the team back for another eval cycle. That is the whole design principle, and it is why section 5 exists at all — nobody asks for validity conditions until the deployment has already drifted past them.
A worked example: document-classification model choice
Concretely, for a mid-market SaaS platform choosing between two hosted LLMs to classify inbound support tickets into routing categories.
Section 1 names the workflow as “inbound ticket → one of 14 routing categories, at ingest, with a human escalation path for low-confidence outputs.” It states what is out of scope: summarisation, reply drafting, and any multilingual traffic, because the eval set was English-only. Naming the exclusion here is what stops a reviewer discovering it in section 4 and losing confidence in everything above it.
Section 2 documents 900 tickets drawn from six weeks of production traffic, split into three named strata — 600 representative examples reproducing live category distribution, 200 edge cases the workflow genuinely produces (multi-issue tickets, tickets with pasted logs, empty bodies with meaningful subject lines), and 100 adversarial inputs. Crucially, it states the selection rule for the edge stratum, not just its size. “Tickets that the current rules engine routed twice” is auditable; “the hard ones” is not. Construction of the set itself is a separate discipline, laid out in what a procurement-grade LLM eval set looks like.
Section 3 carries the rubric table with its signatures. Two pass/fail gates — routing accuracy on the representative stratum, and a hard ceiling on confidently-wrong outputs in the adversarial stratum — plus three weighted dimensions (p95 latency at ingest concurrency, cost per thousand tickets, and behaviour on empty or malformed input). The support operations lead signed the thresholds; the platform engineer signed the measurement method. Both dates precede the first run.
Section 4 is where discipline matters most, because the temptation is to rank. It does not. Each candidate is reported against the rubric independently, and the failures are sorted into a taxonomy the operations lead can reason about: wrong-but-adjacent category (cheap, escalation catches it), wrong-and-distant category (expensive, misroutes to a team with no context), refusal, and truncated output. A model with more total errors concentrated in the adjacent class can be the correct choice. A leaderboard cannot express that; a taxonomy can. How the numbers get framed for a non-technical reader is covered in scoring eval results for a non-technical procurement reviewer.
Section 5 states the run conditions plainly: tested at 40 tickets per minute against a 2,000-token context ceiling, on a named model version and a versioned prompt template committed to the repository. It also names the boundary — the pack makes no claim about behaviour above roughly twice the tested ingest rate, or about the Spanish-language queue scheduled for Q3. Two sentences here save a review cycle later.
Section 6 hands off. Three residual risks, each with a live signal rather than an intention: silent category drift as ticket vocabulary changes (weekly distribution comparison against the eval baseline, ops lead owns it), vendor-side model updates invalidating the run (version pinning plus a re-run trigger, platform engineer owns it), and confidently-wrong outputs on the adversarial class (sampled human review at a fixed rate). This section is the seam between evaluation and operations, and it is the point where our Production AI Monitoring Harness engagement takes over from the eval work.
Which sections does a committee actually read first?
Sections 1, 4 and 6 — in our experience, roughly in that order, and often nothing else on first pass. Section 1 tells the reviewer whether the pack answers their question at all. Section 4 tells them what they are accepting. Section 6 tells them who owns the risk after approval. Sections 2, 3 and 5 are read when challenged — which means they are the sections that determine whether the pack survives a second meeting, and the sections teams most often thin out because nobody read them last time.
The practical consequence is a layout rule rather than a content rule: put the sections a committee reads first at the front, and make the audit sections complete rather than prominent. Depth without navigability reads as evasion.
Making the pack reusable across candidates
The reason to fix the shape is economic. Sections 1, 2, 3 and 5 describe the eval, not the model. Section 4 and the risk register in section 6 are the only parts that change when a third candidate arrives six months later — which is what makes the marginal evaluation cheap and what keeps the second decision comparable to the first. A pack whose workflow definition and rubric are re-derived each time produces two documents that cannot be read against each other.
One boundary worth holding explicitly: reusing the shape is not the same as building a benchmark. The moment the pack starts normalising scores across models for publication, it has changed audience and taken on methodology claims it was never built to defend — a distinction we develop in what a procurement LLM eval is not. Buyer-side context for how this fits a platform team’s wider infrastructure decisions sits on our AI infrastructure and SaaS page.
Frequently Asked Questions
What does a procurement LLM-eval evidence pack contain in practice? Six sections: workflow definition and decision scope, dataset provenance and sampling rationale, the scoring rubric with its signatories, per-candidate results with a failure taxonomy, validity conditions, and residual risks with their monitoring handoff. Each section exists to absorb a specific reviewer question rather than to display work.
Which sections does an approval committee actually read, and in what order? Typically the workflow definition, the results-and-failures section, and the residual-risk handoff — in that order, on first pass. Provenance, rubric and validity conditions get read when the pack is challenged, which is why thinning them out is the most expensive shortcut available.
How should dataset provenance, sampling and the scoring rubric be documented so a reviewer can audit them? State the source, date range and named strata with the selection rule for each, not just counts, and record exclusions. For the rubric, record each dimension’s threshold in workflow terms plus who wrote and approved it, with dates preceding the first run.
How are per-candidate results and the failure taxonomy presented without turning the pack into a leaderboard? Report each candidate independently against the fixed rubric rather than ranked against each other, and sort failures into named classes with counts and examples. A model with more total errors concentrated in a cheap failure class can legitimately win, which a single ranking cannot express.
What validity conditions and residual risks belong in the pack, and how do they hand off to operational monitoring? Validity conditions name the load, context length, prompt version, model version and data distribution the results hold under — and where they stop. Each residual risk is paired with a specific monitoring signal, a threshold and a named owner, which is the seam into an operational monitoring harness.
How do we structure the pack so only the results sections change when the next model candidate is evaluated? Keep workflow definition, provenance, rubric and validity conditions as descriptions of the eval rather than of any model, so a new candidate only rewrites the results section and its risk entries. That also keeps the second decision comparable to the first.
What does a reviewer question look like when the pack is incomplete, and what does that cost in review cycles? It looks like “how were the hard cases selected?” or “who signed the rubric?” — questions the eval usually has answers to but the document does not surface. The cost is a full review cycle, and sometimes a re-run, because an unanswered audit question is indistinguishable from a missing measurement.
If you want a single test for whether a pack is finished, it is this: hand it to someone who was not in the eval and ask them to find the condition under which the recommendation stops being true. If they cannot, which section is missing?
Building your evidence strategy
Three decisions shape every successful evidence pack: which artifacts prove capability, how evaluation traces map to requirements, and whether your documentation survives vendor challenge.
