A procurement-grade LLM eval set is not a pile of prompts. It is a set built in named strata — representative production traffic, the edge cases the workflow actually produces, and adversarial or malformed inputs — each with documented provenance, labelling rules, and a sampling decision someone can defend. The difference shows up the moment a reviewer asks a question the average score cannot answer.
That question is almost always the same one: how does this model behave on the hard five percent? An undifferentiated prompt pile answers it with a shrug. A stratified set answers it with a number, and names the cases the number came from.
What does a procurement-grade LLM eval set look like in practice?
Three strata, sized deliberately, documented separately.
| Stratum | What it contains | What it answers | Sizing signal |
|---|---|---|---|
| Representative | Examples sampled to match real workflow traffic distribution — same input lengths, formats, and topic mix | Does the model handle the ordinary case at an acceptable rate? | Large enough that the pass rate moves meaningfully when a handful of cases flip |
| Edge | Inputs the workflow genuinely produces but rarely — truncated documents, unusual locales, mixed-language input, boundary-length context | Where does behaviour degrade first, and how gracefully? | Enough per named edge category to distinguish a pattern from one bad draw |
| Adversarial / malformed | Prompt-injection attempts, contradictory instructions, empty or corrupt payloads, out-of-scope requests | Does the model fail safely, or fail confidently? | Small but exhaustive across the failure classes you can enumerate |
Each row is a separate reporting line. Blending them into one score is precisely the move that destroys the set’s usefulness, because the representative stratum is the largest and therefore dominates any average — hiding the strata that carry the deployment risk.
A stratified eval set makes the evidence pack answerable rather than merely presentable: reviewers see per-stratum pass rates instead of one blended number. That is the whole design goal.
Choosing representative examples without choosing convenient ones
The failure here is quiet. Someone exports the last thousand requests, filters out the ones that are awkward to label, and ships what remains as “representative.” What they have built is a set that matches the easy traffic distribution, and the eval will report a pass rate the deployment never reproduces.
Sampling from real traffic means recording the distribution first — input length percentiles, document types, request categories, time-of-day or seasonal skew if the workflow has one — then sampling to match it, and writing down where you deviated and why. When production traffic is unavailable (a new workflow, a regulated data set that cannot leave its environment), say so in the provenance note and describe the proxy. A documented proxy is defensible. An undocumented convenience sample is not.
In our experience the sampling conversation is where scoping actually happens: the moment a team tries to describe its own traffic distribution precisely, it discovers which parts of the workflow nobody has characterised. That discovery is worth more than the eval score.
Finding the edge cases instead of guessing them
Edge cases should be discovered, not brainstormed. Four sources reliably produce real ones:
- Support and incident logs. Anything that generated a ticket in the manual version of the workflow is a candidate.
- Distribution tails. The 99th-percentile input length, the rarest document type, the smallest non-empty payload.
- Schema violations already occurring upstream. Most production pipelines silently repair malformed input; the repair logic is a catalogue of edge cases.
- Domain expert interview. Ask the people who do the task manually what makes them slow down. They will name conditions no sampling script would surface.
Each edge case entering the set carries a one-line rationale naming its source. That line is what lets a reviewer six months later distinguish a case that came from a real incident from one an engineer invented on a Friday afternoon.
What adversarial inputs actually test
They do not test fluency, and they are not a security audit. Including a prompt-injection attempt in a procurement eval tests one narrow property: whether the model’s failure mode is refusal and containment or confident compliance. A model that produces a plausible answer to a corrupt input is more dangerous in a production workflow than one that returns an error, and no accuracy metric on the representative stratum will ever surface that difference.
Keep the adversarial stratum small and enumerated by failure class rather than by volume. Ten inputs covering ten distinct failure classes tell you more than two hundred variations of the same injection pattern. The output you record is behavioural, not scalar: what did the model do, and would the surrounding system have caught it?
We treat sampling and stratification discipline as borrowed from benchmark methodology — the LynxBenchAI work on how measurement conditions determine what a score means applies directly here — but the eval itself stays inside procurement scope. No leaderboard, no general-purpose model scoring, one workflow and one decision.
Documentation the reviewer expects attached
The set is only reusable if the next person can re-run it without reconstructing your reasoning. Minimum attachments:
- Provenance per stratum — where each example came from, when it was captured, and what was redacted or synthesised.
- Labelling rules — the written definition of a correct output for this task, plus who applied it and how disagreements were resolved.
- Sampling decisions — the target distribution, the achieved distribution, and every deliberate deviation.
- Exclusions — what was removed and why. Unexplained gaps read as cherry-picking whether or not they are.
- Version tag — a fixed identifier for the frozen set, so candidate model N+1 runs against the same inputs rather than a drifted copy.
Version discipline is what turns the set into an asset. Where the set is documented well enough to re-run, the marginal cost of evaluating the next candidate model drops to execution time rather than a fresh design cycle — and the comparison between candidates stays valid because the inputs did not move underneath it.
The construction checklist above is the input specification for the harness that executes it; our Production AI Monitoring Harness runs what this article tells you to build, and the same strata carry forward into post-deployment monitoring. For teams working through the surrounding procurement question — task specification, scoring rubric, threshold-before-results — the broader AI infrastructure and SaaS practice covers where the eval sits in the decision.
Frequently Asked Questions
What does a procurement-grade LLM eval set look like in practice? Three named strata — representative production examples, real edge cases, and adversarial or malformed inputs — each reported separately with its own pass rate. The set carries provenance, labelling rules, sampling decisions, and a version tag so it can be re-run against the next candidate.
How do we choose representative examples so the set reflects real workflow traffic rather than convenient prompts? Characterise the production distribution first — input length percentiles, document types, request categories — then sample to match it and record every deliberate deviation. Where real traffic is unavailable, document the proxy explicitly rather than filtering to whatever is easy to label.
Which edge cases belong in the set, and how do we find them without guessing? Discover them from support and incident logs, distribution tails, upstream schema-repair logic, and interviews with the people who perform the task manually. Each case entering the set carries a one-line rationale naming its source.
What adversarial or malformed inputs should be included, and what are we actually testing when we include them? Prompt-injection attempts, contradictory instructions, and empty or corrupt payloads — enumerated by failure class rather than by volume. The property under test is whether the model contains the failure or answers confidently anyway, which no representative-stratum accuracy number will reveal.
How large does each stratum need to be before a reviewer can trust the per-stratum result? Large enough that flipping a handful of cases moves the reported rate meaningfully; below that, the stratum reports noise. Adversarial strata are the exception — small and exhaustive across enumerated failure classes beats large and repetitive.
What provenance, labelling and sampling documentation does a reviewer expect attached to the set? Per-stratum provenance, the written definition of a correct output and who applied it, the target versus achieved distribution with deviations named, and an explicit list of exclusions with reasons.
How do we version the eval set so it can be re-run against the next model candidate without redesign? Freeze the set under a fixed version identifier and treat any change to the inputs as a new version, not an edit. Candidates evaluated against different versions are not comparable, and saying so in the evidence pack is cheaper than discovering it during review.
Once the strata exist and are documented, the open question stops being what goes in the set and becomes a harder one: how much divergence between the representative stratum and next quarter’s live traffic is tolerable before the set stops being evidence at all?
Acting on Procurement Grade LLM Eval
Treat Procurement Grade LLM Eval as an engineering problem with a measurable answer, not a positioning question. The teams that do tend to ship the boring, correct version first.
