Write the task specification first, derive the eval set from real workflow traffic, fix the scoring rubric before any model is run, and set the decision threshold before results are visible. That order is the method. Reverse any step and you no longer have an evaluation — you have a written defence of a choice already made.
The naive version is familiar because almost everyone has done it. Someone is asked to “evaluate the models”, so they assemble forty prompts that feel representative, run three candidates, put two engineers in a room to read the outputs, and write up whichever model felt better. The outputs are real, the reading is honest, and the conclusion is still unusable — because nobody can say what the pass criterion was, so nobody can say whether it was met.
What does building a task-specific LLM evaluation for procurement mean in practice?
It means producing four artefacts, in sequence, before you look at a single score: a task specification, an eval set, a scoring rubric, and a documented decision threshold. Each one is a written object with a version number, not a shared understanding.
The sequence matters more than the sophistication of any individual step. An eval whose pass criterion is written after the scores are known is a rationalisation, and a procurement committee will read it as one. That is the single divergence point between an eval that survives review and one that gets sent back.
This article covers construction only. How the resulting evidence maps onto a committee’s specific questions — who signs, what they ask, what the memo looks like — is the job of our broader treatment of procurement-grade AI evaluation for infrastructure and SaaS buyers.
Step 1 — Write the task specification
A task specification is precise enough to score against when a competent stranger could label your eval set from it alone, without asking you a clarifying question. That is the test. Most specs fail it on the first read.
Concretely, it names:
- The input — what actually arrives, including its real length distribution. “Support tickets” is not an input; “customer email bodies, 40–2,400 tokens, 8% containing pasted log excerpts” is.
- The output contract — format, required fields, and what a valid refusal looks like. If downstream code parses the output, the schema belongs here.
- The correctness definition — what counts as right, and separately, what counts as an acceptable near-miss.
- The error asymmetry — which mistake is worse. A confidently wrong classification and an explicit “I don’t know” are not the same failure and must not score the same.
- The operating envelope — latency ceiling, concurrency, cost per request. A model that meets the quality bar outside the envelope has not passed.
The error-asymmetry line is the one teams skip and the one that changes results most often. In our experience reviewing eval designs, once a spec states that unflagged false confidence costs more than an abstention, the candidate ranking frequently reorders — because most default scoring treats both as a single point deducted.
Step 2 — Derive the eval set from traffic, not imagination
Where the eval set comes from determines what the eval can conclude. Cases invented by the team measure the team’s mental model of the workflow; cases sampled from logged production or pilot traffic measure the workflow.
Sample in named strata, and record the count per stratum:
| Stratum | Source | What it establishes |
|---|---|---|
| Representative | Random sample of real traffic, weighted to the production distribution | Baseline behaviour on the common case |
| Known edge cases | Traffic the current process already mishandles; tickets escalated by humans | Whether the candidate improves the actual pain |
| Adversarial / malformed | Truncated inputs, wrong-language, injection attempts, empty fields | Failure behaviour, not fluency |
| Regression | Cases a prior model got wrong and a fix was shipped for | That the new candidate does not reintroduce old defects |
On how many cases: enough that a difference you would act on is larger than the noise in your own labelling. The practical guide is not a formula but a stopping rule — keep adding cases until the per-stratum scores stop moving as you add them, and until every failure mode named in the task spec has at least a handful of cases exercising it. A hundred well-stratified cases with documented provenance beat a thousand scraped ones. Our sibling piece on what goes into a procurement-grade eval set develops the stratum design and labelling rules in more detail.
Record provenance per case — source system, date range, whether it was hand-edited, and whether it contains customer data requiring redaction. Provenance is what lets a reviewer trust the set without re-deriving it.
Step 3 — Fix the scoring rubric before any model runs
A rubric two reviewers agree on has three properties: discrete levels, an anchor example per level, and a written tie-break rule.
Continuous 1–10 scales do not produce agreement. Three or four levels with anchors do — for instance: fully correct and usable as-is, correct but requires editing, wrong but visibly wrong, wrong and plausible. That last level exists because it is the expensive one, and collapsing it into a generic “incorrect” bucket destroys the error-asymmetry work from Step 1.
Validate the rubric before you trust it. Have two reviewers independently score the same 20–30 cases, compute agreement, and revise the anchors wherever they disagreed. Disagreement is a rubric defect, not a reviewer defect. Then freeze it. If the rubric changes mid-run, every earlier score is invalid and the run restarts — which is why the freeze happens before the first model call, not after the first surprising result.
Where an LLM judge is used to scale scoring, it is scored against the human-labelled subset first, and its agreement rate with human reviewers is reported as part of the evidence. A judge whose agreement with your reviewers is unknown is an unmeasured instrument.
Step 4 — Set the decision threshold blind
The threshold is a sentence written and signed before results exist: we will adopt a candidate that scores ≥ X on the representative stratum, produces zero instances of the plausible-wrong failure class on the adversarial stratum, and stays within the latency envelope at target concurrency.
Three properties make it defensible:
- It names gates and trade-offs separately. Gates are pass/fail — safety, format compliance, latency ceiling. Trade-offs are weighted, and the business owner agrees the weights before the run.
- It is signed by whoever owns the workflow, not only by the engineers running the eval.
- It has a timestamp that precedes the first result. Commit it to the repository alongside the frozen rubric; the commit history is the proof.
If no candidate clears the threshold, the correct output is “no candidate clears the threshold” — not a relaxed threshold. That outcome is a genuine procurement result and is far cheaper than discovering it in production.
Quick-answer: the construction checklist
- Task spec written; a stranger could label from it unaided
- Output contract and error asymmetry stated explicitly
- Operating envelope (latency, concurrency, cost) declared
- Eval set sampled from real traffic in named strata, counts recorded
- Provenance and redaction status logged per case
- Rubric discrete, anchored, tie-break rule written
- Inter-rater agreement measured on a 20–30 case subset; anchors revised
- Judge model (if used) scored against the human subset
- Threshold written, gates separated from weighted trade-offs
- Threshold signed and timestamped before the first model run
- Task spec and eval set version-tagged for reuse
- Out-of-scope risks named and handed to monitoring
What belongs outside the eval?
This is where Build Task Specific LLM gets specific. Drift as the input distribution shifts, cost creep as prompts grow, degradation after a silent vendor model update, and rare-event behaviour too infrequent to sample — none of these are evaluable in a fixed offline run, and pretending otherwise is how an eval overclaims.
The honest move is to name them as residual risks and route them to a monitoring plan. That is the boundary between the eval and the harness that watches the deployment afterwards; we treat the operational side of it in our [Production AI Monitoring Harness], which instantiates the task spec, eval set, rubric and threshold built here against a buyer’s actual candidates.
Version the artefacts so the next candidate is cheap
The reuse economics are the reason this discipline pays for itself twice. Tag the task spec and eval set with a version, store them next to the scoring code, and log which version produced which result. When a new model lands three months later — and it will — you re-run rather than rebuild, and the comparison is valid because the basis is identical.
Treat the eval set as append-only. New strata and new regression cases can be added; existing cases are not quietly edited. If a case must change because the workflow changed, that is a new eval-set version, and prior results are labelled as belonging to the old one. Without this, “the model scored better than last time” is an uninterpretable statement.
One structural warning worth holding onto: as the harness gets more general and more candidates get added, the artefact starts to look like a benchmark. It is not one. An eval answers one workflow question for one committee; benchmark methodology — score construction, normalisation, cross-model comparability — is a separate discipline with separate obligations, and the two should not be conflated mid-project.
So the sharper question for a team about to start is not which metrics to use. It is this: can you write down, today, the sentence that says what result would make you walk away from your preferred vendor — and get the workflow owner to sign it before you run anything?
Frequently Asked Questions
What does building a task-specific LLM evaluation for procurement mean in practice?
Building a task-specific LLM evaluation for procurement requires defining precise benchmark criteria that mirror your organization’s actual vendor selection workflows. It means producing four versioned artefacts in a fixed order — a task specification, an eval set drawn from real traffic, a frozen scoring rubric, and a signed decision threshold — before any candidate model is scored. The artefacts, not the score, are what a committee reviews. Reversing the order turns the exercise into a justification of a choice already made.
How do we write a task specification precise enough to score against?
The working test is whether a competent stranger could label your eval cases from the spec alone without asking a clarifying question. That requires naming the real input distribution, the output contract, the correctness definition, the error asymmetry (which mistake costs more), and the latency/cost envelope. Most first drafts fail on error asymmetry.
Where does the eval set come from, and how many cases are enough?
It comes from logged production or pilot traffic, sampled in named strata — representative, known edge cases, adversarial or malformed inputs, and regression cases — with provenance recorded per case. Enough is when per-stratum scores stop moving as you add cases and every failure mode named in the spec has cases exercising it. Well-stratified hundreds beat scraped thousands.
How do we design a scoring rubric two reviewers will agree on?
Use three or four discrete levels with an anchor example for each, plus a written tie-break rule, then measure inter-rater agreement on a 20–30 case subset and revise the anchors wherever reviewers disagreed. Disagreement is a rubric defect, not a reviewer defect. Freeze the rubric before the first model call.
How do we set a pass/fail decision threshold before we see any model scores?
Write it as a sentence that separates hard gates (safety, format compliance, latency ceiling) from weighted trade-offs whose weights the workflow owner agreed to, then commit it to the repository with a timestamp preceding the first result. The commit history is the proof that the criterion predates the outcome.
What belongs outside the eval because only operational monitoring can catch it?
Input-distribution drift, cost creep as prompts grow, silent vendor model updates, and rare events too infrequent to sample cannot be established in a fixed offline run. Name them as residual risks and route them to a monitoring plan rather than implying the eval covers them.
How do we version the task spec and eval set so the next model candidate can be scored on the same basis?
Tag both with a version, store them beside the scoring code, and log which version produced which result. Keep the eval set append-only: add strata and regression cases, never silently edit existing ones, and bump the version when a workflow change forces a case to change so prior results stay attributable.
Procurement teams need evaluations, not benchmarks
Every vendor will show you MMLU scores; your job is to replace those with twenty examples from your actual contract documents and measure what breaks.
