Two questions get asked in every LLM procurement review, and they sound similar enough that teams answer them with one document. “Is this score right?” and “is this choice defensible for us?” are not the same question, and they do not have the same owner. The first belongs to benchmark methodology. The second belongs to the procurement evidence pack. Collapsing them into a single artefact produces something that neither survives methodological scrutiny nor answers what the committee came to approve.
We see this most often as a methodology write-up bolted onto a vendor recommendation — twelve pages explaining how scoring works, three pages saying which model to buy. The committee reads the twelve pages, finds a design choice it dislikes, and defers the decision. The buying question never gets asked.
What is the boundary between an evidence pack and benchmark methodology?
Benchmark methodology defines the measurement construct: what capability is being measured, how tasks are sampled into a taxonomy, how outputs are scored, and what has to be recorded for a third party to reproduce the run. It is a standing asset. It does not know who you are, and it should not — a methodology that changes shape per buyer is not a methodology.
A procurement evidence pack applies a methodology to one buyer’s situation: their prompt distribution, their load profile, their risk tolerance, their vendor-update exposure. It is dated to a decision and it names a signer. It exists to close a committee’s questions, not to establish how scoring works.
The divergence point is the moment someone asks whether the score itself is trustworthy. That question is methodological, and the evidence pack should route it rather than answer it. In our work this is the boundary we hold: TechnoLynx builds the procurement evidence pack; LynxBenchAI owns the benchmark methodology layer the pack applies.
Which layer owns which question
| Question | Layer | Owner |
|---|---|---|
| What construct is being measured, and is it the right one? | Methodology | LynxBenchAI |
| How are tasks sampled into a taxonomy? | Methodology | LynxBenchAI |
| How is an output scored — rubric, grader, tie-breaks? | Methodology | LynxBenchAI |
| Can a third party reproduce this run? | Methodology | LynxBenchAI |
| What is accuracy on our prompt distribution? | Evidence pack | TechnoLynx |
| What does a decision cost at our load profile? | Evidence pack | TechnoLynx |
| Which failure modes exceed our risk tolerance? | Evidence pack | TechnoLynx |
| Who signed off, on what date, against which model version? | Evidence pack | TechnoLynx |
The table is also a drafting test. If a section of your pack answers a row from the top half, you have written methodology inside a procurement artefact, and a reviewer will treat the whole document as a methodology proposal.
Citing a methodology without republishing it
The pack cites the methodology the way a lab report cites a protocol: by name, by version, by the specific configuration used, with a link to the standing document. What goes in the pack is the application — which subset of the task taxonomy was run, how many prompts were drawn from the buyer’s own corpus, which scoring rubric was applied unchanged, and any deviation from the standard configuration with a reason.
Restating the methodology in the pack’s own words is the failure to avoid. Two descriptions of the same protocol drift apart the moment either one is edited, and the pack then contains a claim about scoring that the methodology owner never made. Reference, do not paraphrase.
What happens when the vendor ships a new version
This split earns its keep on the second procurement cycle. When a model vendor releases a new version, the methodology layer is unchanged — construct, taxonomy and scoring are properties of the measurement, not of the model. Only the buyer-specific application is re-run: the same prompt distribution, the same rubric, a new set of results and a new sign-off date.
Buyers who blurred the layers rebuild both. They re-argue the benchmark design with every model update, and they carry methodology disputes into a procurement conversation that should be about workload fit. The re-run scope question — which triggers force which sections to be regenerated — is developed further in our work on re-validating an evaluation pack when the vendor ships a new version.
What we decline to put in the pack
Some things belong outside the evidence pack by design, and saying so early is faster than being asked later:
- A defence of the benchmark construct itself. If the committee doubts the measurement, that is a methodology conversation with the methodology owner, held before the pack is commissioned rather than during approval.
- Cross-methodology score comparisons. Numbers produced under different constructs are not comparable, and a pack that lines them up in one table is asserting something it cannot support.
- A general model ranking. The pack answers one workload question. It is not a leaderboard, and it does not claim a model is better in general.
- Reproducibility guarantees for a third party’s harness. The pack states what was run and how; whether an outside party can replicate the harness is a property of the methodology’s reproducibility design.
Each of these has a destination. Naming the destination — rather than absorbing the question — is what keeps the pack short enough to be read and specific enough to be approved.
Explaining the split to a committee
Committees generally accept the boundary when it is framed in terms of accountability rather than scope. The line we use: the methodology is a standing, published construct that anyone can inspect and challenge on its own terms; the pack is our application of it to your workload, and we own every number in it. That framing tells the committee where to send a methodology objection without implying the pack is unwilling to answer for its own contents.
For a regulated buyer the boundary is not a convenience, it is a provenance requirement. Every score in the pack has to trace to a named methodology version, a named configuration, and a dated run — which is only possible when the methodology is a separate, versioned artefact rather than prose embedded in a procurement document. Where the evidentiary standard rises beyond that, our note on what changes between regulated and unregulated procurement evidence works through the differences.
The wider structure this sits inside — what an evidence pack contains, how it is organised, and how it is defended — is covered across our AI governance and trust practice.
The remaining uncertainty is not where the line sits but who is trusted to hold it. A methodology owned by the same party that recommends the purchase invites exactly the scrutiny the split was meant to avoid, and that is a governance question rather than a technical one.
Frequently Asked Questions
What does the boundary between the procurement evidence pack and benchmark methodology (LynxBenchAI) mean in practice?
On Evidence Pack vs Benchmark, the evidence points one way. When applied to Evidence Pack vs Benchmark Methodology, in real deployments it is a division by question type. Anything about how a score is constructed — measurement construct, task taxonomy, scoring rubric, reproducibility — belongs to the methodology layer. Anything about whether a specific model is the right choice for a specific workload at a specific risk tolerance belongs to the evidence pack.
How does the evidence pack cite a methodology without republishing or restating it? By naming the methodology, its version, and the exact configuration used, with a link to the standing document, then documenting only the application: which taxonomy subset was run, how many prompts came from the buyer’s corpus, and any deviation with a reason. Paraphrasing the methodology inside the pack creates two descriptions that drift apart.
Who owns each layer over time, and what happens to each when the model vendor ships a new version? The methodology is a standing asset maintained by its owner and is unchanged by a vendor release, because construct and scoring are properties of the measurement rather than the model. The evidence pack is dated to a decision, so a new model version triggers a re-run of the buyer-specific application only — same prompts, same rubric, new results, new sign-off date.
What should TechnoLynx decline to put in a procurement evidence pack, and where does the buyer get it instead? We decline to defend the benchmark construct itself, to compare scores produced under different methodologies, to publish a general model ranking, or to guarantee reproducibility of someone else’s harness. Those are methodology-layer questions and belong with the methodology owner, addressed before the pack is commissioned rather than during committee approval.
Who owns methodology design
Evidence packs document performance on fixed test cases; benchmark methodology—sampling strategy, statistical power, eval harness design—remains your internal responsibility. Revisit it when your workload shifts.
