An evaluation pack is dated the day the committee signs it off. The model behind it is not. Somewhere between that sign-off and the next audit question, the vendor pushes a point release, and the approved evidence quietly starts describing a system that no longer exists. The pack is still on the shared drive, still signed, still nominally valid — and it now covers a model version nobody is running.
The fix is not a bigger pack. It is a maintenance layer: a written list of vendor-side events that trigger re-validation, a defined re-run scope for each trigger, and a version-delta record that keeps successive model versions comparable on the buyer’s own prompt distribution. That is what turns an approval into something that survives a release cadence.
A procurement-grade evaluation pack should be treated as a versioned artefact with an explicit re-validation trigger list, not as a one-off procurement deliverable filed after sign-off.
What does keeping the evaluation pack current actually involve?
Three things, and only three.
First, the pack states which model version each piece of evidence was gathered against — model identifier, endpoint, and the default parameters in force at the time. Without that, no later comparison is possible, because there is no baseline to compare against.
Second, the pack carries a trigger list: the vendor-side events that oblige a re-run, and the ones that are logged without one. This is a decision written in advance, when nobody is under time pressure. Deciding it during a release week produces whatever answer is cheapest that week.
Third, each trigger maps to a re-run scope — which pack sections get re-evidenced, and which stand. A version bump that changes tokenisation defaults touches cost-per-decision and possibly latency; it does not automatically invalidate the governance sign-off trail. Scoping the re-run is what keeps the loop affordable enough that it actually happens.
In our experience the third item is where packs fail. Teams accept the principle of re-validation, discover that “re-validate” has been left to mean “re-run everything”, and quietly stop doing it after the first cycle.
Which vendor events trigger a re-run, and which are just logged
The trigger list below is a starting template. It should be edited to match the vendor’s actual release behaviour and the risk profile of the workload, then attached to the pack as a maintained section.
| Vendor-side event | Re-validation scope | Why this scope |
|---|---|---|
| Major model version bump (e.g. a new generation of the same family) | Full re-run: task accuracy, failure-mode catalogue, cost-per-decision | Behaviour, tokenisation and pricing can all move together; the old baseline is not portable |
| Minor / point release on the same family | Task accuracy on the buyer’s prompt distribution + the highest-risk failure modes | Accuracy drift is the common outcome; governance and integration evidence usually stands |
| Default-parameter change (temperature, max tokens, system-prompt handling) | Reproducibility checks + any evidence that depended on deterministic settings | Same weights, different behaviour; only the settings-dependent sections are affected |
| Endpoint or API surface change | Integration and latency evidence; re-confirm the harness still measures what it claims | A harness that silently degrades produces evidence about the harness, not the model |
| Safety-filter or moderation update | Failure-mode catalogue, especially refusals and false-positive blocks on legitimate prompts | Refusal behaviour is a failure mode, and it moves independently of accuracy |
| Pricing change | Cost-per-decision only | Nothing about model behaviour changed; the economics section did |
| Deprecation notice for the approved version | Full re-run against the named successor, before the deprecation date | The approval’s subject is being withdrawn; this is a deadline, not a trigger |
| Documentation or marketing change with no behavioural claim | Log, no re-run | Recording it preserves the audit trail cheaply |
Two rules make this list work in practice. Every logged-but-not-re-run event still gets a dated entry, because a gap in the log is indistinguishable from inattention six months later. And any event whose scope is genuinely unclear gets escalated to the pack owner rather than being resolved by default to “log it”.
Recording the version delta so the committee can read it
A re-run produces a second set of numbers. On its own, that is not useful — the committee does not want two tables, it wants the difference between them and a judgement about whether the difference matters.
A version-delta record does that job. It holds, for each re-validated section: the previous model version and its result, the new version and its result, the delta, and a stated verdict — within tolerance, degraded, improved, or requires a decision. The tolerance is the important field, because it has to be set before the re-run rather than negotiated afterwards against the number that came back.
Two properties are non-negotiable if the delta is going to mean anything. The prompt distribution must be the same one used for the original approval, held stable across versions; and the scoring rubric must be unchanged. Change either and the delta measures the measurement, not the model. Keeping a frozen, versioned test set alongside the pack — the prompts, the expected-behaviour annotations, the harness configuration, ideally under the same version control as any evaluation code — is the cheapest way to guarantee this. A behavioural delta between two model versions is only interpretable when the prompt distribution and scoring rubric are held constant across both runs.
This is also where vendor release notes stop being sufficient. Release notes describe changes the vendor considers notable across its whole customer base. They cannot tell one buyer whether a specific extraction task on a specific document format got worse, and that is precisely the question the committee will ask.
Detecting the changes the vendor does not announce
Not every behavioural change arrives with a version number. Endpoints get re-pointed, defaults get adjusted, safety filters get tuned. The trigger list handles announced events; silent changes need a standing probe.
The mechanism is unglamorous: a small, fixed canary set of prompts — a few dozen is usually enough — run on a schedule against the production endpoint, with outputs diffed against the recorded baseline. Scheduling it in CI (a cron-triggered job in GitHub Actions or GitLab CI is the usual arrangement) and logging results to something like MLflow gives you a dated series rather than a one-off impression. A sustained divergence, not a single sample, is the signal worth acting on; sampling noise from a non-deterministic endpoint will otherwise generate false alarms until people stop reading the alerts.
Canary drift raises a trigger. It does not by itself constitute evidence for the committee — that comes from the scoped re-run it triggers.
What the approval status is between release and re-validation
This is the question that gets skipped, and it causes real trouble during an audit.
The honest position is that an approval is granted against a specific model version, and a vendor release does not revoke it — but it does open a stated interval during which the approved evidence and the running system diverge. Naming that interval explicitly is stronger than pretending it does not exist. A practical posture: the approval stands, the divergence is recorded with a date, a re-validation owner and a target completion date are named, and any pack section not yet re-evidenced is marked as covering the prior version.
That way the answer to “which model version does the approved evidence cover?” is always available. When it is not available, the approval is nominally intact but evidentially stale, and a challenge to the model choice lands with nothing behind it. Recovering from that position typically means a second approval round — the cost the whole loop exists to avoid.
The re-validation loop is one component of a broader governance posture; we set out how it fits alongside model risk documentation and sign-off trails under AI governance and trust. The pack it maintains — its section structure and the evidence each section carries — is developed in what sits inside an LLM evaluation evidence pack beyond the leaderboard.
Where the buyer’s loop ends and benchmark methodology begins
There is a clean division of labour here, and blurring it produces packs that are neither good methodology nor good procurement evidence.
Benchmark methodology owns the measurement construct: what a task suite should contain, how scoring is defined, what makes a run reproducible, how a result may legitimately be read. That is LynxBenchAI’s territory, and the methodology it publishes is what a buyer re-applies on each cycle — see /lynxbench-ai for how measurement is designed and how far a result can be read.
The re-validation loop described here owns something narrower: when to re-apply that methodology, to which sections, against which baseline, and how the resulting delta is recorded for a committee that has to defend a buying decision. The methodology tells you how to measure. The loop tells you when your evidence expired.
The open question is cadence. A trigger-driven loop is reactive by design, and for most workloads that is the right economics. Where a workload is regulated or the vendor’s release behaviour is opaque, a fixed re-evidencing cadence may be the stronger commitment — and deciding which of the two a given pack owes is a judgement we would rather see made explicitly at sign-off than discovered during an audit.
Frequently Asked Questions
What does keeping the evaluation pack current when the model vendor pushes a new version mean in practice?
Model version updates trigger a mandatory re-validation cycle that encompasses performance baselines, safety checks, and compliance benchmarks. It means three maintained things: every piece of evidence records the model version, endpoint and default parameters it was gathered against; a written trigger list says which vendor events oblige a re-run; and each trigger maps to a defined re-run scope. Without the scope definition, “re-validate” collapses into “re-run everything”, and the loop stops happening after one cycle.
Which vendor-side events should trigger re-validation, and which can be logged without a re-run?
Major version bumps, point releases, default-parameter changes, endpoint changes, safety-filter updates and deprecation notices all trigger a re-run at differing scopes. Pricing changes touch only the cost-per-decision section. Documentation changes with no behavioural claim are logged with a date and no re-run — but they are still logged, because a gap in the log looks like inattention later.
How do you scope a re-run so only the affected pack sections are re-evidenced rather than the whole task suite?
By mapping each trigger to the sections whose evidence could plausibly have moved. A pricing change affects economics, not behaviour. A default-parameter change affects settings-dependent and reproducibility evidence. A major version bump affects everything, because behaviour, tokenisation and pricing can move together. The mapping is fixed in advance, not negotiated during a release week.
How is the version delta recorded so the committee can compare the new version against the approved baseline?
In a version-delta record holding, per section, the prior version and result, the new version and result, the delta, and a verdict against a tolerance set before the re-run. It is only interpretable if the prompt distribution and scoring rubric are held constant across both runs, which is why the test set is frozen and versioned alongside the pack.
What happens to the approval status of a model choice between a vendor release and the completed re-validation?
The approval stands — it was granted against a specific version — but the divergence is recorded with a date, a named re-validation owner and a target completion date, and any not-yet-re-evidenced section is marked as covering the prior version. That keeps “which version does the approved evidence cover?” answerable at all times.
How do silent endpoint or default-parameter changes get detected when the vendor does not publish a version bump?
Through a standing canary set — a few dozen fixed prompts run on a schedule against the production endpoint, outputs diffed against the recorded baseline, results logged so you have a dated series rather than an impression. Act on sustained divergence, not single samples, or endpoint non-determinism will generate false alarms until the alerts get ignored.
Where does the buyer’s re-validation loop stop and LynxBenchAI’s benchmark methodology begin?
Benchmark methodology owns the measurement construct — task suites, scoring, reproducibility, and how far a result can legitimately be read. The re-validation loop owns when that methodology is re-applied, to which pack sections, against which baseline, and how the delta is recorded for a committee. Methodology tells you how to measure; the loop tells you when your evidence expired.
Model version bumps demand selective re-validation
Not every vendor release requires a full re-run; isolate the deltas, re-test only the affected eval slices, and document which claims still hold. Revisit it when your workload shifts.
