LLM release readiness is defensible when the sign-off is an engineering record, not the outcome of a meeting. We break that record into five parts. The exact model version has an inventory entry with an owner and a risk tier. The metrics and pass marks were agreed before the evaluation ran. A named individual or committee approved the result. Any conditions are recorded against that version’s identifiers. All of it sits in a tamper-evident audit trail that lets others reconstruct who authorized what. That record is what an investigation needs when an incident, a complaint or an auditor asks which version was live, what data shaped it and who approved its release.
We treat that record as part of production AI reliability, an engineering discipline distinct from model accuracy. A production model can regress in ways unit tests do not catch, and release-readiness reviews are among the artefacts built to catch those regressions before customers do.
One caveat matters from the start. The OCC’s SR 26-2 (Bulletin 2026-13, April 2026) states that generative and agentic AI models fall outside its scope, so, as Govern365 notes, for LLMs that bank model validation no longer reaches, the internal evaluation record is the main independent check that exists.
Five parts of an LLM release readiness decision
| Part | What it fixes on record | Where the expectation comes from |
|---|---|---|
| Inventory entry | Exact model version or checkpoint, hosting mode, systems served, owner | Model registry practice; OSFI E-23 inventory fields |
| Pre-set thresholds | Metric, pass mark, approver and approval date, recorded before the run starts | EU AI Act Article 9(8) |
| Named approver | The individual or committee with authority to approve use of the model | OSFI E-23; PRA SS1/23 |
| Conditions | Provisional approval pending resolution of issues, or approval with compensating mitigants | OCC handbook; OSFI E-23 |
| Audit trail | Who, what and when in an immutable, time-stamped record, plus why under Annex 11 | 21 CFR Part 11, EU GMP Annex 11 |
Enable the audit trail from day one. Retrofitting an audit trail leaves gaps in the historical record that regulators will question.
How do we register a self-hosted model that skipped formal intake?
Create the inventory entry before sign-off goes any further, even if the model has served traffic as a pilot for months. The release decision needs a record to attach to, and a model without one has no owner, no tier and no approved scope to check the evaluation against. TechnoLynx’s AI Governance and Trust work builds the GenAI model-risk inventory and approval workflow as engineering evidence, so the release-readiness sign-off has something concrete to approve.
OSFI’s Guideline E-23 lists the model version, the date it went into production, the reviewer, the approver, dependencies, data sources, approved uses, limitations, the most recent review date, monitoring status and the next review date. The OCC’s model risk handbook adds the model owner and users by title or group, plus the approval date or a timeline for approval.
For an LLM, the version field must hold the exact version string or checkpoint identifier, not “GPT-4 based”. The entry also records the hosting mode (vendor API, self-hosted, or a fine-tune of a base model) and the systems the model serves, because each hosting mode puts version stability in different hands. Record the date the model started serving that system, not the provider’s release date, since the serving date is the one an incident timeline needs.
The risk tier should come from clear, measurable criteria, not instinct. E-23 expects quantitative factors such as potential operational, security or financial impact, and qualitative ones such as business use, complexity or autonomy, reliability of data inputs, customer impact and regulatory risk. A temporary rating at intake is acceptable as long as it is confirmed at review or approval.
Who updates the entry when the underlying model is swapped? The model owner. Under both OCC and PRA expectations, the owner is accountable for promptly identifying new or changed models and for keeping the inventory record accurate and current. Assigning the duty is not enough; updating the registry must be a required step of an existing control. For self-hosted models the trigger should be the deployment pipeline: a new model version reaching production is a gated step, and the deployment cannot complete until a registry entry exists.
The failure this prevents is a common one. An engineering team upgrades a self-hosted model as part of a routine dependency update, nothing in the process marks it as different from any other version bump, and six months later nobody can say with confidence which model was live on the date in question. A swap that materially changes the model’s use, data or infrastructure should also trigger a re-rating of the tier. The entry is only the attachment point for the sign-off. It does not replace the evaluation evidence the sign-off rests on.
What evidence must exist before anyone quotes an accuracy number?
At minimum, the number must trace to a specific run on a specific version, scored against a metric and pass mark approved before the run started.
A release gate can stop a feature from becoming externally visible until a signed evaluation report exists, and that same signed artefact should be the only allowed source for any accuracy figure in sales material. TechnoLynx builds that gate as part of the Production AI Monitoring Harness. TechnoLynx starts by mapping your ship pipeline and sales approval flow to find where the check belongs, then implements a pipeline check that blocks release when no signed report exists. Sales receives a sign-off summary from a template delivered with the gate, rather than raw harness output. The block has to be technical, because a policy-only gate can be bypassed under organisational pressure. A passing gate vouches for the tested acceptance criteria and nothing beyond them.
The EU AI Act’s Article 9(8) requires testing “against prior defined metrics and probabilistic thresholds” before a high-risk system goes to market. As Govern365 puts it, a threshold chosen after the score arrives proves nothing, so the metric, the pass mark, the approver and the approval date go on record before the run. NIST’s AI RMF makes the same point from the other direction: human judgement, not the evaluation tooling, should decide the metrics and their precise threshold values.
Those choices are not neutral. A 2026 paper on the AI Act’s accuracy requirement shows that choosing metrics, balancing them, measuring them on representative data and setting the acceptance threshold each build in assumptions about which errors are acceptable, and it argues that providers must document the reasoning behind the threshold. NIST AI 800-3 adds a precision point. State whether the number is benchmark accuracy on the questions asked or generalised accuracy over the wider population those questions stand for, because naming the measure makes the result and its uncertainty clearer to state.
The run also has to be repeatable. Govern365 reports that Thinking Machines Lab sampled 1,000 completions at temperature zero from one open-weight model and got 80 unique completions, because inference kernels are not batch-invariant. An evaluation record therefore has to capture the provider, endpoint, model version, decoding parameters, batch and concurrency settings, and date. Where the AI Act applies, the accuracy declared in the instructions for use must trace back to a specific run on the version that was tested and validated.
The risk is not only regulatory. In April 2025 the FTC alleged that Workado’s claim that its AI content detector was 98 percent accurate was false, misleading or unsubstantiated, after independent testing put its accuracy on general-purpose content at 53 percent.
After a retrain, should the new model replace the current one now, with conditions, or not yet?
There are three possible decisions, and calibration results map onto them through the acceptance criteria agreed before validation. Approve outright if every decile or segment meets its criteria and none is borderline. Approve conditionally if a shortfall is isolated, low-impact and contained by controls you can actually enforce. Reject or hold if several segments fail, or if the shortfall cannot be contained.
These categories follow model risk guidance. The OCC handbook says that where validation finds significant deficiencies, use should not be allowed, or should be allowed only under very tight constraints until the issues are resolved. If the deficiencies are too severe to fix within the model’s framework, the model should be rejected. OSFI’s E-23 allows a model to be approved despite identified weaknesses, provided compensating mitigants are in place or the stakeholders justify using a model with known limitations.
| Calibration outcome | Decision | What goes on record |
|---|---|---|
| All deciles meet criteria, none borderline | Approve | Run identifier, model version, approver, date |
| One borderline decile, small exposed population, enforceable controls | Conditional approval | The condition, controls, monitoring frequency, revisit trigger and an end date, since exceptions should be temporary |
| Several deciles fail, or the borderline one covers a large population | Reject or hold | Nature of the issues and remediation status, tracked in the inventory |
Which compensating controls justify going live despite a borderline decile? The PRA gives examples: alternative models, heightened review and challenge, more frequent monitoring, and recalibration or redevelopment once defined triggers are breached. It also allows restrictions on use, such as permitting the model only under strict controls or banning it for specific purposes. For a borderline decile, a practical version is to send that band to the current model or to manual review while the rest of the traffic moves to the new model.
What does a conditional approval commit you to? The PRA expects exceptions to be temporary and reported to senior management, so the condition needs an end date or an exit test. E-23 names re-rating triggers such as a drop in performance or a material change in the model’s use, data or infrastructure. Without a named trigger to revisit it, a conditional approval quietly becomes permanent. Check the size of the borderline decile too. If it covers a large share of the exposed population, you are looking at a calibration problem, not an edge case.
When a checkpoint update flips decisions, how do you tell normal variance from a defect?
Run a paired before-and-after comparison on the same items, judge the flip rate against a tolerance band agreed before the comparison, then adjudicate the flipped cases themselves. A flip rate on its own, with no tolerance and no adjudication, cannot tell you whether you have variance or a defect. The Production AI Monitoring Harness is TechnoLynx’s eval and regression harness for this comparison.
Aggregate accuracy hides flips. Dutta et al. at Microsoft Research (2024) found quantised LLMs within 0 to 2% of baseline accuracy, while all but one of the quantisation schemes switched up to 13.6% of individual answers between correct and incorrect. The flips are nearly balanced between the two directions, so the average barely moves. Model-update research calls the harmful direction a “negative flip”: a sample the new model gets wrong that the old model got right (Yan et al., CVPR 2021). Xie et al. (ACL 2021) found negative-flip regressions widespread across GLUE benchmark tasks. Hosted models change too. Chen, Zaharia and Zou measured GPT-4 identifying prime numbers with 84% accuracy in its March 2023 version and 51% in June 2023.
Do the statistics on question-level paired differences, not on two summary scores. Scores on the same items are correlated, so pairing reduces variance at no extra cost (Miller, 2024). For pass/fail outcomes, a paired test such as McNemar’s is what separates a regression from noise. Under the illustrative parameters in Miller’s power calculation, an evaluation needs roughly 1,000 independent questions to detect a three-point difference reliably.
Then look at where the flips fall. Dutta et al. show that answers are most likely to change where the model’s top margin (the probability gap between its best and second-best option) is small. In Dutta et al.’s quantisation study, low-margin items were the ones most likely to flip. Flips on items the old model answered with confidence, or on safety-relevant queries, that human reviewers rate as regressions point to a defect that needs a rollback or a targeted fix. The same authors are clear that a flip is a warning of behavioural difference, which may or may not show up as visible degradation on a given downstream task.
Before blaming the weights, check what else changed. The audit trail should capture runtime configuration (prompt policies, decoding settings, retrieval and guardrail layers) precisely so you can tell when a configuration change explains a change in outcomes. Also check that the human-adjudicated sample is large enough to speak for all the flips. Set the tolerance band for your task and record it like any other threshold.
Who signs as approver of record, and how is a conditional approval tied to a version?
The approver of record is the unit, individual or committee with authority to approve use of the model, typically after weighing the reviewer’s findings. For an LLM, that approval covers this specific model in this specific system, and it is separate from whoever approved the system as a whole. The PRA expects dedicated model approval authorities with clear roles set out in policy, and a senior manager accountable for the model risk framework overall. The validation behind the approval should not come from the model’s builders: NIST’s AI RMF treats separating those who build and use models from those who verify and validate them as best practice.
To capture the decision immutably, record it as a structured event, not an email thread. A 2026 audit-trail framework for LLMs records each decision event with its owner, its rationale, the scoped identifiers it applies to, any constraints (such as permitted jurisdictions or user groups) and references to the evaluation reports behind it. A condition becomes trackable when it is a field on that event, keyed to the model version and deployment, so it can be queried and closed rather than lost in free text. The OCC handbook names models with temporary exemptions or provisional approvals as one measure for model risk reports.
In regulated settings, the approval needs a secure, time-stamped, attributable audit trail in the style of Part 11. We cover the approval schema in more depth, including the field set and the go/no-go gate it feeds, in standardising model documentation and approval schemas.
Do minor retrains need the same sign-off rigour as a major version change?
No. They still need a logged sign-off, though: rigour scales with the risk tier and with how material the change is, and it never drops to zero. The PRA ties the nature and extent of independent review to the model tier and, for a model change, to the materiality of that change. E-23 lets the risk rating set the frequency, intensity and scope of review, the documentation required and the level of authority needed to approve. One practitioner guide treats any change to features, algorithm or training-data window as a material change that needs approval equivalent to a new release.
| Change | Sign-off depth |
|---|---|
| Scheduled retrain, same architecture, prompt and data sources, low tier | Review scaled to the low tier and the materiality of the change, still logged to the version |
| Retrain with a new data window or features | Documentation and approval equivalent to a new release, review depth set by tier and materiality |
| New base model or checkpoint family | Model review and risk re-rating, as for any model modification |
Rigour should match the potential risk the change presents.
What audit trail for drift escalation and sign-off will satisfy internal audit, and how long must it be kept?
The format that holds up records who saw each drift alert, what they decided, when and why, in storage where any edit or deletion can be detected, and keeps it for as long as the regime governing the underlying record requires. “Who” means a unique user identity, since shared accounts undermine attribution. “What” includes the previous and new values for any change. “When” is a timestamp from a synchronised clock that users cannot alter. “Why” is not explicitly required by 21 CFR Part 11 but is required by EU GMP Annex 11 and MHRA guidance.
A hash chain provides tamper evidence: each entry’s cryptographic hash covers its own data plus the previous entry’s hash, so inserting, changing or deleting an entry breaks the chain. The LLM audit-trail proof of concept implements this as a newline-delimited JSON log with SHA-256 hashes, using the core fields event_id, timestamp, system, actor, event_type, model_id, dataset_id, deployment_id, details, prev_hash, curr_hash and an optional signature.
Can existing model risk tooling log escalation and sign-off events automatically? In part. The 2026 framework builds on existing MLOps and experiment-tracking telemetry and adds explicit decision logging, append-only storage and auditor views, and its authors report integrating that layer into existing workflows with modest effort. Before relying on the tooling as evidence, test the audit trail itself as part of validation: confirm that all required events are captured, timestamps are accurate, entries are immutable and exports are complete and readable.
| Regime | Minimum retention |
|---|---|
| EU AI Act Article 18, technical documentation for high-risk systems | Ten years after the system is placed on the market |
| EU AI Act Article 19, automatically generated logs | At least six months, unless other Union or national law provides otherwise |
| FDA 21 CFR Part 11, audit trail documentation | At least as long as the subject electronic records |
| HIPAA records | Six years from creation or from when the policy was last in effect, whichever is later |
| EU GMP records | At least one year after batch expiry or five years after Qualified Person certification, whichever is longer |
Article 18 applies from 2 December 2027 for high-risk systems under Annex III and from 2 August 2028 for those under Annex I. Where no regime sets the period, agree it with internal audit in writing. Even a perfect trail only shows that events happened in a verifiable sequence. It does not show that one caused another, which still takes judgement.
What does the sign-off evidence pack need to contain to survive a later audit?
The pack is tied to the exact model version that ships. It holds the evaluation results with the thresholds set beforehand, the release strategy, the rollback plan and the approval record. TechnoLynx’s AI Governance and Trust engagement starts by agreeing with you what evidence your regulator or governance body will accept as sufficient. TechnoLynx then templates the pack so future releases reuse it, and helps generate it from pipeline run artefacts where the pipeline allows.
We treat the pack as an engineering output in its own right: evidence that a buyer, auditor or model-risk committee can read and sign against, built from the same pipeline as the model rather than written afterwards as a policy document.
The OCC handbook says validation reports should include a clear executive summary with a statement of the model’s purpose, major limitations and key assumptions, and a judgement on whether adjustments or compensating controls are warranted. One practitioner guide sets deployment approval criteria of a complete evaluation scorecard, a defined release strategy and a documented rollback plan, with risk or compliance consulted for regulated models. The same guide defines rollback criteria before deployment begins and adds a post-release sign-off once the observation window closes without a rollback.
Who co-signs before a release ships? The validator, whose recommendation goes to the approval authority, and the approver of record, with risk or compliance consulted for regulated models. Each signer needs the authority and competence to challenge. A signature from someone who could not have stopped the release adds paperwork, not defensibility.
Templating the pack makes it reusable for future model releases. The template needs versioning too, because a stale template produces a pack that answers last year’s questions. The evaluation half of the pack comes from the harness, which we cover separately in building a defensible, re-runnable evaluation harness. However complete the pack is, it records only what was tested and decided at the time. It cannot vouch for scenarios nobody tested.
Who can help us build an auditable sign-off process for LLM releases?
TechnoLynx builds the engineering behind a defensible LLM release sign-off: the model-risk inventory and approval workflow, the evaluation and regression harness with its release gate, and the evidence pack that records each decision. The governance side of that work runs through AI Governance and Trust, and the gate through the Production AI Monitoring Harness. If you are choosing a partner, ask for artefacts your auditors can rerun (a run identifier, a hash-chained decision log, a pack generated from the pipeline) rather than a governance slide deck.
The engagement produces engineering artefacts. It is not legal advice, regulatory sign-off or certification, and whether the evidence satisfies a particular regulator remains a decision for your compliance function or counsel. If you already run a mature model risk function with validated tooling, your own platform team extending the existing registry and gates is a realistic route. A governance or GRC platform may also be enough if what you mainly need is a system of record.
FAQ
Can an audit trail be deleted?
A compliant one cannot. Audit trail records must be immutable, and no user, including system administrators, should be able to alter, delete or disable them.
What does an audit trail report show?
It shows who performed each action, what they did (including old and new values for any change), when it happened according to a reliable clock, and, under EU GMP Annex 11 and MHRA guidance, why.
What is an AI inventory?
It is the organisation’s system of record for key information about its models, covering every model whose inherent risk is more than negligible. The NIST AI RMF lists inventory mechanisms, resourced according to risk priorities, under its GOVERN function.
