A reliability audit report is engineering evidence about one deployed feature at one point in time. It tells a release reviewer what is covered, what is unmonitored, and what remediation is ranked next. It says nothing about legal conformity, and the moment it gets forwarded upward as proof that a feature is “certified safe”, it stops being useful and starts being a liability.
We see the confusion arrive the same way almost every time. The audit lands, the deliverable looks substantial — an eval-coverage map, a drift-monitor inventory, a release-readiness checklist, an ownership matrix — and someone in the reporting chain reads “audit” as “sign-off”. The document gets attached to a customer security questionnaire. Occasionally it gets cited in a regulatory submission. Nobody intended to overclaim; the artefact simply looked official enough to carry weight it was never built to carry.
What does “a reliability audit is not a safety certification” mean in practice?
The distinction is about who is making the claim and what the claim is anchored to.
An audit is a measurement exercise. It examines the operational surface of a specific deployed feature — how regressions would be detected, how they would be contained, who would be paged — and records the current state honestly, gaps included. The output is descriptive. Its authority comes from the fact that an engineer looked at the running system and wrote down what was actually there.
A safety certification, or more precisely a conformity assessment, is a legal determination made against a defined standard by a party with the standing to make it. It asserts that a system meets stated requirements. Its authority comes from the standard and the assessor’s accreditation, not from the depth of the engineering work behind it.
Those are different objects with different failure consequences. An audit that finds twelve unmonitored surfaces has done its job well. A certification that finds twelve gaps has failed.
Claims you can make, and claims that cross the line
The most practical way to hold this boundary is to write down, once, which sentences the deliverable supports. Every team we work with that skips this step ends up litigating it during an incident instead.
| Claim | Supported by the audit? | Why |
|---|---|---|
| “Eval coverage for this feature spans these input classes; these classes are uncovered.” | Yes | Direct observation, dated, scoped to the feature examined. |
| “These four drift surfaces are instrumented; label-feedback latency is not.” | Yes | Inventory of what exists in the running system. |
| “Rollback for this model version has a tested path and a named owner.” | Yes | Verifiable against the deployment configuration. |
| “Remediation items are ranked; items 1–3 are release-blocking.” | Yes | An engineering judgement the audit is qualified to make. |
| “This feature is safe.” | No | Unbounded, undated, and not tied to any standard. |
| “This feature is compliant with [regulation].” | No | Conformity is assessed against a standard by an accredited party. |
| “This feature has been certified.” | No | No certification body issued anything. |
| “We are audit-ready.” | No | Engineering readiness and legal audit-readiness are different states. |
The line is easier to see when you notice that every supported claim carries a scope, a date, and an object. Every unsupported claim is a general statement about the system’s status. Generality is the tell.
The divergence shows up at the first incident
This boundary looks academic until something breaks in production, and then it becomes the whole story.
If the team treated the deliverable as engineering evidence, the first incident is navigable. The eval-coverage map says whether the failing input class was in scope. The drift-monitor inventory says which monitor should have fired, and whether it existed at all. The ownership matrix says who held the pager and who had authority to pull the feature. The remediation ranking says whether this failure mode was already known and deferred — which is uncomfortable, but it is a defensible position with a paper trail.
If the team treated it as a certification, none of that is available in the form anyone needs. There is a document asserting the feature was fine, an incident proving it was not, and no traceable argument connecting the two. The organisation now has a claim it cannot defend, and the engineering value of the original work is buried underneath the overclaim.
We have found that naming the boundary in the first scoping conversation — before anyone has seen a deliverable — is what keeps the audit useful downstream. It costs one paragraph in a statement of work.
Where the audit legitimately gets consumed
Engineering evidence has a proper destination: the release gate. A release reviewer deciding whether a model version ships needs exactly what the audit produces — coverage, instrumentation state, rollback viability, ownership — and needs it as an input to a judgement, not as a substitute for one. That is a gate decision made by people inside the organisation who are accountable for the outcome. It is not a stamp handed down from outside.
The same evidence has a second legitimate use when a formal safety case genuinely is required. The eval-coverage map and the ownership matrix are inputs to a conformity file. They reduce the evidence-gathering effort substantially, because the hardest part of assembling a safety case is usually finding out what the system actually does rather than what the design document claims. What they do not do is move conformity-assessment status. Where a formal safety case is required and the audit’s outputs feed it, the assessment still has to happen; the audit shortens the runway, it does not replace the flight.
This is also why the audit’s scope boundary matters commercially. Teams that separate engineering evidence from a regulatory safety case avoid paying for the wrong workstream — and avoid re-running an audit whose output was never structured for a conformity file in the first place. The audit moves incident rate, time-to-detect, time-to-rollback, and eval-coverage delta. Those are the metrics to hold it to.
Why one audit pass is not the end state
A reliability audit is a snapshot. The input distribution moves, the model version changes, the retrieval index gets rebuilt, someone adjusts a threshold during an incident and never writes it down. Everything the audit measured has a decay curve.
What the audit can leave behind is durable: named monitors, a tested rollback path, an ownership matrix that survives a reorg, and an eval suite that runs on a cadence rather than on request. That standing capability is the deliverable that keeps working — the audit report itself is the point-in-time reading. We build this handover deliberately, and it is the substance behind our [production AI monitoring harness](Production AI Monitoring Harness); the broader engagement shapes are described on our services page.
For the full picture of what a production reliability audit examines and how the surfaces are prioritised, see our production AI reliability audit scope checklist, which covers the five operational surfaces this boundary discussion assumes.
Who signs off, if nobody is certifying?
Someone inside your organisation does — and that is the correct answer, not a gap in the process. A release reviewer, a platform lead, or an engineering manager accepts the residual risk on the record, with the audit’s evidence in front of them. Accountability stays where the deployment decision sits.
The uncomfortable version of this is that the audit does not let anyone off the hook. It gives the person signing better information about what they are signing, and it makes the unmonitored surfaces visible before they become an incident rather than after. That is a smaller promise than “certified”, and a considerably more useful one.
The question worth carrying into your next scoping conversation: if your deployed AI feature failed tomorrow, would your reliability evidence tell you which monitor should have fired — or only that someone once concluded things looked fine?
Frequently Asked Questions
What does “a reliability audit is not a safety certification” mean in practice?
Reliability audits measure operational consistency under real-world conditions, not compliance with predetermined safety thresholds. It means the audit describes the observed state of one deployed feature on a given date — coverage, instrumentation, rollback, ownership — rather than asserting conformity with a standard. Descriptive findings are the deliverable; a determination of legal conformity is not, because that requires an accredited assessor working against a defined standard.
What claims can we legitimately make from an AI reliability audit deliverable, and which claims cross the line?
Scoped, dated, object-specific claims are supported: which input classes evals cover, which drift surfaces are instrumented, whether the rollback path is tested and owned. General status claims — “safe”, “compliant”, “certified”, “audit-ready” — are not supported, because none of them is anchored to a standard or an assessment.
Can we attach a reliability audit report to a customer security questionnaire or a conformity file?
You can share it as engineering evidence, provided it is labelled as such and not presented as certification. In a conformity file the eval-coverage map and ownership matrix are useful inputs that reduce evidence-gathering effort, but the conformity assessment itself still has to be performed separately.
Why does a single audit pass not replace ongoing eval coverage and drift monitoring?
The audit is a point-in-time reading, and every surface it measured drifts afterwards — input distributions move, model and index versions change, thresholds get adjusted during incidents. What persists is the standing capability the audit leaves behind: named monitors, a tested rollback path, and evals that run on a cadence.
Who signs off on an AI feature release when nobody is certifying it?
An accountable person inside the organisation — typically a release reviewer, platform lead, or engineering manager — accepts the residual risk with the audit’s evidence in front of them. The audit improves the quality of that decision; it does not transfer the accountability outside the organisation.
Certification versus operational review
Reliability audits measure whether your system degrades gracefully under load; safety certification evaluates whether outputs meet regulatory or ethical guardrails. If AI Reliability Audit is on your roadmap, the next step is to map it onto your own constraints rather than copy a reference architecture.
