Two documents, two change cadences, two reviewers. The reliability artefacts answer whether the detector is still trustworthy; the runbook answers what the on-call operator does when it fires. Most teams ship them as one bundle, and that bundle is the reason a threshold change takes three weeks instead of two days.
The bundle looks complete at go-live. Sensitivity thresholds, drift telemetry, escalation tiers and the “what to do at 3am” procedure all sit in one document, owned by whoever edited it last. Then reality applies pressure from two directions at once. A data scientist retunes a threshold because the plant baseline moved; nobody knows whether the operator step downstream still matches. Operations edits the same document mid-incident because a phone number changed; the calibration evidence quietly loses its audit trail. Neither party did anything wrong. The document was asked to serve two owners with incompatible review cycles.
What does the boundary between reliability artefacts and runbooks mean in practice?
The split is not by topic. It is by what event forces a change.
Reliability artefacts version against model and data change. When you retrain, when the baseline window is recomputed, when the input distribution shifts — the sensitivity-calibration record, the false-positive review queue and the drift telemetry all need re-examination. Their reviewer is whoever can argue about score distributions.
Runbooks version against process and organisation change. A new shift pattern, a merged control room, a changed escalation contact, a revised permit-to-work rule. Their reviewer is whoever owns the operational consequence of a wrong call at 3am.
Ask of any paragraph in your current document: if the model is retrained tomorrow, does this sentence need review? If yes, it is a reliability artefact. If the answer is instead only if the org chart changes, it is runbook material. Sentences that answer yes to both are the ones causing the damage, and there is exactly one legitimate member of that class.
The divergence point is the escalation tier
Escalation tiers are the one object both sides genuinely touch. The reliability side defines them because tiers are a function of detection confidence — a low-confidence detection reaching a human without paging a control room is a calibration decision, not a staffing decision. The runbook side consumes them because tiers determine who gets woken up.
The rule we apply: escalation tiers are defined once, inside the validation pack, and referenced by identifier from the runbook — never copied. A runbook that says “Tier 2: notify the duty engineer within 30 minutes” is fine. A runbook that says “Tier 2: score above 0.87 sustained for five minutes” has swallowed a calibration decision, and the moment the detector is retuned that line becomes a lie nobody notices.
Copying is what breaks the audit trail. Two copies of a tier definition do not stay identical; they diverge on the first edit made in a hurry, and after that neither document can be trusted as the source.
Decision rubric: which side does this artefact belong on?
| Artefact | Side | Versions against | Reviewer | Runbook may |
|---|---|---|---|---|
| Sensitivity-calibration evidence (baseline window, threshold basis, accepted alert volume) | Reliability | Model + data change | Data / validation owner | Reference tier ID only |
| Drift telemetry definitions and breach limits | Reliability | Model + data change | Data / validation owner | State who is notified on breach |
| False-positive review queue schema and cadence | Reliability | Model + data change | Data / validation owner | Point operators at the queue |
| Escalation-tier definitions (confidence → tier mapping) | Reliability (canonical) | Model + data change | Data owner, countersigned by operations | Reference, never restate |
| Per-tier notification and acknowledgement steps | Runbook | Process + org change | Operations / control-room owner | Own outright |
| Diagnostic checks and containment actions | Runbook | Process + org change | Operations / control-room owner | Own outright |
| Contact lists, shift handover, permit rules | Runbook | Process + org change | Operations / control-room owner | Own outright |
| Retune request procedure (how ops asks for a threshold change) | Runbook, referencing the pack | Process change | Both, jointly | Own the request; not the decision |
The last row matters more than it looks. Operations needs a documented way to say “these alerts are wrong” that lands somewhere with a reviewer attached. Without it, the escalation path for a bad detector is muting the channel.
What a runbook should never contain about the detector
Three things, specifically: threshold values, model version pins, and any explanation of why an alert fired in probabilistic terms. Each of them is a statement about the detector’s current calibration, and the runbook has no mechanism to notice when it goes stale. A runbook that says “this alert usually means bearing wear” is encoding a claim about the detector’s precision on one failure class that no one has re-validated since go-live.
What the runbook should contain instead is a pointer: the tier, the queue, and the named artefact where the reasoning lives. We see this pattern regularly in industrial engagements — the runbook shrinks by a third once the detector explanations are removed, and it gets read more because it is now purely actionable.
How the boundary collapse announces itself
The failure is observable before anyone admits it. Three signals, in the order they usually appear:
- Alerts with no matching runbook step. A fired alert arrives at an operator whose document has no entry for it, because the detector gained a new detection class and the runbook was not part of that change set. Track this directly: percentage of fired alerts that resolve to a documented runbook step.
- Stalled retunes. A threshold change sits waiting because it requires an operations sign-off that only exists because the threshold lives in an operations-owned document. Track time-to-retune after a drift-telemetry breach.
- Muted channels. Operators stop trusting the mapping between alert and action, and mute. By this point the system is out of use even though every availability metric is green. The alert-mute path is worth understanding on its own terms — why anomaly systems silently lose trust traces the full sequence.
Clean separation is what lets a threshold change ship without a runbook rewrite, and a runbook edit ship without touching calibration evidence. Teams that reference escalation tiers rather than duplicating them tend to retune in days rather than stalling behind an operations review cycle (observed across TechnoLynx operational-anomaly engagements; not a benchmarked rate). The outcome that matters is the one the parent work measures: alerts still in active use six or more months past go-live rather than muted within a sprint. We develop that measure and the artefact set behind it in our work on production AI reliability, and the structural causes of month-3 abandonment are covered in the artefacts that keep an anomaly system in active use.
Ownership in a typical engagement
In industrial, energy and telecom operational-anomaly work, the reliability side sits with the validation owner — the person who can reconstruct why a threshold is what it is. The runbook side sits with the control room or NOC, and in energy work it almost always has an incumbent owner already: SCADA runbooks predate the anomaly system by years. That asymmetry is useful. Do not try to take the runbook; take the tier definition and hand back a reference.
Handoffs are recorded as a two-line changelog entry on each side. Reliability side: threshold changed, basis, reviewer, date, tier mapping unchanged or changed. Runbook side: tier reference re-verified on date X against pack version Y. Neither side edits the other’s document. That is the whole protocol, and it survives contact with a hurried Tuesday afternoon.
This rubric is scoped to equipment, grid and network anomaly — machines, assets and infrastructure. People-surveillance and behaviour-tracking use cases sit outside it deliberately; the reviewer questions there are legal and ethical before they are statistical, and a change-cadence split does not begin to cover them.
Frequently Asked Questions
What does the boundary between anomaly-system reliability artefacts and incident-response runbooks mean in practice? Does an anomaly qualify as a reliability artifact or trigger incident response protocols? It means splitting one document into two with different triggers for revision. Reliability artefacts are reviewed when the model or data changes; runbooks are reviewed when the process or organisation changes. If a paragraph would need re-reading after a retrain, it belongs on the reliability side.
Which artefacts belong on the reliability side, and why do they version against model and data change? Sensitivity-calibration evidence, drift telemetry, and the false-positive review queue. Each is a statement about how the detector currently scores a moving baseline, so a retrain or a distribution shift invalidates it directly — while a shift-pattern change leaves it untouched.
Which documents belong on the runbook side, and what should a runbook never contain about the detector? Notification steps, diagnostic checks, containment actions, contacts and handover rules. A runbook should never carry threshold values, model version pins, or probabilistic explanations of why an alert fired — it has no mechanism to notice when those go stale.
How should escalation tiers be defined once and referenced by both sides? Define the confidence-to-tier mapping inside the validation pack, give each tier a stable identifier, and let the runbook cite the identifier while owning only the human actions attached to it. Copying the mapping into the runbook guarantees divergence at the first hurried edit.
Who owns each side in an industrial or energy engagement, and how are handoffs recorded? The validation owner holds the reliability artefacts; the control room or NOC holds the runbook, and in energy work usually already did. Handoffs are a changelog line on each side — threshold changed with basis and reviewer on one, tier reference re-verified against pack version on the other.
What are the observable signals that the boundary has collapsed? Alerts arriving with no matching runbook step, retunes stalled behind an operations sign-off they should not need, and finally muted channels. The first two are measurable early; the third is the point at which the system is already out of use.
Drawing the handoff line between detection and response
Reliability artefacts should tell you that something broke and where; incident response owns everything after the Slack notification fires.
