We hold that an anomaly detection system stays in use after go-live only while operators keep believing its alerts. Production reliability, distinct from model accuracy, is delivered as engineering artefacts that catch alert fatigue and missed events. The ones that matter most are a sensitivity-calibration record that says why each threshold sits where it does, drift telemetry that separates a new normal from a fault, and a documented false-positive review queue that shows every suppression decision. Without them, operators learn to ignore the alerts, which is the alert-mute failure mode.

Google’s SRE book describes a Bigtable team whose SLO-based email and paging alerts fired so heavily that engineers spent their time triaging noise and often missed the problems that actually affected users. A 2025 industry paper by practitioners at a cloud provider puts it more bluntly: any alerting feature with a high false-positive rate is likely to be ignored soon.

Here, anomaly detection means industrial, energy and telecom operations, not behaviour tracking; we explain where we draw that line in industrial anomaly detection versus people surveillance.

What artefacts keep an anomaly detection system in active use past month 3?

We pair each artefact with the specific way of losing trust that it catches. Read the table as a checklist: if one row has no owner and no stored record, the matching failure has nothing to catch it.

Artefact	What it records	Failure it catches
Sensitivity-calibration record	The baseline statistic, threshold multiplier, window, confidence level and the written reason for each choice	Limits computed from too little data; anomalies masked by non-normal, multimodal data
Drift telemetry	Sustained change in the input distribution against the training baseline, plus each retrain-or-not decision	Flagging a new normal as anomalous, or quietly learning a fault as normal
False-positive review queue	Each dismissed alert, the reason, and any suppression rule with its scope	A noisy source getting normalised, and broad suppression hiding a real event
Escalation-tier evidence	Which alerts page a person and which go to a dashboard	Paging on causes or subcritical problems instead of symptoms

That is deliberate. Production reliability is a different discipline from model accuracy, and the deliverable is the set of artefacts that catch regressions, not a dashboard.

A strong benchmark score buys no operator trust

Research on time-series anomaly benchmarks gives good reasons to discount reported detection performance. Kim et al. (AAAI 2022) showed that point adjustment, an evaluation protocol most time-series anomaly papers apply before scoring, can overestimate detection so much that even a random anomaly score looks state of the art. Wu and Keogh argue that most individual exemplars in the common time-series anomaly benchmarks have at least one of four flaws, so much of the apparent progress in recent years may be illusory.

Even when the scoring is clean, the metric may not fit the job. Lavin and Ahmad’s Numenta Anomaly Benchmark paper (2015) points out that plain precision and recall ignore time and give no credit for early detection. The same paper notes that in an expensive industrial machine a false negative can mean a production outage, while a false positive means a technician looks at the data more closely, so the two errors should not carry equal weight.

There is also the question of whether a detected anomaly points anywhere useful. A 2025 study of log-based anomaly detection found that baseline methods ranked the true root-cause event first only 13.70% of the time (HR@1), so their alerts were often unrelated to the actual cause even when they correctly detected that something was wrong.

When anyone presents detection performance as evidence, we would ask three things. Which scoring protocol was used? Was the threshold chosen with access to the full series and its ground truth (the ideal threshold many benchmarks assume, which a streaming system never has)? Do the alerts point operators towards causes?

How do the 3-sigma rule and MAD relate to calibration evidence?

The 3-sigma rule and the median absolute deviation (MAD) matter here because the calibration record has to say which one the threshold rests on, and why that choice holds for this data. The 3-sigma rule, for instance, assumes the data is normally distributed.

The NIST/SEMATECH handbook gives the textbook property of the 3-sigma rule: signalling only on points beyond ±3σ produces a false alarm every 371 points on average, and adding the Western Electric (WECO) rules raises that to about one every 91.75 points. NIST also warns that these false-alarm properties assume the mean and standard deviation are known, and that limits computed from limited data can behave quite differently.

Twitter’s engineers reported in 2017 that the time series they collected in production were seldom, if ever, normally distributed, so they judged the 3-sigma rule not applicable. Multiple modes in the data inflated the standard deviation by up to 5% in their analysis, which masked some true anomalies. On one heavily anomalous series, their median/MAD variant (S-H-ESD) correctly flagged 29.68% of points, against 1.11% for the mean/standard-deviation version (S-ESD).

Leys et al. (2013) make the same point with eight values (1, 3, 3, 6, 8, 10, 10 and 1000): a mean ±3 standard deviations rule does not flag the 1000. They recommend median ±2.5 MAD as a default, and add that the threshold should be justified, with the justification making clear that something other than cherry-picking guided the choice.

For an operational anomaly system, that turns into concrete content for the calibration record:

• the distribution check (normality, modality, seasonality) that justified the chosen statistic;
• the statistic, the multiplier and the written reason for the multiplier;
• the lookback window and confidence level (Twitter’s production setup, for example, used α = 0.05 over the last 14 days of data);
• the false-alarm rate expected under those assumptions, next to the rate observed in shadow running;
• recalibration triggers, which NIST frames as at least 30 new data points with no known process change, or a major process change.

Teaching these statistics in general is a separate subject. What matters on this page is that a threshold with no recorded rationale cannot be defended when someone asks why the system stayed silent.

Sensitivity is an operating point you record, not a setting you find

Tuning an alert threshold means balancing false-positive against false-negative rates. A 2019 study on a power-generation control network, using one month of operational data, tested attack detectors at three attack-sensitivity degrees. The average false-positive rate was 11.3% with a reduced feature set and 20.6% with the full one, and one group of test data reached 41.5%. The same authors made the thresholds dynamic so they followed changing operating conditions.

Sensitivity has an operating point, chosen by weighing which error costs more, as in the Numenta industrial-machine example above. The industry paper cited above notes that threshold setting is critical in real systems yet usually treated as a separate issue, and that it is often unclear how to set one effectively. If you are choosing between search tools for that job, we compare them in Hyperopt vs Optuna threshold tuning.

We treat signing off a threshold as a governance step. ISA-18 alarm management is an established lifecycle for alarm changes. Under ISA-18, every change to an alarm requires authorisation, goes back through rationalisation and design, and triggers updates to the documentation and to operator training. Periodic audits then review the work processes and performance of the alarm system. We would model threshold sign-off on that lifecycle, and the detail of what goes into the record is covered in documenting tuning evidence for sign-off. The point of the record is that the person who approved the threshold can later show what was known, what was decided and why, whichever way the system performs.

Drift telemetry separates a new normal from an incident

Drift is a sustained shift to a new normal; an anomaly is a sporadic deviation from it, and drifted logs should be used to retrain the model. Getting that wrong cuts both ways. Treat drift as anomalies and the system falsely flags the new normal activity after a change. Treat anomalies as drift and the system learns the fault as normal. The Numenta authors define anomalies relative to past behaviour, so a new pattern stops being anomalous once it persists. So the decision to retrain on a new baseline needs human involvement. The Bayes Factor method proposed for log anomaly pipelines raises a drift alert and asks a user to decide whether to retrain.

Google’s ML Test Score adds a further cause: upstream partial outages, version upgrades and other changes in source systems can change what a feature means without producing values strange enough to trip other monitors. What to instrument, and how drift alerts are tuned, is covered in drift telemetry beyond a classical model.

The false-positive queue is evidence, and it has a blind spot

We would run the false-positive review queue separately from incident triage. Security-operations guidance warns that when both share one queue, a noisy detection can stall analysis of a real infection. The same guidance says suppression should be narrowly scoped and reviewed for side effects, because the main failure mode is normalisation: a team seeing frequent false positives from one source trusts it less and misses the malicious case. It also asks that any suppression or auto-close rule leave the original artefact, the affected asset and the reason for the downgrade visible.

The queue also has a structural limit. Feedback in streaming anomaly detection is usually one-sided: people annotate false positives but rarely supply labels for false negatives. A clean queue can therefore show precision improving while saying nothing about missed events, so we would pair it with recall measured from incident retrospectives. On escalation, Edge Delta’s 2025 vendor guidance sets paging alerts at above 90% precision, on the grounds that on-call engineers otherwise start ignoring them. Treat that as a vendor’s rule of thumb, not a measured result. How the queue becomes an auditable record is covered in the false-positive review queue as evidence.

Where reliability artefacts stop and the incident runbook starts

Reliability artefacts deal with the quality of alerts. Runbooks set out the next steps once an alert fires. Security-operations guidance frames a similar split: false-positive handling as a classification problem and malware triage as an evidence problem. The calibration record, drift log and review queue answer “should this alert exist, and at this sensitivity?” The runbook answers “what do we do now that it fired?”. The handover points between the two are covered in artefacts versus incident runbooks.

How do these artefacts fit into an existing SCADA or observability stack?

We would put the artefacts into the tools operators already work in. In process industries, that means mapping them onto alarm management, whose lifecycle already covers shelving, nuisance alarms and assessment:

1. Alert budget. An operator should see no more than one to two alarms every ten minutes. Anomaly alerts that reach the operator’s alarm list count against that same budget.
2. Suppression as shelving. Under ISA-18, shelved alarms reappear after a preset period so they are not forgotten. We would implement false-positive suppressions as time-bounded shelving that expires and returns for review, not as permanent deletions.
3. Review queue as nuisance-alarm work. Identifying nuisance alarms (chattering, fleeting or stale ones) is a key activity in the alarm lifecycle. We would feed the false-positive queue into it.
4. Telemetry on dashboards, not pagers. The SRE book recommends dashboards for subcritical problems and reserves paging for symptoms. Drift telemetry belongs on that dashboard.
5. Automated assessment. In the alarm lifecycle, performance is compared frequently against the alarm philosophy, usually by an automated process. Calibration targets can be written into the same philosophy document.

On the IT side, an ICALEPCS 2023 paper argues that large facilities running complex SCADA systems should adopt observability tooling (real-time metrics, anomaly detection, log and trace analytics) as a core part of their platforms. Dynatrace, a vendor in that market, claims that this kind of observability also supplies the audit trail that frameworks such as IEC 62443, NERC CIP and NIS2 increasingly expect. Whichever stack carries the telemetry, model and feature versions should be tracked so changes stay auditable.

The same pattern across monitoring workloads

We would apply the same artefact set while the signals change. In condition monitoring we would apply the calibration record to vibration, temperature and electrical baselines. The condition monitoring equipment and vibration monitoring software pages cover how sensors and analysis tools feed it, and transformer monitoring applies it to one asset class. Line-side vision inspection is covered in industrial CV inspection production reliability.

If a language model summarises alerts for operators, its serving cost is covered in KV-cache reuse with RadixAttention. If you train the detection models yourself, FP8 training for anomaly-detection models explains how lower-precision training works in practice. Given the root-cause findings above, any such summary should be checked against the events it claims to explain.

The question still open at handover

Every artefact on this page can be in place and the system can still lose trust, because none of them works without an owner after the project team leaves. The calibration record goes stale without recalibration triggers, the review queue fills without a reviewer, and drift alerts pile up without someone authorised to accept a new baseline. Missed events are the one gap the artefacts cannot close on their own: operators label false positives, not false negatives, so recall has to come from incident retrospectives that someone must keep running. For a plant-level view of how these roles were assigned and kept up, see the worked industrial example.

FAQ

What is an anomaly detection system?

An anomaly detection system flags patterns that do not match a data stream’s past behaviour, including single-point spikes and changes over time. In practice it is used either to alert a person or to trigger an automated response.

What is another term for anomaly detection?

Outlier detection is a widely used alternative term. Novelty detection is used when the model is trained only on normal data.