A sensitivity threshold is not a config value. It is a decision that traded recall for operator attention, and someone accepted that trade on a particular day, on a particular baseline window, for a particular incident class. If that acceptance is not written down in a form a reviewer can read months later, the threshold has no defence — and the first noise complaint after go-live will lower it blind.
That is the practical case for a tuning-evidence record. Not compliance theatre; a way to make the next re-tune a bounded engineering change instead of a guess.
The scope here is operational anomaly detection on industrial, energy and telecom assets. People-surveillance and behaviour-tracking sensitivity tuning is a different problem with different governance, and we are not addressing it.
What does documenting sensitivity-tuning evidence for reviewer sign-off mean in practice?
It means every threshold in the system has a record that answers four questions without anyone consulting the engineer who set it: what statistical basis produced the number, which historical incidents it would have caught and missed, how many alerts per operator per shift it implies, and who accepted that trade-off.
In our engagements the failure is rarely that teams have no reason for a threshold. They usually have a good one. The reason lives in a Slack thread, a notebook cell, or one person’s memory of a Tuesday afternoon. Six weeks later operators say the alerts are too noisy, nobody can say what the threshold was chosen to protect, and sensitivity gets lowered until the complaints stop. Recall on the incident class the system existed to catch goes with it, quietly, because nobody is measuring what the change cost. A few rounds of that and the channel is muted — the failure mode we describe in the reliability artefacts that keep anomaly systems trustworthy.
A tuning-evidence record is the artefact that converts “the alerts feel noisy” into a specific, answerable question about a specific number.
The threshold decision record
One record per threshold, per asset class. It should be short enough that a reviewer reads it in five minutes and complete enough that it survives the departure of whoever wrote it.
| Field | What it captures | Why a reviewer needs it |
|---|---|---|
| Threshold ID and scope | Which signal, which asset class, which operating mode | Thresholds rarely generalise across duty cycles; scope prevents silent reuse |
| Statistical basis | The rule that produced the number — e.g. 3-sigma on a robust mean, or a MAD-based cut at k=4.5 | Distinguishes a fitted value from an eyeballed one |
| Baseline window | Exact date range, sample count, and which operating modes were included or excluded | A threshold fitted on a summer window is a different threshold in winter |
| Incident replay result | Which historical incidents the threshold catches, which it misses, at what lead time | This is the recall statement; without it, “sensitivity” is an abstraction |
| Implied alert budget | Alerts per operator per shift on the replay window | The number the operations owner is actually accepting |
| Accepting reviewer | Name, role, date | Makes the trade attributable rather than ambient |
| Re-tune triggers | Pre-agreed conditions that force a review | Converts drift response from ad-hoc to scheduled |
The fields are not interchangeable. The two that get dropped most often — incident replay and implied alert budget — are the two that make the record decision-grade. Everything else describes how the number was produced; those two describe what it will cost and what it will buy.
How is a statistical baseline recorded rather than presented as a bare number?
Write the rule, the fit, and the exclusions — not the output. “Threshold = 3.2” tells a reviewer nothing. “Threshold = mean + 3σ of the robust baseline, fitted on 14 Feb–28 Mar, 41 days, startup and CIP cycles excluded, n=59,000 samples, giving 3.2 on the scaled vibration index” tells them what will change when the plant changes.
The distinction matters because 3-sigma and median-absolute-deviation cuts behave differently on the same data. Sigma-based thresholds inflate when the baseline window already contains the fault signature you are trying to detect; MAD-based cuts resist that but are harsher on genuinely multi-modal signals. Recording which one you used, and why, is what lets a later reviewer challenge the choice rather than the number. Where the threshold came out of an automated search rather than a closed-form rule, the study configuration and seed belong in the record too — an Optuna study that is not reproducible is not evidence.
Evidencing what a lowered threshold gives up
The noise complaint is where the record earns its keep. An operator says the system fires too often. The naive response is to raise the threshold until the complaint stops. The evidenced response is to state, before the change lands, what recall the new value would have had against the same historical incident set.
The comparison is mechanical once the replay harness exists:
- Re-run the historical incident set at the proposed threshold.
- Report incidents still caught, incidents newly missed, and the change in mean detection lead time.
- Report the new implied alerts per operator per shift on the same replay window as the original fit.
- Attribute the current false-positive volume to causes, using the dispositions captured in the false-positive review queue — noise from a genuinely changed plant is not the same problem as noise from a badly fitted threshold, and only one of them is fixed by moving the number.
- Append the change to the existing record. Never overwrite. The history of a threshold is part of its evidence.
That last point is the one teams resist and later regret. A threshold that has moved four times in six months is telling you something about the baseline that no single current value can express.
Stating the alert budget so a reviewer can reject it
The alert budget is the sentence a reviewer signs. It should read as a number per operator per shift, on a named window, with the recall it buys attached: “At this threshold, a control-room operator sees an estimated 4–6 alerts per 12-hour shift, and the detector catches 7 of the 9 bearing-degradation incidents in the 2024–25 historical set at a median lead time of 31 hours.”
Two properties make that statement reviewable. It is falsifiable against production data within a fortnight, and the operations owner can say no to it — which is the point. A budget nobody could reject is not a budget. In practice, the useful conversation is not about the threshold at all; it is about whether four to six interruptions a shift is what that team can absorb alongside everything else on their screens. Engineering cannot answer that. Operations can, and the record is where their answer gets written down.
Pre-agreed re-tune triggers
Thresholds should be reviewed on conditions, not on complaints. Naming the conditions in advance removes the political charge from a re-tune — it is a scheduled event, not an admission that something was wrong.
Triggers we typically write into the record: a shift in the input distribution beyond a stated bound on the drift telemetry; a sustained alert rate outside the accepted budget for two consecutive weeks; a confirmed missed incident of the target class; a change in operating regime such as a new product line, a seasonal load transition, or a major maintenance event; and a model or precision change — retraining, or a move to a lower numerical precision, shifts the score distribution the threshold was calibrated against and invalidates the fit. In the energy carveout the seasonal trigger does most of the work, because baseline windows there move with load in a way that has to be anticipated rather than detected.
Where tuning evidence ends
The record answers why is the threshold what it is and who accepted the trade. It does not answer what does the operator do at 03:00 when it fires — that is runbook territory, with a different owner and a much faster change cadence. Mixing them means every runbook edit made during an incident quietly disturbs the calibration audit trail; we set out the boundary between reliability artefacts and incident-response runbooks separately for that reason.
Keep them in separate documents with separate reviewers. The tuning record changes when the plant or the model changes. The runbook changes when the response process changes. They reference each other; they do not merge.
The open question in most engagements is not what to record — the fields above are stable across industrial, energy and telecom deployments — but who owns the record once the delivery team leaves. A tuning-evidence artefact with no named steward is an artefact that stops being appended to, and an un-appended record is worse than none: it looks authoritative while describing a threshold that moved three re-tunes ago.
Frequently Asked Questions
What does documenting sensitivity-tuning evidence for reviewer sign-off mean in practice? On Documenting Anomaly Sensitivity Tuning, the short answer is this. It means each threshold carries a written record of its statistical basis, the baseline window it was fitted on, the historical incidents it catches and misses, the alerts per operator per shift it implies, and the named person who accepted that trade. The test is whether a reviewer who was not present can evaluate the threshold months later without asking the engineer who set it.
What fields must a single threshold decision record contain to be audit-readable months later? Threshold ID and scope, statistical basis, exact baseline window with exclusions, incident replay result, implied alert budget, accepting reviewer and date, and pre-agreed re-tune triggers. Incident replay and alert budget are the two fields most often dropped and the two that make the record decision-grade.
How do you evidence the recall traded away when sensitivity is lowered after an operator noise complaint? Re-run the historical incident set at the proposed threshold before the change lands, and report incidents still caught, incidents newly missed, and the change in detection lead time alongside the new alert volume. Append that comparison to the existing record rather than overwriting the previous entry.
How is the alert budget stated so a reviewer can accept or reject it? As a number of alerts per operator per shift on a named window, with the recall it buys attached in the same sentence. It must be falsifiable against production data within weeks and genuinely rejectable by the operations owner — a budget nobody could decline is not a budget.
How are statistical baselines such as 3-sigma or MAD recorded as the basis of a threshold rather than presented as a bare number? Record the rule, the fit and the exclusions: which estimator, which date range and sample count, which operating modes were excluded, and the resulting value on a named scale. Sigma-based and MAD-based cuts behave differently on the same signal, so naming the choice lets a reviewer challenge the method rather than the number.
What pre-agreed triggers should force a re-tune, and how is each re-tune appended to the existing evidence record? Typical triggers are drift beyond a stated bound, a sustained alert rate outside the accepted budget, a confirmed missed incident of the target class, an operating-regime change, and any retraining or numerical-precision change. Each re-tune is appended as a new dated entry with its own replay result and accepting reviewer, never as an overwrite.
Where does tuning evidence stop and incident-response runbook territory begin? Tuning evidence answers why the threshold is what it is and who accepted the trade; the runbook answers what an operator does when it fires. They have different owners and different change cadences, so they stay in separate documents that cross-reference each other.
Four artifacts reviewers check before signing off on tuning
Parameter sweeps, holdout confusion matrices, edge-case annotations, and rollback procedures must all be timestamped. Documenting Anomaly Sensitivity Tuning rewards teams that measure first and argue later — start with the smallest instrumented slice and let the numbers settle the design.
