Pick a sensitivity threshold for an anomaly detector by reporting the best objective a search found, and you have a number that works on the day you tuned it and nobody can defend six weeks later. The framework you reach for — Hyperopt or Optuna — is usually chosen by habit, and habit picks the wrong thing to optimise for. Both frameworks will find you a good sigma multiplier or MAD cutoff. Only one of them, used deliberately, leaves behind evidence a reviewer can sign against. That distinction is the whole game for an operational anomaly system. The threshold is not the deliverable. The justification for the threshold is the deliverable, because when drift telemetry flags a shift in three months, someone has to decide whether to re-tune or mute the alert — and they can only make that call if the original tuning run is still inspectable. What are Hyperopt and Optuna, and what does the choice actually decide? Both are black-box hyperparameter optimisation libraries. Both search a parameter space by evaluating an objective function and using past trials to propose the next candidate. Hyperopt’s default is Tree-structured Parzen Estimator (TPE) search over a space you declare with hp.uniform, hp.loguniform, and friends; you drive it with fmin and it hands you back the best point it found. Optuna also defaults to TPE, but you drive it with a study object that samples parameters imperatively inside an objective function, and — this is the part that matters — it persists every trial to a storage backend. Reduced to the search algorithm, the two are close to interchangeable. TPE against TPE, on a low-dimensional threshold space, will land in a similar neighbourhood. If you only care about the best objective value, you can flip a coin. The naive framing stops there: treat both as black boxes, report the best value, ship the threshold. The expert framing treats the search itself as calibration evidence. What objective encoded the false-positive/true-positive trade-off? How was the search space over sigma or MAD bounded, and why those bounds? Can a reviewer re-run the study and get the same answer? On those questions the two frameworks diverge sharply, and the divergence is about reproducibility and audit trail — not about which one finds a marginally lower loss. Defining a tuning objective that encodes the FP/TP trade-off Before either framework matters, the objective has to be right, because a search optimises exactly what you ask for and nothing else. For an anomaly detector, “best” is never a single accuracy number. A threshold that fires on every borderline event catches real faults and buries operators in false positives; a threshold that only fires on the obvious catches nothing new. The objective must encode that trade-off explicitly. In practice we define the objective over a labelled or semi-labelled window of operational data and score the candidate threshold against an operating-point constraint rather than a free-floating metric. A common shape is to maximise true-positive rate subject to a false-positive budget — for example, the recall achievable while holding daily false alerts under a fixed count. That budget is a business input, not a tuning parameter, and writing it into the objective is what makes the tuned threshold mean something. This is the same discipline that separates a real calibration exercise from a vanity sweep. A hyperparameter sweep only helps a line-side model when the objective reflects the operating constraint it will live under; the same is true for a sensitivity threshold. Get the objective wrong and it does not matter which framework you use — you have precisely tuned the wrong thing. Comparison: Hyperopt vs Optuna for a threshold study that must survive review The table below compares the two on the axes that decide whether a tuning run becomes a defensible artefact. This is a structural comparison of the frameworks’ defaults and native features, not a benchmark of search quality. Axis Hyperopt (default use) Optuna (default use) Search algorithm TPE, random, adaptive TPE TPE, CMA-ES, random, grid Study persistence Not built in; you persist Trials yourself (e.g. pickle / MongoDB) Native RDBStorage (SQLite/PostgreSQL) — every trial durably recorded Trial history for review Only if you wrote the plumbing Queryable out of the box (study.trials_dataframe()) Pruning of unpromising trials Not native Native pruners (median, successive halving) Resume / re-tune later Manual reconstruction Reopen study by name from storage Search-space definition Declared up front in the space dict Defined imperatively in the objective (trial.suggest_float) Audit-trail readiness Requires deliberate engineering Default behaviour The pattern here is consistent across the anomaly-tuning work we see: Hyperopt can absolutely be made auditable, but you have to build the persistence and history capture around it. Optuna gives you that scaffolding as the default. When the tuning run has to become an audit artefact rather than a throwaway script, defaults win, because the thing that does not get built is usually the thing nobody scoped time for. How does framework choice affect reproducibility and reviewer sign-off? A reviewer signs a threshold when they can answer four questions: what objective was optimised, what space was searched, what trials were run, and why the chosen value beats its neighbours. A stateless Hyperopt loop that returns only best answers the last question poorly and the middle two not at all — the trial history evaporated when the process exited. Optuna’s persisted study answers all four by construction. The storage backend holds every trial’s parameters and objective value, the search space is reconstructable from the objective code, and study.trials_dataframe() produces the trial table a reviewer reads directly. This is the reproducibility property that matters operationally: not that the search is deterministic to the bit, but that the evidence for the decision outlives the run. A persisted study is inspectable calibration evidence; a returned scalar is a claim with no working shown. None of this is exclusive to Optuna as a library. You can attach the same rigour to Hyperopt, and you can attach it to a Weights & Biases sweep tuning anomaly-baseline sensitivity with documented evidence. The framework choice sets the default; the discipline is yours. What we care about is that the tuning study becomes part of the production-AI reliability validation lens — the sensitivity-calibration evidence a validation pack requires — and that it does so without heroics. Bounding the search space so the tuned threshold stays defensible An unbounded or badly bounded search space produces a “best” value that a reviewer cannot trust, even if the framework recorded everything perfectly. The bounds are themselves calibration evidence. For statistical thresholds like a sigma multiplier on a Gaussian baseline or a scaling factor on median absolute deviation (MAD), the space should be bounded by what the underlying statistics can defend, not by what the optimiser can explore: Anchor to the estimator’s meaning. A 3-sigma threshold on a well-estimated Gaussian corresponds to a known tail probability. Bounding the search to roughly the 2–5 sigma band keeps every candidate interpretable; a value the optimiser found at 1.1 sigma is mathematically optimal on your window and operationally indefensible. Prefer robust statistics where the baseline is contaminated. MAD-based thresholds tolerate outliers in the baseline period far better than standard-deviation thresholds, which is why we bound and tune the MAD scaling factor when the training window itself contains occasional faults — a common condition in industrial telemetry. Bound to the false-positive budget, not just the metric. If a threshold value implies more daily alerts than operators can action, it is out of bounds regardless of its objective score. Record the bounds and the reasoning in the study alongside the trials. When a reviewer asks “why not lower?”, the answer is in the artefact, not in someone’s memory. When drift telemetry flags a shift, re-tune instead of mute This is where the persisted study earns its keep. Six months after go-live, drift telemetry flags that the signal distribution has moved. The operator now has two options: mute the alert because it “keeps going off,” or re-tune the threshold against the new distribution. Muting is what happens when the original study is gone — nobody remembers what the threshold encoded, so the safe-feeling move is to silence it, and the anomaly system quietly dies. A reopenable Optuna study changes the default. The operator (or engineer) reopens the study by name, feeds a fresh labelled window, and re-runs the same objective with the same bounds. The output is a new threshold with a new trial history that plugs straight back into the validation pack. The one-time hyperparameter run has become a maintainable calibration process. That is the difference between an anomaly system in active use past its first drift event and one muted within a sprint. The same closed loop — tuned threshold, drift signal, re-tune — is what keeps threshold-heavy systems honest in the field. The applied energy carveout shows the same reproducible-calibration pressure where operational thresholds carry real cost, and the pattern generalises well beyond anomaly detection into any system whose sensitivity is a tuned parameter. FAQ How does hyperopt vs optuna work in practice? Both are black-box hyperparameter optimisers that search a parameter space using an objective function and past trials — both default to Tree-structured Parzen Estimator search, so on a low-dimensional threshold space they land in similar places. In practice the meaningful difference is not search quality but what each leaves behind: Optuna persists every trial to a storage backend and can prune unpromising trials, while a default Hyperopt loop returns only the best point unless you build persistence yourself. How do you define a tuning objective that encodes the false-positive vs true-positive trade-off for an anomaly detector? Score candidate thresholds against an operating-point constraint rather than a free-floating accuracy metric. A common shape is to maximise true-positive rate subject to a false-positive budget — for example, the recall achievable while holding daily false alerts under a fixed count. That budget is a business input written into the objective, which is what makes the tuned threshold mean something operationally. How does the search framework choice affect whether a sensitivity-calibration study is reproducible and reviewer-signable? A reviewer signs a threshold when they can see what objective was optimised, what space was searched, what trials ran, and why the chosen value beats its neighbours. A stateless Hyperopt loop that returns only the best value loses the trial history when the process exits; Optuna’s persisted study answers all four questions by construction because the storage backend keeps every trial. The reproducibility that matters is that the evidence for the decision outlives the run. How do Optuna’s study storage and pruning compare to a hyperopt search when the tuning run must become an audit artefact? Optuna provides native RDB storage (SQLite/PostgreSQL) and pruners as defaults, so the trial history and a queryable trial table exist without extra plumbing. Hyperopt can be made equally auditable, but you have to build the persistence and history capture around it. When the run must become an audit artefact, Optuna’s defaults reduce the risk that the audit-trail work simply never gets scoped. How should the search space over statistical thresholds like sigma or MAD be bounded so the tuned value stays defensible? Bound the space by what the underlying statistics can defend, not by what the optimiser can explore. Anchor a sigma multiplier to interpretable tail probabilities (roughly a 2–5 sigma band), prefer MAD-based thresholds when the baseline period is contaminated by outliers, and treat the false-positive budget as a hard bound. Record the bounds and the reasoning in the study so a “why not lower?” question is answered by the artefact. When drift telemetry flags a shift, how does a persisted tuning study let you re-tune instead of muting the alert? A reopenable study lets an engineer reload it by name, feed a fresh labelled window, and re-run the same objective with the same bounds to produce a new threshold and trial history. Without that study, the safe-feeling move when an alert “keeps going off” is to mute it — which quietly kills the anomaly system. Persistence turns a one-time hyperparameter run into a maintainable re-tuning process. The threshold is not the artefact — the study is If there is one thing to carry out of the hyperopt-vs-Optuna decision, it is that you are not choosing a search algorithm. You are choosing whether your sensitivity threshold arrives with its working shown. The framework that makes the study persistent, resumable, and reviewable is the one that keeps an anomaly system alive past its first drift event — and a threshold nobody can re-derive is the undocumented-artefact gap that turns alerts into noise. When you scope anomaly-tuning work, ask which framework makes the calibration evidence a byproduct of the run rather than a task you hope someone remembers to do.
