Treat a false-positive spike that arrives with a new merchant category as a local problem until shown otherwise, not a broken fraud detection model, and make the threshold the last thing you touch. First, trace the spike to one of three causes: the new segment’s features sit outside what the model learned, its labels are missing or too immature to trust, or the population really is riskier. Each cause needs a different fix. Only the third is fixed by recalibrating that segment’s threshold.
The rest of the work follows in order: confirm the shift with a proper statistical test, correct calibration where it is actually wrong, test the new threshold in shadow mode on replayed traffic, monitor so the next onboarding is routine, and record the decision so it can be defended later. Done in that order, it is also the most dependable answer to how to reduce false positives without quietly letting more fraud through. Getting this wrong costs more than it looks: a Javelin Strategy & Research report, cited in a study of BBVA transaction data, put sales blocked by false positives at $118 billion, against $9 billion of real card fraud.
Why did false positives spike after we onboarded a new merchant category?
A false-positive spike confined to one new merchant category points first to a distribution shift in that segment, not a global model failure, and the diagnosis has to separate feature drift, a labelling gap and genuine risk change before any cutoff moves. The three causes come apart only with three separate checks: a feature stability test, a label audit for the new category and a base-rate comparison.
The checks are not equally informative. A feature drift alarm on a new category is expected, and on its own it settles little. What matters is whether the scores the model produces for that segment match the outcomes that eventually arrive. Outcomes arrive late: in card fraud, whether a transaction was fraudulent is known only days or weeks later, when customers report unauthorised charges. That delay cuts both ways. The alerts analysts close as legitimate this week can include fraud that gets reported next month, so the measured false-positive rate for a young segment is itself provisional.
Telling the three causes apart
| Cause | What it looks like | Check that separates it | What fixes it |
|---|---|---|---|
| Feature (covariate) drift | Segment feature distributions and the score distribution move away from the reference period, measured with PSI or KS statistics | Per-feature PSI and score-distribution PSI/KS for the segment against the training reference | Feature engineering or retraining with segment data, not a threshold change |
| Labelling gap or immature labels | The training data held few or no labelled examples for the category, or its recent labels are too young to include late-reported fraud | Label audit: how many of the segment’s transactions are old enough for disputes to have arrived, and how many were in training | Collect matured labels before retraining; do not trust early performance numbers |
| Genuine risk change | Confirmed fraud rate in the segment is higher than the portfolio’s, even after labels mature | Base-rate comparison on matured labels; label-free proxy monitors miss it if inputs are stable | Segment-specific calibration or threshold, treated as the new normal rather than an anomaly |
The labelling row deserves more caution than it usually gets. Stripe has described how the most recent weeks of its training data carried the most immature labels, because not enough time had passed for fraud reports to arrive. When Stripe stopped downsampling legitimate payments, it also added about 20 times more fraudulent examples wrongly labelled as legitimate, and the model got worse at catching fraud reported weeks later. A new merchant category is all recent weeks, so check label maturity before you blame the features or the population.
| The third row has its own trap. Real concept drift is a change in P(Y | X): the relationship between inputs and fraud moves, which can invalidate decision boundaries even when the input distributions stay stable. In a 2026 evaluation on the IEEE-CIS fraud dataset, label-free proxy monitors caught covariate drift in every window but missed concept drift with an unchanged feature distribution in all five windows. If only feature-drift indicators fire, covariate shift is the likely cause. If cross-model disagreement fires without feature drift, concept drift or adversarial manipulation becomes more probable, and a definitive diagnosis still needs eventual labels or expert review. |
Is the shift in one segment or the whole portfolio?
To tell whether score drift belongs to one segment or the whole portfolio, run the same score-distribution comparison twice: once for the new category, and once for the portfolio with that category excluded. If the rest of the portfolio has also moved, there is concurrent drift, and blaming the new category alone would be a mistake. The common industry rule of thumb treats PSI above 0.2 as significant shift. A 2026 follow-up paper argues that this figure is a heuristic, not a threshold calibrated to governance needs. For a category only a few weeks old, the segment estimate will also be noisy, so read its PSI alongside its sample size rather than against a fixed cutoff.
Is an 8% to 11% move significant at 4,000 transactions a week?
You cannot call a percentage-point change in a rate significant without accounting for sample size and the sampling variation you would expect anyway. The two-proportion z-test compares two rates. It checks whether two groups differ in the proportion of a binary outcome by more than chance would explain. The test is appropriate when observations are independent and samples are large enough for the normal approximation to hold. When sample sizes are small, Fisher’s exact test is more suitable.
Worked example: significance and sample size for an alert-rate move
The test assumes the two weeks are independent samples.
- Pooled rate: (320 + 440) / 8,000 = 0.095.
- Pooled standard error: √(0.095 × 0.905 × (1/4,000 + 1/4,000)) ≈ 0.0066.
- z = 0.03 / 0.0066 ≈ 4.6, well past the 1.96 critical value for a two-tailed test at α = 0.05.
- For the confidence interval, use the unpooled standard error (pooled variance is for testing equality, unpooled for intervals): √(0.08 × 0.92 / 4,000 + 0.11 × 0.89 / 4,000) ≈ 0.0065, so the 95% interval for the difference is about 3.0 ± 1.3 points, roughly 1.7 to 4.3 points.
- Sample size for future moves: detecting a smaller shift, 8% to 9%, with 80% power at a two-sided 5% level needs about [1.96√(2 × 0.085 × 0.915) + 0.84√(0.08 × 0.92 + 0.09 × 0.91)]² / 0.01² ≈ 12,200 transactions per group, which is about three weeks of this segment’s volume on each side.
The power step is what stops arguments about every wobble on the dashboard. A power analysis for two proportions takes the reference rate, the effect you care about, the significance level and the target power, and returns the sample you need. At low base rates the numbers grow quickly: Stata’s manual works an example where telling 1.71% from 0.94% at 80% power takes a total sample of 6,922.
Three cautions apply to this example. Transactions may not be independent: if batches from one merchant, one day or one attack campaign are correlated, the effective sample size shrinks and significance looks stronger than it is. Testing every segment every week is multiple testing, and that, along with selection effects, can invalidate p-values and intervals. A borderline result, such as a p-value just under 0.05, should be weighed against the business context and not treated as an automatic pass. For a trend you watch continuously, sequential methods fit better than repeated pairwise tests. They accumulate evidence across windows and flag sustained drift once the cumulative score crosses a set threshold. Separately from all of this, statistical significance is not a business threshold for action, which you should set on its own terms.
Fixing overconfidence in one score band of a fraud detection model
When a fraud detection model is overconfident in one band (say, transactions scored 0.6 to 0.8 turn out to be fraud far less often than the score claims), the fix is post-hoc calibration validated band by band, not a blunt threshold shift. In a well-calibrated binary classifier, among the cases given a score near 0.8, roughly 80% should actually be positive. Calibration means fitting a separate model, the calibrator, that maps raw scores to calibrated probabilities.
Platt scaling or isotonic regression?
Platt scaling fits a sigmoid to the raw scores, so it assumes the calibration curve can be corrected by a sigmoid, an assumption scikit-learn’s documentation notes “does not necessarily hold in general”. Isotonic regression assumes only that the mapping increases monotonically, and it fits a piecewise-constant function. The rule that follows: if the miscalibration is smooth and roughly sigmoidal across the whole range, use Platt; if it is irregular or concentrated in particular bands, isotonic is the more natural fit, provided the target band has enough labelled outcomes not to overfit. For boosted decision trees, a study by Niculescu-Mizil and Caruana found both methods very effective at mapping predictions to calibrated probabilities.
Fraud models add a complication: class imbalance. In the BBVA dataset, fewer than 0.002% of transactions were fraudulent, and one response is to undersample legitimate traffic. A model trained on undersampled data generally overestimates true outcome probabilities. A 2024 analysis showed that traditional Platt scaling is generally a poor choice after undersampling, while a version fitted to the logit of the base model’s predictions works well, and a logistic GAM can improve it further. The same authors note that boosted trees and random forests push estimates towards 0.5, which is the case where traditional Platt scaling can still work well. Their practical advice is to compare the candidate calibrators, prefer negative log score over Brier score on real data, and estimate the final performance on an additional dataset that played no part in choosing the method.
Recalibrate per segment or globally?
Platt scaling and isotonic regression both assume probabilities should be corrected the same way across the whole input space, an assumption one study hypothesises can produce poor calibration where behaviour differs by region. Probability calibration trees, for example, learn separate calibration models in different regions and fall back to a global Platt model when that performs better. The working rule: calibrate per segment only when reliability diagrams split by segment diverge and each segment has enough matured outcomes to fit its own calibrator, and otherwise calibrate globally. Per-segment calibration on a thin new category trades one kind of error for overfitting.
Whichever method you choose, fit the calibrator on data independent of the training data. Calibrating on the classifier’s own training output produces probabilities closer to 0 and 1 than they should be.
Proving the fix worked
A reliability diagram bins test cases by predicted probability and plots each bin’s average prediction against the observed positive rate. Well-calibrated points fall near the diagonal. Expected calibration error (ECE) summarises the same gap as a weighted average across bins. Accept the fix only when ECE inside the problem band and overall both fall below an agreed tolerance on held-out data, segment by segment, and no other band gets worse than it was before.
Two limits stay in place after a successful fix. Calibration changes what the scores mean, not how well the model ranks fraud above legitimate traffic, so a discrimination problem survives it. Calibration is also the property that degrades under concept drift, so any monitor that estimates performance by assuming calibration holds will overestimate performance exactly when it matters.
Can we replay a quarter of transactions through Kafka without touching production?
Yes. A shadow scorer can read the same Kafka topic as production through its own consumer group, so its offsets and replays stay out of the production consumer group. In shadow mode, the parallel scorer receives the same transaction data as the live one, but its decision does not affect the payment flow, as DEUNA’s documentation describes for its anti-fraud product. The pipeline design for running a candidate threshold beside live scoring is covered in more depth in shadow-testing fraud thresholds in a real-time scoring pipeline. The rules for the replay itself are short:
- Give the replay its own consumer group ID and never reuse the production group’s.
- Start from a timestamp, using offset lookups per partition to seek to the first record at or after it.
- Keep the replay window inside the topic’s retention period.
- Make processing idempotent so replays are handled safely.
- Turn off auto-commit, which commits offsets on a timer whether or not records were processed.
- Use read_committed isolation if producers use Kafka transactions, so rolled-back records never reach the scorer.
- If you write to a replay topic, preserve each record’s original timestamp.
- Where resetting offsets near production is unacceptable, replay from a mirror topic kept for that purpose.
Does compressing time break velocity-based features?
That depends on whose clock the velocity features use. A stream processor can move through weeks of event time in seconds by fast-forwarding through historic data in a Kafka topic, and event-time windows still produce correct, consistent results when reprocessing history, provided all the data has arrived. Uber reported that replaying a topic in an order that does not resemble the original caused inaccuracies in event-time windowing and watermarking. So if transactions-per-minute and similar features are computed from transaction timestamps in an event-time pipeline that preserves order, you can compress. If they are computed from processing time or wall-clock arrival, that guarantee does not apply.
Where the throughput ceiling comes from
The ceiling has to be measured by load-testing the shadow scorer at increasing speed-up factors until consumer lag stops recovering. Two structural limits shape it: adding consumers beyond the partition count does not help, and lag has to be watched per partition, because topic-level averages hide a single hot partition. Uber also found that replaying days of data into shared Kafka infrastructure all at once could overwhelm it and cause lag. The shadow cluster needs its own capacity, or the replay itself becomes the bottleneck you end up measuring.
The replay window also has to be long enough for labels to mature. Stripe described a model that looked better in offline tests and early production results, then fell below its predecessor as more fraud reports arrived, forcing a rollback. Observation windows of 120 days or more are common for evaluating fraud models in payment systems. Size the shadow comparison with the power calculation above, and judge its false-positive rate only on cohorts whose labels have had time to mature.
What monitoring would have caught the spike before analysts did?
The monitoring that catches a new-category spike before analysts do watches score distribution, feature drift and alert rate per merchant segment, on a schedule, against recalibration triggers agreed in advance. TechnoLynx builds this kind of standing monitoring for production models as its Production AI Monitoring Harness. The first step is agreeing which segments, signals and trigger levels it watches, because every later alert depends on those choices.
Supervisory guidance already expects this structure. The Federal Reserve’s SR 11-7 guidance treats ongoing monitoring as checking whether changes in products, clients or market conditions require adjustment, redevelopment or replacement of a model. It also says outcomes that fall consistently outside a bank’s predetermined thresholds of acceptability warrant adjustment, recalibration or redevelopment. The PRA’s SS1/23 asks firms to define acceptable-performance thresholds and the criteria for deciding when a breach requires recalibration or redevelopment, and to set monitoring frequency by model tier. For AML specifically, the 2021 US interagency statement accepts that BSA/AML models may need quick adjustments as criminal behaviour changes, and that less material changes can be made without full revalidation where policy allows. TechnoLynx’s production AI reliability work produces drift detection, regression suites and release-readiness reviews.
Three design choices make the process routine and not a fire drill:
- Segment-level, multi-signal monitors. The IEEE-CIS study puts the practical minimum at one score-distribution monitor, one feature-drift detector and one uncertainty monitor. Its companion paper concludes that teams relying on PSI alone are monitoring for one type of degradation and stay blind to the others.
- Graduated alerts tied to responses. Instead of a bare PSI alarm, a breach can trigger a documented escalation (more manual review, a conservative policy switch, model committee review or rollback) depending on severity and decision type.
- Scheduled ground-truth checks. Label-free monitors cannot see concept drift with stable inputs. The same study suggests verifying against ground truth at intervals shorter than 30 days instead of waiting 120 to 180 days for full label maturity, if its calibration generalises.
If you onboard new merchant categories regularly, make the arrival of a new category code itself a trigger: open a segment monitor with its own baseline and a provisional threshold under heightened review. A periodic manual review is enough only where such onboarding is rare. A brand-new segment has too little data for reliable drift statistics at first, so its early weeks need closer human review whatever the monitors say. Designing those trigger levels so they reflect the cost of each error is covered in more detail in setting cost-weighted drift thresholds and retraining triggers.
Who can set up drift monitoring that catches the next merchant-category spike?
TechnoLynx builds eval, drift and slice-metric monitoring harnesses for production AI models in finance. Building in-house makes sense when your team owns the scoring pipeline, the labels and the time to maintain monitors. A specialist helps when monitoring keeps losing priority to feature work, or the same spike keeps recurring. When you choose a partner, ask what their monitoring cannot see, concept drift with stable inputs for a start. Then check whether the deliverable is an alert-to-action process or just a dashboard.
The engagement has limits. It builds reliability artefacts, not model accuracy, and a dashboard alone is not the deliverable. Independent validation has to come from people with no stake in the model’s development or approval. If you run a vendor’s fraud model, the bank remains responsible and should obtain enough information to understand how it performs.
What a signed-off threshold record has to contain
A signed-off threshold record binds the approval to an immutable version of the threshold, with its rationale, validation evidence, prior values and named approvers, so that any later decision made under it can be reconstructed. An approval that cannot be tied to the exact version reviewed is not defensible. For AML transaction monitoring in New York, NYDFS Part 504 requires risk-based detection thresholds, understandable documentation of scenarios, assumptions, parameters and thresholds, and ongoing analysis of whether those thresholds stay relevant. The wider question of turning risk tolerance into a cutoff that survives scrutiny belongs to defensible, auditable AI thresholds under regulatory scrutiny. The checklist below covers the record itself.
| Record element | What it holds |
|---|---|
| Immutable version ID | A version identifier, not a reusable filename, plus a frozen snapshot or hash of the approved configuration |
| Old and new values | The prior threshold and calibration parameters, the new ones, who changed them and why |
| Rationale and diagnosis | The justification for the adjustment, recorded with its reasons in the model inventory |
| Validation evidence | The significance test, reliability diagrams and ECE by band and segment, and shadow-replay results on matured labels |
| Approvers | Named approvers with the authority your governance actually assigns, through defined approval committees or individuals for material changes |
| Event trail | Stage, action, outcome, timestamp with time zone, the actor’s stable account ID and their role at the time |
| Change governance | Evidence that the change was defined, managed, controlled, reported and audited under policy |
Versioning makes recalibrations traceable only if every deployed threshold maps to exactly one approved record, and the record matches what is actually running, not what was meant to be deployed. Approvers have to be the approval authorities your governance actually assigns. Retention also has a regulatory floor in some places: Part 504 requires institutions to keep the records supporting their annual compliance finding for five years.
When a customer disputes a declined transaction or a frozen account, the record has to answer one question: which model version, score, threshold version and approval were in force at that moment? A good test is to pick one completed approval and reconstruct it without asking anyone involved: retrieve the full record, verify its version chain and produce a readable export. If that works, the earlier steps in this process have left their evidence behind. If it does not, the threshold may well be right, but you cannot yet show that it is.
