A defensible AML alert sampling methodology for threshold tuning comes down to four decisions. You stratify the alert population, including the activity that fell below the current threshold, so that rare, high-severity segments stay visible. You allocate the review sample by Neyman or risk weighting, with a floor for small strata. You size each stratum from a power calculation for the smallest shift in the missed-suspicious-activity rate you need to catch. And you compare months with a test that matches how you actually look at the data. The stratum boundaries, the shift you are trying to detect and any stratum-specific acceptance threshold all have to be written down before the sample is drawn. That record is what lets a threshold decision survive an examiner’s question about how it was reached.
The quantity worth monitoring is the rate at which suspicious activity goes unreported: alerts closed as non-productive that should have been escalated, and activity that never alerted because it sat just under a threshold.
A stratified sampling design for closed and below-the-line alerts
AML alert QA works best as a stratified random sample drawn from two populations: alerts already closed as non-productive, which are re-reviewed to find missed suspicious activity, and events just below the current threshold, which are sampled to test whether the line is in the right place.
TechnoLynx builds this kind of sampling QA as an engineering deliverable. TechnoLynx starts by agreeing the QA objective and the shift you need to detect, then implements the stratification and power calculations against your real alert distributions, wires the matching test into the validation pipeline, and back-tests the whole procedure on historical alerts before it runs monthly. The work sits within TechnoLynx’s AI governance and trust offering.
One published template exists for the below-the-line half. A SAS paper describes visualising alerted aggregate wire amounts within a customer segment, dividing the alerts between the 60th and 90th percentiles into three tranches, drawing a simple random sample from each tranche and sending the combined sample to the investigative team to label each alert productive or non-productive. The same paper samples below the 60th percentile to show that alerts there are not productive, and calls the iterative process of setting the lower threshold this way “below-the-line” testing. It also recommends that investigators triage below-threshold samples without knowing they are below the line. That blinding matters more than it looks: once a reviewer knows an alert is a test case, the label is no longer a measurement of normal disposition quality.
For the closed-alert half, candidate strata are merchant category, closure reason, customer risk tier and the threshold tranche the alert fell into. Stratification needs that auxiliary information on every unit in the population, not just the sampled ones. Strata pay off when they are internally homogeneous, because all sampling error comes from within-stratum variance and anything pushed into between-stratum differences disappears from the estimator. The same reference warns against blindly optimising stratum boundaries on historic data, since strata should hold up when the population shifts. Keep the cross-classification coarse. Three risk tiers by four closure reasons already makes twelve cells, and each one needs sufficient sample for reliable estimates of its rate and variance.
Neyman allocation in stratified sampling, and where risk weighting overrides it
Neyman allocation gives each stratum a sample in proportion to its size times its standard deviation, n_h = n · N_h S_h / Σ N_k S_k, which minimises the variance of the overall estimate for a fixed total sample. For a binary outcome such as “missed / not missed”, S_h is √(p_h(1 − p_h)). When every stratum has the same standard deviation, the formula reduces to proportional allocation.
There is a catch for AML. A stratum with a miss rate near zero has a small S_h, so Neyman allocation gives it few reviews, even when it is a rare, high-severity segment you most want to watch. Neyman’s objective is the precision of the population-wide estimate, not the ability to detect a shift inside one stratum. The Wikipedia summary of the method lists the practical limits: it needs prior knowledge of stratum standard deviations, the allocations have to be rounded, and very small strata may get too few units for reliable estimation.
Three corrections handle this:
- Full-coverage strata. When an allocation exceeds the stratum’s population, take every unit in that stratum with probability 1. The recursive Neyman algorithm has been proven optimal under the upper bounds n_h ≤ N_h, so capping and redistributing is not a hack.
- A floor. If a stratum would get fewer than 20 to 30 units, move a few from the other strata to it, giving up a little variance-optimality in exchange for a more robust sample and usable stratum variance estimates.
- Risk weighting. Where severity rather than variance should drive the allocation, set a minimum per high-risk stratum from its own power calculation (below) and let Neyman allocate the rest.
When stratum standard deviations are unknown, a pilot sample estimates them. Research on stratified subsampling found that pilot-based stratified sampling lost efficiency with a small pilot, and cites recommendations for a pilot of about half the final sample. We read that as a sensible direction rather than a settled rule. For a monthly programme, previous months’ re-review results can serve as the pilot, provided the alert population has not changed underneath them.
Whatever the allocation, results must be weighted back. The population miss rate is p̂ = Σ (N_h/N) p̂_h, and each reviewed alert in stratum h stands for N_h/n_h alerts. Heavily sampled strata therefore carry small weights per alert. Reporting the raw pooled rate from a disproportionate sample overweights whichever strata you oversampled. For intervals on the weighted estimate, replace the sample size by the effective sample size, the sample size divided by the design effect.
How to determine sample size in stratified sampling for a 2% shift
| Per-stratum sample size depends on four inputs: the baseline miss rate, the minimum detectable difference, the confidence level and the power. Change any one and the number changes, so the calculation has to be rerun for each stratum and each population. The NIST/SEMATECH e-Handbook defines the shift to detect as δ = | p₁ − p₀ | and derives the sample size using the normal approximation to the binomial. For the two-proportion z-test, choosing the sample size is closely tied to choosing the minimum detectable effect. |
Worked example: sizing one stratum for a 2-point rise in the miss rate
Assumptions (inputs, not findings): baseline miss rate p₀ = 5%, the shift to detect is a rise to p₁ = 7% (δ = 0.02), two-sided α = 0.05 (z = 1.96) and power 80% (z = 0.8416).
- The one-sample formula against a known baseline is N′ = [z_{α/2}√(p₀q₀) + z_β√(p₁q₁)]² / δ².
- √(0.05 × 0.95) = 0.2179, and × 1.96 = 0.4272. √(0.07 × 0.93) = 0.2551, and × 0.8416 = 0.2147.
- (0.4272 + 0.2147)² / 0.0004 = 0.4120 / 0.0004 ≈ 1,031.
- Apply the Fleiss, Levin and Paik continuity correction that NIST recommends, N = N′ + 1/δ: 1,031 + 50 = 1,081 re-reviewed alerts.
If the baseline is not treated as known, and you instead compare two monthly samples with a two-proportion z-test, the per-sample requirement roughly doubles. With pooled p̄ = 0.06: [1.96√(2 × 0.0564) + 0.8416√(0.0475 + 0.0651)]² / 0.0004 ≈ 2,213 per month. As a check on the arithmetic, the same formula at p = 0.30 with a 10% relative shift (0.03 absolute) gives about 3,762 per group, or roughly 7,525 in total. That matches the analytic two-proportion z-test requirement of 7,520 reported in the SPRT-z paper (arXiv, 2026).
The baseline assumption moves the answer a lot (same δ, α and power, one-sample formula with continuity correction):
| Assumed baseline miss rate | Detect a rise to | Re-reviews per stratum |
|---|---|---|
| 2% | 4% | 533 |
| 5% | 7% | 1,081 |
| 10% | 12% | 1,906 |
A misestimated baseline sizes the sample wrongly in either direction, which is one more reason to refresh baselines from recent months rather than from a launch-time study.
What does a worked power calculation look like for the smallest alert stratum?
Now take a hypothetical small high-risk stratum of 300 closed alerts a month. Applying the finite population correction to 1,081 gives 1,081 / (1 + 1,080/300) ≈ 235. At that point you are reviewing almost 80% of the stratum, so a census is simpler. Even a census has a ceiling. Treating the month’s 300 alerts as draws from the underlying process, the same formula solved for δ shows that 300 reviews reliably detect a rise from 5% to roughly 8.9%, a shift of about 4 points, not 2. Oversampling cannot rescue a stratum whose population is simply too small for the shift you named.
Two honest responses exist. One is to require the 2-point detection at population level, through the weighted estimate, and accept a coarser per-stratum guarantee. The other is to ask the small stratum a different question: not “has the rate shifted?” but “can we show the rate stays below a tolerance?”. The OCC’s Comptroller’s Handbook on sampling methodologies (2020) sizes exactly that kind of attribute test from a confidence level and a tolerance rate, assuming zero expected exceptions. Its lookup table gives 299 items for a 1% tolerance at 95% confidence and 59 for 5%, independent of population size.
Choosing the test: two-proportion z-test, Wilson intervals or a sequential design
The two-proportion z-test compares this month’s miss rate with a baseline sample’s rate. It is appropriate when observations are independent pass/fail outcomes and samples are large enough for the normal approximation to hold. Independence deserves a second look in re-review, because the test assumes each observation is independent of the others.
The interval matters as much as the test. The standard Wald interval covers poorly when samples are small or proportions are near 0 or 1. A comparative study in complex surveys (JSSAM, 2017) found the Wald interval badly flawed even when the true sampling variance is known, and recommended Wilson or Bayes uniform-prior intervals instead. Clopper-Pearson, the exact interval, is conservative: a rare-event study (arXiv, 2021) found its sample sizes about 40% larger than Wald, Wilson or Agresti-Coull, and the JSSAM authors advise it only where conservative coverage is paramount. The rare-event study also found that the Wilson interval needed the smallest sample to reach its coverage targets.
For low rates, set precision relative to the rate. The same study recommends a relative margin of error, ε/p, between 0.1 and 0.5. At a 5% miss rate, ε_R = 0.5 means a margin of ±2.5 points.
| Situation | Test or interval | Why |
|---|---|---|
| Large strata, rates well away from 0 | Two-proportion z-test | Normal approximation holds |
| Small strata or rates near 0 | Wilson interval (Newcombe method for differences) | Better coverage than Wald |
| Conservative coverage required, e.g. a critical stratum | Clopper-Pearson | Exact, but wider |
| Weighted population estimate | Wilson on effective sample size | Accounts for the design effect |
| Results inspected before the month’s sample is complete | Group sequential test | Bounds false positives under repeated looks |
Should a monthly alert QA use a sequential design when prior spot-check data exists?
A monthly alert QA needs a sequential design only if someone looks at the results before the month’s sample is complete. Spotify’s experimentation team (2023) notes that a group sequential test run with no interim looks is exactly the traditional z-test. If compliance wants weekly reads so that a sharp deterioration triggers action before month end, the choice then turns on whether the maximum sample can be estimated in advance.
When it can, a group sequential test gives the highest power, whether data arrives in batches or as a stream. The cost is that overrunning the planned maximum inflates the false positive rate, while falling short only makes the test conservative. Investigator dispositions arrive in batches, which suits the group sequential test’s design. When the maximum cannot be estimated, always-valid inference guarantees a bounded false positive rate under any stopping rule, at some cost in power.
The SPRT-z variant adds futility stopping as well as efficacy stopping. Across 46 online product experiments, its authors (arXiv, 2026) report a median saving of 45.5% of the fixed-horizon sample, while the worst case used 3.4 times the requirement. They recommend a burn-in before any decision, for example at least 100 per group, and advise against relying on their bias-corrected estimates and intervals when only a few looks are expected. Four weekly looks a month falls in that few-look case, so SPRT-z’s bias-corrected estimates would not be relied on; for monthly AML QA we would reach for a group sequential test first.
Prior spot-check data has two legitimate uses. It can set the baseline and variance inputs for the power calculation, or it can serve as an informative prior in a Bayesian adaptive design. Either use assumes the spot checks reflect current conditions: the same scenarios, thresholds, investigator team and customer mix. Data from before a threshold change biases the priors or boundaries it informs. Whatever you choose, the interim-review schedule and stopping rules have to be fixed in advance, because ad hoc stopping brings back the inflated false positives the design exists to prevent.
Different acceptance thresholds for different alert strata
A stricter or looser acceptance threshold for one stratum is defensible only when a documented reason makes that stratum different: a higher cost of a missed case, a regulatory tolerance specific to that product or customer type, or a materially different base rate. Without that record, a looser threshold for a low-risk stratum looks like threshold-shopping, however reasonable it is.
The OCC handbook sets a comparable discipline for examiners: they must not change the tolerance rate or the confidence level after selecting the sample items, and judgemental rather than random selection invalidates the inference. The New York DFS transaction monitoring rule, Part 504.3, requires pre- and post-implementation testing, documentation of detection scenarios with their assumptions, parameters and thresholds, and ongoing analysis of threshold effectiveness, according to a Davis Polk summary of the final rule. SR 26-2, issued in April 2026, replaces SR 11-7 and the 2021 interagency statement on BSA/AML model risk, and emphasises risk-based model risk management tailored to each institution’s model risk profile. A per-stratum threshold table, with its rationale and the date it was fixed, is the artefact that connects the sampling plan to those expectations.
The baseline moves, so the sampling plan has to be rerun
The weakest point in a monthly plan is the baseline. New scenarios, re-tuned thresholds, a new investigator cohort or a shift in customer mix all change the alert population, and a sample sized against last quarter’s miss rate quietly loses power. Deciding when a change in that rate should trigger re-tuning is a monitoring-policy question, and it is covered in our piece on designing drift triggers and retraining policies from ROC thresholds. Re-tuning thresholds to cut false positives without losing sensitivity is a separate exercise again, and it needs this sampling plan to validate its result.
The re-review label is a measurement too. If two investigators would disagree on the same closed alert, part of the “missed” rate is reviewer noise, and the agreement logic from gage R&R for pass-fail inspection applies before any shift is attributed to the thresholds.
The same design carries over to other QA problems, such as checking extracted contract clauses by contract category or validating a detector across subgroups, with different strata and metrics. Clinical subgroup validation is the exception: the regulatory stakes and the multiplicity across subgroups call for a biostatistician, not a generic formula.
