To validate a new AI fraud detection pipeline before full rollout, replay historical alerts with confirmed outcomes through it in shadow mode, next to the incumbent system, and compare its decisions with what actually happened before it is allowed to act on a customer. Synthetic traffic supports that replay but does not replace it. Choose the generator with an explicit rule based on data volume, scenario diversity and engineering risk, use the simplest option that passes, and treat any metric earned on synthetic data as a pipeline test, not as its production calibration. If the pipeline has components that depend on language, score each language on its own.
Shadow-mode replay of historical fraud alerts, as a procedure
A shadow replay answers one question: if this candidate had been live during a known period, what would it have caught, missed and wrongly flagged compared with the incumbent?
- Freeze a time-ordered replay set. Take historical alerts and transactions with confirmed outcomes and keep them in chronological order. The ChequeMark authors (arXiv, 2026) point out that a random train-test split lets training data include fraud that happened after some test cases, which breaks the conditions of real deployment. They describe a sequential split as the more rigorous test of how a model handles new fraud tactics.
- Run the candidate beside the incumbent on identical inputs. Record both systems’ scores and decisions, and let neither one trigger an action.
- Compare at the operating threshold, not only on ranking metrics. Count catches, misses, false positives and every case where the two systems disagree.
- Add synthetic traffic for scenarios the history lacks. ChequeMark’s synthetic-data checks inject high-velocity bursts, dormant-then-burst behaviour, seasonal spikes and edge cases such as a domestic-currency account receiving cheques from foreign institutions, specifically to test robustness.
- Tune thresholds on real outcomes. As a next step, the same paper recommends threshold tuning and policy calibration through retrospective validation against confirmed real-world outcomes.
- Write down the promotion and retraining triggers before go-live. As one example, ChequeMark proposes a retraining alert when the 30-day average PR-AUC falls more than 5% below baseline, or when a Kolmogorov–Smirnov test on key features gives p < 0.01.
The ChequeMark authors (2026) state plainly where synthetic data stops being useful. It suits end-to-end pipeline testing, feature-engineering validation, benchmarking under controlled class imbalance and rule stress testing, but final threshold tuning and policy decisions need governed real data under operational conditions. Their own caveat explains why: tree-based models can learn the rules that generated the synthetic data, which inflates the reported metrics. Read synthetic-data metrics as an upper bound.
The replay report has value even if the candidate never ships. It is a dated artefact showing where the new design beats the old one and where it falls short, and it gives the team something usable before a rollout decision has been made. Once the replay passes, the next stage is running the same comparison continuously on live traffic. That stage is covered in shadow-testing fraud thresholds in a real-time scoring pipeline.
Generative model, parametric simulator or off-the-shelf tool: which synthetic fraud traffic fits?
The choice between a generative model, a statistical simulator and an off-the-shelf tool for synthetic fraud test traffic is a make-or-buy decision, scored here on three criteria. Use a generative model only when there is plenty of real data, the scenarios need complex dependence structure, and it measurably beats a simpler baseline on the task you care about. In every other case, use the simpler option.
TechnoLynx offers the LLM Selection Pack for teams that want this choice made as an explicit, re-runnable comparison scored on their own tasks.
This rule rests on a tutorial by Cheng, Zhu and Xie (arXiv, 2026). In their tutorial, generative models are most useful when the relevant distribution involves complex dependence, conditioning, tail behaviour or high-dimensional structure, and when training samples are abundant. Even then, they should be used only if they improve task-level metrics compared with simpler baselines. The same authors note that in low-dimensional, data-limited or well-structured problems, empirical distributions, parametric models, kernel density estimation or the bootstrap may be simpler, more transparent and equally effective.
Scoring the three options
| Option | Choose it when | What breaks it |
|---|---|---|
| Off-the-shelf simulator or library | You need schema-valid traffic to exercise the plumbing quickly, and the tool’s domain is close to yours. PaySim, for example, is an agent-based mobile money simulator whose second version was built from aggregated transaction logs of a real service. | Domain mismatch. Lightweight tools such as Faker generate large volumes quickly but do not model correlations. |
| Parametric or calibrated statistical simulator | Real data is limited, the scenarios can be written as rules, or you need rare scenarios by design. Model-based simulators encode domain knowledge and can generate rare or extreme cases that are absent from the data. | Realism is limited to what the rules capture, and J.P. Morgan AI Research (2024) describes a trade-off between model accuracy and simplicity. |
| Generative model (CTGAN, TVAE, Bayesian network, diffusion) | Data is abundant, the feature space is wide, and the model beats the simulator on your task metric. | Behavioural fraud signals, minority-class collapse and scaling, all covered below. |
In an intrusion-detection study (arXiv, 2026), a model trained only on CTGAN-generated data scored F1-macro 0.96 on real CICIDS2017 traffic (78 features), against 0.97 for the model trained on real data. The same approach dropped to 0.52 on UNSW-NB15 (45 features) and 0.24 on Kitsune (7 features). The authors put the practical boundary at about 30 numeric features. That is network traffic, not payments, so take it as a signal about feature width rather than a threshold to apply to fraud data. The same study found that mixing real and synthetic data without controlling class proportions lowered F1-macro to 0.87. Augmentation needs the same parameterised discipline as purely synthetic data.
The best generator also depends on the metric. A study of synthetic training for fraud detection (arXiv, 2024) found that AUROC and recall favoured neural-network generators, F1 and precision favoured Bayesian-network generators, and strict interpretability constraints favoured Bayesian networks. The pipeline’s operating metric therefore has to be fixed before the comparison starts.
Why behavioural fraud signals rarely survive generation
A 2026 benchmark of tabular generators on fraud data showed that statistical fidelity and train-on-synthetic, test-on-real (TSTR) scores do not predict whether behavioural signals survive. CTGAN had the second-highest TSTR AUROC (0.798) yet the worst graph-motif score on the Amazon FDB dataset (99.7 times the real-data noise floor). GaussianCopula had the lowest AUROC (0.523) and a better motif score on that dataset. On IEEE-CIS timing, burst and velocity patterns, the best generator, TVAE with conditional sampling, still measured 24.4 times worse than real-data sampling variability.
The mechanism is structural. Row-independent generators produce each row with no memory of other rows, so they cannot reproduce shared device or IP structure across accounts. They also destroy the burst fingerprint, where a short gap between events is followed by another short gap. The authors conclude that velocity-rule calibration, fraud-ring detection and sequence-level anomaly detection should not use these generators as a drop-in substitute for real data.
Two failures show up in a fraud-rate check before any evaluation. Unconditional TVAE sampling collapsed a 3.5% fraud rate to about 0.03%. In the 2024 fraud study, GaussianCopula and CopulaGAN produced no fraudulent transactions at all. Checking the fraud rate of generated data is cheap, so it belongs at the start of any comparison. If a generative model does earn its place, TechnoLynx’s Generative & Agentic AI R&D practice builds and tunes it.
When the simple simulator wins, and how to defend that
A parametric simulator is good enough when the decision depends mainly on low-dimensional summaries, or when a generative candidate fails to beat it on the task metric. To a steering committee, a thesis examiner or a model-risk reviewer, “the simulator was enough” is a finding and not a gap. It only counts as one if the comparison was set up before the result was known.
Writing on how to publish null results says the methods section has to show the result is meaningful and not an artefact of a flawed design. It recommends an a priori power analysis showing the study could have detected a meaningful effect if one existed. An ICML 2024 position paper makes the same argument: negative results have value when the experimental design was sound and could discriminate well enough to give confident results. In practice, the criteria, metric, minimum detectable difference and sample size should be written down before the generative candidate is trained. Otherwise the simpler choice is a default, not a result. The conclusion also only holds for the workload it was tested on, so a different fraud product with graph-shaped signals needs the comparison run again.
Can we break the quality score out by language so one bad language isn’t averaged away?
Yes. If the candidate pipeline includes a language component, for example a model that drafts alert narratives or classifies customer messages in six languages, its shadow-replay output should be scored, sample-gated and given a threshold per language rather than reduced to one blended number. TechnoLynx builds per-language slice metrics through the Production AI Monitoring Harness, its eval, regression, drift and slice-metric harness.
A blended score can hide a failure because the errors in one language average out across the others. In medical imaging, Oakden-Rayner et al. (2019) found that subgroups hidden inside an overall score can show relative performance differences of more than 20%. In multilingual evaluation, the EMNLP 2025 work on language disparities notes that collapsing performance into a single number undermines fairness when no single model leads in every language. There is a second reason to set thresholds per language. The XQ-MEval authors (arXiv, 2026) found that systems of equal quality receive different metric scores in different languages, and that scores fall at different rates in different languages as quality drops. A single shared threshold therefore means something different in each language.
What is the minimum sample size per language before a score can be trusted?
The minimum depends on the variance of your metric and the smallest change you need to detect, so it should be derived for each language. Two published anchors give a sense of scale. Miller (Anthropic, 2024) works through an example with fictional but reasonable parameters in which detecting a 3-point difference with 80% power and a 5% false-positive rate requires about 969 independent items, and recommends at least 1,000. Applied to six languages, that is roughly 6,000 scored items per comparison window if every language needs its own verdict (6 × 1,000). NannyML’s documentation estimates that 625 observations keep sampling-driven variation in accuracy below 0.02 standard deviation in its example, and recommends 500 for univariate drift detection.
Translated test sets need one more correction. If the same item appears in all six languages, the items are not independent, and clustered standard errors are required. In Miller’s real-world example the cluster adjustment was as large as 3X. If you skip it, every language looks more precise than it is.
The structure that follows from this is: segment the evaluation harness output by language, hold any alert until a language reaches its minimum sample for the window, and compare each language against its own baseline and threshold. Low-traffic languages may rarely reach the minimum. Herlihy et al. (2024) show that ordinary stratified estimates are much less accurate for small subgroups, and propose structured regression as a way to get usable estimates for them. Six independent alerts also multiply the chances of a false alarm. A 2026 study found that under continuous monitoring across many features and time windows, even detectors with nominal false-positive rates can alert frequently because of random variation. Base each threshold on that language’s baseline variance and recalibrate it when the traffic mix or content type changes. The same reasoning applies to other buckets; see per-bucket accuracy gates and why average metrics hide regressions.
What would it take to add per-language drift tracking to the dashboard?
Adding per-language and per-slice drift tracking is a change to the monitoring data model. It is not a new quality model. Predictions and ground-truth labels have to carry language and slice identifiers (channel or whatever unit you make decisions about), and the aggregation layer has to group by those keys before it computes precision, recall and drift. TechnoLynx would take this in six steps. First, confirm which slices actually drive decisions. Then review the current logging schema for missing tags, design the extended metrics schema, and implement it in the pipeline and dashboard. Next, validate the new metrics against a labelled holdout for each language. Finally, deploy scheduled per-language drift checks.
Drift tracking per language also needs a baseline distribution for each language and a scheduled comparison against it, using a population stability index or a Kolmogorov–Smirnov test. Monitoring platforms impose their own limits. Snowflake’s ML Observability documentation, for example, requires baseline data for drift calculation (adding it later means dropping and recreating the monitor), accepts only string categorical segment columns, and allows a maximum of five segment columns per monitor.
Two limits are worth stating before anyone commits to this. If past logs did not record language or slice identifiers, history before that point cannot be broken down by them. The change also adds visibility without improving model accuracy. If the only question is overall system health, the blended metric may be enough, and the engineering cost of the breakdown is justified only when decisions such as retraining or per-language thresholds depend on it.
Who can build per-language monitoring for our fraud replay pipeline?
TechnoLynx builds the monitoring layer that turns shadow-replay output into per-language and per-slice quality and drift signals, with sample gates and thresholds that each team can act on. Whether to build it in-house depends on whether your team already owns the logging schema and has time to keep thresholds calibrated. If both are true, an internal build is reasonable. If neither is, an outside build is the better fit. When choosing a partner, ask how they derive per-language sample sizes and how they would handle translated, clustered test items. The Production AI Monitoring Harness is the engagement for this.
This engagement does not suit teams that need regulatory sign-off on a fraud model, because that requires an independent validator with no part in building the model. It also does not suit teams that only need overall health reporting, where the existing blended metric may already be enough.
What the replay report should already contain
The two parts of this article come together in one artefact: a replay report that gives results per scenario and per language, each with its sample count and confidence interval, and labels synthetic-traffic results as upper bounds. A report built that way answers both the build-versus-simulate question and the “is any language failing” question with the same evidence. It also shows which of those answers still depend on real outcomes that have not arrived yet.
FAQ
Is 100 a big enough sample size?
Rarely, for a quality score you plan to act on. Detecting a 3-point difference with 80% power needs about 1,000 independent items in Miller’s (2024) worked example.
What is the concept of a make-or-buy decision?
A make-or-buy decision weighs building a capability internally against acquiring it from outside, on cost, risk, control and time to value. For synthetic fraud traffic, a third option, a simpler technique, belongs in the comparison too.
