Yes. A real-time scoring pipeline can shadow-test a new fraud threshold before rollout, without waiting for a batch cycle. Shadow testing fraud thresholds means the candidate threshold scores a copy of live (or replayed) traffic and logs the decision it would have made, while the active threshold keeps full control of what customers see. Signals such as alert volume, block rate and how far the two thresholds disagree arrive within minutes. Catch rate is slower. It depends on confirmed fraud labels, and those take days to months to arrive.
Can we shadow-test a new fraud threshold before full rollout?
A streaming fraud pipeline can run a candidate threshold in shadow next to the live one, provided both decisions are logged side by side for every transaction. The vendor definitions agree on the mechanism. In Amazon SageMaker AI a copy of each inference request goes to the shadow variant, and only the production variant’s response goes back to the caller. Azure Machine Learning describes traffic mirroring the same way: the mirrored share is copied to the shadow deployment, and the results clients receive do not change.
Uber’s engineering team describes shadow testing as running the candidate in parallel with production on identical live inputs and logging its outputs for comparison, without affecting user-facing predictions, and reports it as part of more than 75% of its critical online use cases. DoorDash’s anti-fraud team published a similar sequence for fraud models: backtest first, and only once the backtests look consistently good, move to production as shadow traffic at 1% volume.
Each threshold is a cut-off applied to the model’s output: a transaction is flagged when its predicted fraud probability exceeds the chosen value. Applying a second cut-off to an already computed score adds only one comparison per transaction.
For each transaction we would record the transaction ID, the score, the live decision, the shadow decision, a timestamp and enough context (merchant, channel, amount band) to slice the disagreements later. The log Stripe keeps for its counterfactual holdback method is a useful baseline: transaction ID, original score, propensity, whether the original policy would have blocked, and the action actually taken. One staged-deployment study frames the comparison as a divergence between old and new outputs on the same input stream, used to find regressions and drift.
Shadow deployment also catches faults that sandbox evaluation misses. In one 2026 study of staged upgrades to AI components, using embodied agents as its case, sandbox evaluation caught most faulty candidates, and the faults caught only in shadow were latency-inflation cases that stayed inside the sandbox’s success envelope but showed up when compared against the live version. The same study notes that shadow is the only stage that can detect live-context divergence, because that divergence is defined by comparison with the active version.
Why separating feature computation from scoring lets you test without a batch cycle
Decoupling features from scoring lets any number of decision rules read one computed score. Streamkap describes the common real-time fraud pattern: account balance, recent transaction amounts and device fingerprint sit as keys in Redis, change data capture keeps them current, and the scorer reads them in under a millisecond, with no batch job in the hot path. In its latency profile, a database change reaches the cache in roughly 100-150 ms. One open-source Kafka scoring pipeline gives a related reason for putting Kafka between producers and the scorer: it decouples the two, absorbs spikes as consumer lag, and keeps the stream replayable.
The result is that feedback latency on alert volume is bounded by the pipeline’s end-to-end latency and the refresh rate of your dashboards, not by a reporting batch. Uber reports detecting prediction drift and surfacing anomalies within minutes using a streaming Apache Flink job. Adyen’s managed risk experiments refresh their results every 15 minutes.
When scoring is batch-only, a shadow test runs at the batch cadence. The test is still worth running. Just don’t expect a near-real-time loop from an architecture that doesn’t have one.
Shadowing has a cost. A mirrored stream is real extra load on the scorer, the feature store reads and the logging sink. Azure caps mirrored traffic at 50 percent specifically to limit the effect on an endpoint’s bandwidth quota, which defaults to 5 MBps. We would give it its own capacity, or mirror only a sample.
Which threshold signals arrive in minutes and which take months
The table below sets out what each signal can tell you and when.
| Signal | When it is available | What it tells you |
|---|---|---|
| Alert or block volume under each threshold | Within the pipeline’s refresh interval; Adyen refreshes experiment results every 15 minutes | Workload and customer-friction change, not correctness |
| Decision divergence between live and shadow | As fast as decisions are logged | Where the candidate disagrees, and on which segments |
| Manual investigation outcomes | Minutes to days, per a 2021 e-commerce fraud research agenda | Early, partial labels for reviewed alerts only |
| Fraud notifications via escalation | Days to weeks, per the same source | Labels for missed fraud |
| Card disputes and chargebacks | Issuer decisions usually take 60-75 days, and the full dispute lifecycle 2-3 months, per Stripe’s documentation | Confirmed fraud outcomes, and so catch rate |
Stripe has described leaving a 60-day buffer before the present when validating fraud models, because disputes can take a week to two months to come in. A same-day result has, at most, some manual investigation labels; escalations and disputes take days or longer. Treat anything that claims more as a guess.
Shadow testing versus A/B testing a fraud threshold
A shadow threshold makes no live decisions. An A/B test lets the candidate decide for part of the real traffic, so it affects real customers. Adyen’s documentation says outright that its risk experiments are not simulations: a share of live transactions is randomly assigned to the current profile or the new one and processed on the live account. If you set 20% of traffic, each profile handles 10%.
That difference changes what you can learn. A shadow test leaves customers untouched but inherits the live threshold’s blind spots. When the live threshold blocks a transaction, nobody finds out whether it was fraud, because only approval reveals an outcome and rejection reveals nothing. Blocked transactions carry a suspicion, not a label. In practice this cuts one way. If the candidate is stricter than the live threshold, the extra transactions it would block were approved in reality, so their outcomes eventually arrive and you can score the candidate on them. If the candidate is looser, the transactions it would approve were blocked in reality, and the shadow test can never tell you how many of them were fraud.
An A/B test reports fraud and chargeback rates for the variant arm, at the price of real impact on those transactions. It is also limited to the variants you chose to run, two in a standard A/B test.
Using a holdback to measure what the live threshold blocks
One way to see past a block is a control group: always approve a certain share of transactions. That gives you unbiased labels, but it means deliberately letting some fraud through. The 2021 research agenda suggests using such a control group, or another unbiased source, to optimise the threshold when automated blocking has removed the labels.
Stripe has described a version of this for fraud models. Among transactions the policy would block, let a small random fraction through, for example with probability 0.05, and observe what happens. Precision is then the fraudulent share of those released transactions. A refinement makes the release probability depend on the score, through a propensity function. Many transactions are let through just above the threshold, fewer at higher scores, and almost none at the top, so the evaluation budget goes where the model is least certain. Each allowed transaction is weighted by one over the probability that it was allowed, and precision and recall are computed from those weights.
There are two costs. The estimates are statistical: the more you let through, the narrower the error bars and the higher the fraud cost, which is the exploration-exploitation trade-off. And a steep propensity curve gives rare released transactions very large weights. When Stripe monitored precision and recall over a rolling 90-day window, a single high-weight transaction could move the numbers sharply. The benefit is considerable: once the holdback data exists, it can evaluate arbitrarily many candidate thresholds and policies, not just the two an A/B test compares. For threshold work that is the main attraction, because the question is rarely “50 or 62?” and usually “where on the curve?”.
How long should a shadow test of a new fraud threshold run?
The run length depends on what the threshold has to prove. Volume, divergence and latency checks settle quickly. Catch rate cannot settle until the labels have matured. For the fast checks, we would run long enough to cover the traffic’s normal weekly rhythm (weekend and payday patterns included) before trusting the volume shift. For catch rate, vendor guidance and dispute timelines set a floor. Adyen recommends running a risk experiment for at least one month to get accurate fraud and chargeback rates, and the dispute timelines above mean some fraud labels are still arriving two to three months later.
Running longer is not free. Fraud is adversarial, and fraudsters adapt their behaviour over time to evade detection, so features that worked can stop working. In one 2026 forward-in-time evaluation, average precision fell from 0.925 on the validation period to 0.537 on the later test period. The longer the run, the more any difference you see mixes the threshold’s effect with drift in the traffic. Delayed labels make this worse: until labels arrive, a change in the fraud relationship is only visible when it comes with a change in the input distribution.
Pass and fail criteria should be written down before the run starts: the acceptable alert-volume band, the maximum divergence on named segments, the latency budget, and the catch-rate test to apply once labels mature. This is the same discipline as choosing thresholds on a validation set so that the test set is not biased in your favour.
How to read false positives when comparing a shadow threshold against the live one
Early in a shadow run, an extra alert from the candidate has no label yet, because labels arrive with a delay. Count it as a false positive only once its label window has passed.
The aim is to minimise the expected cost of errors, not the number of false positives. False positives add friction for genuine customers or waste investigators’ time, while false negatives let fraud through. Cost-sensitive evaluation puts a price on all four cells of the confusion matrix, for example the lost income from a blocked transaction against the cost of a fraud. Raising a threshold cuts false positives and analyst workload while missing more fraud, so the choice should follow operational capacity and risk tolerance rather than a fixed cut-off. Where investigators can review only so many alerts, precision among the top-K alerts describes the queue they actually work.
A replayed open-source example shows the scale at stake. In that Kafka scoring pipeline, an early version of the routing rules sent about 59% of all traffic to review. Recalibrated rules brought the replay’s review rate to 12.2%. The same replay routed 21 of 24 labelled frauds to review (87.5% capture) and reported the three misses openly. It also states that it is a replay, not a live stream. Two lessons follow. Check alert volume first: the 59% version produced a queue no team can work. And a catch rate measured on two dozen labelled frauds is a direction, not a result.
Watch the timing of any false-positive spike as well. Platform changes, such as a new payment process, cause drift, and novelty detectors flag the resulting outliers alongside new fraud types. If the live and shadow thresholds diverge sharply right after a release, check the release before blaming the threshold. Measuring catch rate properly also needs a held-out labelled set that reflects your real fraud mix and class imbalance, which is a separate piece of work from the shadow run itself.
Replaying last quarter’s transactions through a shadow scorer the same day
A same-day replay starts from Kafka’s replayable log, provided it still retains the window you need. A separate consumer group reads the production topics into an isolated scorer running the candidate configuration, writes only to its own sink, and its decisions are compared offline against production. TechnoLynx can build this as a same-day replay gate on your existing Kafka and scoring service, starting by agreeing what pass or fail means and how much history the replay must cover. TechnoLynx’s AI engineering for finance team can build fraud-scoring replay tests and release gates of this kind.
Kafka suits this because it is a durable, ordered log that can be replayed to rebuild state. Separate consumer groups each read the same topics independently, so a shadow group does not move production offsets. If the retention period does not cover the quarter, the log must first be backfilled into a replay topic.
Three caveats decide whether the replay can be trusted:
- Strictly read-only towards production. The shadow scorer must not write decisions, fire alerts or update shared state. If it does, the test contaminates the system it is meant to protect.
- Feature state must be rebuilt, not borrowed. The same open-source pipeline warns that replaying an old event after newer ones can count the newer events in its velocity window, because the window is computed by event time while the state is current. A replay that reads the live feature cache scores last quarter’s transactions with today’s history.
- Labels limit what same-day can judge. The replay finishes in a day, but the labels for the replayed period mature on the dispute timeline, 2-3 months for the full lifecycle.
Synthetic data for edge cases and the fuller replay design are covered in our piece on shadow-mode replay and synthetic data strategy for fraud pipeline validation.
Treat the threshold change as a release
A threshold change goes to production and can regress silently in the ways production AI does, through alert fatigue or missed fraud. Shadow testing serves as a pre-release gate for that change. It confirms quickly that the new cut-off produces a workload you can staff, at the latency you can afford, and that it disagrees with the old one only where you expected. It cannot confirm, on day one, that the change catches more fraud. That answer waits on the label clock, so promote on volume evidence and schedule a catch-rate review for when the labels mature.
