You log the feature values your serving path actually produces, then compare each feature’s distribution against the training baseline. The Population Stability Index (PSI) compares binned distributions against the training baseline, and the Kolmogorov-Smirnov (KS) test suits continuous features where tail shifts matter. A breach is a warning that calls for investigation, not proof that the model has degraded. A pass does not prove that training and serving compute features the same way either.
TechnoLynx builds this check as part of the Production AI Monitoring Harness. The engagement starts by mapping your feature store and your training and serving paths to find where leakage or staleness could get in. It then adds a point-in-time replay test in CI and a scheduled PSI/KS job in production, with per-feature thresholds calibrated by backtesting the model’s precision and recall against past drift episodes.
How do you check for data drift between training data and production data?
Start by logging what the model actually saw. Google’s Rules of Machine Learning recommends saving the features used at serving time and piping them to a log, even if only for a small fraction of traffic, so that serving and training can be checked against each other. The ML Test Score paper (Breck et al., Google) calls the training/serving skew test perhaps the most important test and also the least implemented.
With serving logs in place, the comparison has two levels. Where serving examples carry identifiers, feature values for the same example should be identical at training and serving time. Where they don’t, you compare distribution statistics (minimum, maximum, mean, fraction missing) and tune the alert thresholds to keep false positives low. The baseline should be the exact feature values used to train the model version currently deployed. In streaming systems the tests run on sliding windows, for example the last hour of events against baseline statistics.
Three details decide whether the check finds anything useful:
- Slice the comparison. Global aggregate metrics hide skew on small, high-value segments, so scope monitoring by country, device or segment. For fraud, segment by fraud type, because card-not-present fraud, account takeover and card testing have different detection economics.
- Test nulls separately. Evidently’s documentation notes that its default drift tests do not react to more empty values, and it recommends separate tests on the share of nulls.
- Layer it. Data quality checks come first (null rates, type mismatches, out-of-range values), then input distribution monitoring, then prediction drift.
Data drift, concept drift and training-serving skew need different fixes
All three can show the same symptom: offline metrics up, online metrics flat. Their causes and fixes differ.
| Fault | What has changed | Fix |
|---|---|---|
| Training-serving skew | Training and serving get different views of the data from the same pipeline (an engineering fault) | Shared transformation code, served-feature logging, point-in-time joins |
| Data drift (covariate shift) | The input distribution P(X) has moved, for example a fraud model trained on transactions averaging $50 starts seeing mostly high-value purchases | Retrain on recent data, or reweight if past data is still informative |
| Concept drift | The relationship P(Y given X) has changed, so the same inputs now call for different predictions | Retrain on fresh data, sometimes change the architecture |
The MLflow guide puts the key distinction plainly: skew produces systematic errors that look like drift but need pipeline fixes, not retraining. Retraining on data produced by a serving bug teaches the model the bug as a pattern, and later predictions reproduce it. Fraud models have one more failure route: like ranking and spam systems, a fraud intervention model shapes the data it is later trained on, and sharing feature code does not fix that. Terminology adds confusion too. Google’s broader use of “training-serving skew” includes genuine distribution change, which should not be misdiagnosed as a code bug.
PSI or KS for feature drift: which test fits which feature
PSI bins a variable and compares the share of observations in each bin between the training and serving populations. As one fraud-operations guide writes it, PSI = Σ (Actual% − Expected%) × ln(Actual% / Expected%) across feature buckets. It was originally developed for credit scorecard monitoring, which is why it is so common in financial services. Because PSI is the sum of two KL divergences, it is unbounded and can blow up when distributions barely overlap. If a bin is empty in one population, the value goes to infinity. An October 2025 arXiv paper on credit risk scorecards argues that PSI is preferable where missing real drift costs more than a false alarm.
KS measures the maximum distance between two cumulative distribution functions. It works well for continuous features and is sensitive to shifts in the tails, which is where fraud signals and anomalies often sit. It is non-parametric, so it needs no assumed form for the data. It does need ordered values, which makes it awkward for categorical attributes, while PSI only needs the proportions per level. Report the KS distance as a magnitude rather than a yes/no test result, as a 2025 arXiv study on distribution shift does. In that study, a KS distance of only 0.02 already meant the test distribution was 12% different from the training distribution.
| Feature | Test to start with | Why | Watch for |
|---|---|---|---|
| Continuous behavioural feature (amount, velocity) | KS distance, or PSI on quantile bins | Tail-sensitive, no distribution assumption | KS flags trivial shifts at large sample sizes |
| Categorical feature (merchant category, country) | PSI, or L-infinity distance per value | KS needs ordered groups; L-infinity names the value that moved | Empty bins make PSI infinite |
| Binary feature | Proportion difference test | Evidently’s default for features with two or fewer values | |
| Missingness | Separate share-of-nulls test | Default drift tests ignore nulls | Null spikes after upstream schema changes |
Evidently picks its default test from column type, reference size and number of unique values. TensorFlow Data Validation’s L-infinity comparator only detects skew on categorical features, while its Jensen-Shannon comparator covers both numeric and categorical.
Why default PSI and p-value thresholds mislead a fraud model
The usual PSI bands are: below 0.1 no significant shift, 0.1 to 0.25 moderate shift worth investigating, 0.25 and above significant shift needing action. Those bands come from credit scoring model monitoring and were adopted widely. They are not a published standard for payments. The ranges people quote differ, too. The same credit risk paper treats PSI below 0.02 as stable and above 0.10 as significant change. Google Cloud’s model monitoring alerts on every feature at a distance of 0.3 unless told otherwise, and Evidently’s defaults flag drift at a p-value of 0.05 or less, or a distance of 0.1 or more for distance metrics.
Generic numbers fail for four reasons:
- The feature matters. The MLflow guide notes that a PSI of 0.15 on a stable demographic feature may be noise, while the same value on a real-time behavioural feature may need immediate investigation.
- Volatility differs. High-volatility features can tolerate a higher PSI, and low-variance features may warrant action at lower values.
- Sample size changes what PSI means. Simulations in the Population Resemblance Statistic paper on arXiv found that fixed PSI thresholds call for rebuilds too often at n=50 even with no shift, and rarely trigger at n=500 even with moderate shifts.
- Significance tests over-fire at ML scale. Breck, Polyzotis et al. (MLSys 2019) replaced 0.01% of 100 million N(0,1) points with N(0,2) samples, and a chi-square test still alerted in 7 of 10 trials.
Credit risk practice gives a working principle: the greater the loss exposure if the model is wrong, the smaller the tolerated deviation should be. The tolerance should also shrink as sample size grows. Interpretable distances make thresholds easier to set. An L-infinity rule such as “allow changes of up to 1% per value” is easy to reason about, and every alert names a culprit value.
Calibrating per-feature thresholds: a procedure
- Measure each feature’s normal day-to-day distance first, since some drift between days is always expected.
- Rank features by how much a shift in each one moves the fraud model’s precision and recall at its operating threshold, using a backtest.
- Replay historical drift episodes and record which threshold would have fired before the decision metrics moved.
- Set each threshold above the noise floor and tighten it where loss exposure is higher.
- Size each window so the metric is statistically meaningful. Rare fraud-positive segments need longer windows than the bulk of traffic.
- Write down the rationale for each threshold and review thresholds on a fixed cycle.
Putting a money value on false alarms and missed drift is a separate design problem, covered in setting cost-weighted drift thresholds and retraining triggers.
Scheduling data drift detection and deciding what a failed check triggers
Cadence follows how critical the feature is. One production guide recommends computing PSI daily for important features and tracking the trend, since a gradual rise points to creeping drift and a sudden jump points to an abrupt change. One fraud-operations guide calls for PSI on primary features monthly at minimum, and daily or real-time for critical features where the tooling supports it. Run PSI on the score distribution as well as the inputs. A score shift with no feature-level change may mean fraud patterns that the model’s features are not capturing. Wiring these jobs into training and deployment is part of MLOps engineering.
A failed check should open an investigation, not start retraining automatically. The same fraud-operations guide sets investigation at PSI above 0.1 on primary features, and a retraining evaluation at PSI above 0.25 or an AUC decline of more than 2 points over a rolling 90-day holdout. A sustained PSI breach is a reason to prepare an investigation, not to retrain immediately. Population stability measures cannot tell a real population shift from an unstable attribute, so someone has to find the reason. Triage first sorts the breach into skew, drift or concept change, because skew is fixed in the pipeline and drift by retraining.
At scale, a triage queue handles the few features that breach each day, and thresholds are relaxed during a defined warm-up window after each deploy, when cache misses and autoscaling cause short-lived skew. Seasonality shifts and campaign launches cause false positives, so they need dynamic thresholds or human review. Who owns the recalibration decision, and which statistical trigger formally starts it, is a governance question for each model. Drift detection is one of the artefacts of production reliability.
With a shared feature store, is a scheduled distribution check enough?
No. A feature store gives you one feature definition with two reads: training reads from the offline store, serving reads from the online store, and both come from the same upstream computation. Point-in-time (as-of) joins make sure each training label only joins feature values known before its timestamp. Feast, for example, scans backward from each entity timestamp up to the feature view’s TTL. With the created-timestamp flag on, it reproduces what the online store would have served at each event time.
Even with all that in place, a distribution check cannot prove parity. A serving feature can be shifted, clipped or wrongly encoded and still produce a similar-looking histogram. If the same production payload, model bundle and feature values score differently in offline replay and in production, a change in the outside world cannot explain the difference. That is implementation skew. Compare features one at a time before looking at aggregates, because one high-impact mismatch can hide inside a harmless overall distribution. Code-path skew that preserves marginal distributions passes schema checks entirely. datarekha gives an example: a production “30-day rolling average” computed over a slightly different window than the training query, so the features looked the same but weren’t.
| Point-in-time parity test | Scheduled PSI/KS check | |
|---|---|---|
| Catches | Temporal leakage, code-path divergence | Statistical drift in live inputs |
| Runs | Before deployment, against replayed serving requests | On a schedule in production |
| Blind to | Population change when both paths agree | Skew that preserves the histogram |
Run both, because each catches a class of failure the other cannot see.
Who can build automated skew and drift checks for our fraud model?
TechnoLynx builds the complete check for a fraud model: a point-in-time replay test that gates releases, a scheduled PSI/KS job on serving snapshots, and per-feature thresholds backtested against the model’s precision and recall. Before deployment, both checks are validated against injected synthetic skew and drift, then run with alerting. When choosing a partner, ask how thresholds will be justified per feature and how the tests are shown to fire on known skew before they go live. The work is delivered as the Production AI Monitoring Harness, starting with a review of your feature store and your training and serving paths.
If you only need standard monitoring, managed options exist: Google Cloud’s model monitoring supports per-feature skew thresholds, Evidently lets you choose PSI, Jensen-Shannon, Wasserstein or custom tests with your own thresholds, and TensorFlow Data Validation detects training-serving skew and day-to-day drift through schema comparators.
Does a feature store with as-of joins remove skew across regional pipelines?
A feature store with as-of joins does not remove every class of skew. Shared definitions and point-in-time joins stop divergent transformations and future leakage. They cannot fix pipelines that run outside the store. TechnoLynx would start by tracing the root cause of each past skew incident against your regional pipeline architecture, to separate what an enforced online/offline contract closes from what needs process change. It would then replay those incidents through the new parity checks to confirm they would have fired.
Separate implementation logic for batch and online paths is perhaps the most frequent cause of skew, and unified feature logic, written once and reused everywhere, is the most effective fix.
Parity should be checked continuously, not just at redeploy, because breaks happen between deploys. Chronon, Airbnb’s open-source feature platform, logs inference queries, its own responses and their timestamps so that online/offline inconsistency can be measured automatically. Its documentation is also frank that 100% consistency is not always possible: streaming features lag by milliseconds to seconds, and batch features refreshed daily are inconsistent around midnight until the upload finishes, which can take minutes or hours. Tolerances therefore have to allow for that timing, and float comparisons need an explicit epsilon. Freshness needs its own alert as well. A feature store guarantees schema parity, not freshness parity, so stale weekend features can slip past distribution checks calibrated on weekdays. Monitor feature age and feature-vector parity failures directly, and compare feature values in shadow mode, since shadow serving that compares only predictions hides the root cause.
Should a pre-deployment replay gate check feature parity before release?
Yes, and it should be paired with continuous per-feature monitoring after release. A replay test takes a sample of recent serving requests, recomputes their features through both the training and serving paths, and compares them element by element across thousands of vectors. TechnoLynx builds this as a release gate wired to the point-in-time feature store. It catalogues the features feeding the model, defines what skew means for each one (distribution, null rate, timing), and backtests the gate against a historical incident to confirm it would have fired in time.
For each sampled request, keep the request ID, prediction timestamp, feature-contract version, model-bundle version, transformed feature vector, missingness indicators and resulting score. A candidate model whose features show fresh skew on the calibration set then blocks rollout. The payoff can be concrete. Breck, Polyzotis et al. report that Google Play found features always missing from serving logs but present in training, and removing that skew raised the app install rate on the main landing page by 2% in an A/B test.
The gate has limits. Offline replay only uses successful requests, so it misses serving failures. Per-feature parity can miss feature pairs that are valid on their own but inconsistent together. If the feature store’s own point-in-time correctness has not been validated, both checks compare against a baseline that is already wrong.
FAQ
What does the p-value represent in a Kolmogorov-Smirnov test?
A small p-value suggests the two distributions are significantly different. With large serving samples, KS detects almost any shift, including ones too small to matter.
