An online feature store is worth running when three things are true at once: several models consume overlapping features, the serving path needs values the request does not carry, and more than one team writes feature logic. Below that line, one versioned transformation package imported by both training and serving, with a test asserting identical output, gives you the parity you need at a fraction of the operating cost. One feature definition feeds an offline store that keeps every timestamped value for training sets, and an online store that keeps only the latest value per key for low-latency lookups. An as-of join at training time makes sure no row sees a value from its own future.
Do we need an online feature store at all, or will a shared library do?
One practitioner guide sums up what a feature store buys in two parts: reuse of feature definitions across models and teams, and symmetry between the offline and online paths. If a programme has neither, “the component is overhead with a good reputation”. Another practitioner text puts the threshold in nearly the same terms: multiple models, genuine point-in-time needs, and a serving latency budget that warehouse queries cannot meet.
Google’s Rules of Machine Learning recommend reusing code between the training and serving pipelines wherever possible. The Hopsworks SIGMOD 2024 paper suggests a versioned Python package as one way to keep on-demand transformations consistent across pipelines. Point-in-time lookups need an offline store that retains the history of feature values, not just the latest value.
Two partial setups are also supported. Starting with only the offline store and point-in-time-correct training sets removes leakage before you serve a single feature online. The online store only earns its place once a real-time serving requirement exists. The reverse also exists: SageMaker Feature Store lets a feature group be offline only, online only, or both. A Feast push source can drop its batch source when features are only needed online, embeddings being the example given.
Which setup fits your situation
| Your situation | What is enough |
|---|---|
| Fewer than the three adoption conditions in place | Shared versioned transformation package plus a parity test on a fixed fixture |
| Training needs historical reconstruction, no live request path | Offline store with point-in-time joins; no online store yet |
| Values only ever read live and never used to build training sets | Online-only feature group or push source without a batch source |
| Overlapping features across many models, a live path needing values the request lacks, several teams writing features | Full store: one definition, offline and online stores, materialisation into both |
What is an online feature store, and how does it differ from the offline store?
The two stores hold different shapes of data. AWS’s SageMaker documentation states that the online store keeps only the record with the latest event time per key, while the offline store keeps all records as a historical look-up. The online store is built for low-millisecond reads and high-throughput writes behind real-time predictions. The offline store serves exploration, model training and batch predictions. The offline store may hold years of history across billions of rows, while the online store holds one row per entity per feature view.
The practical consequence is easy to miss: the online store does not keep history. Databricks documents that publishing a time-series feature table pushes a snapshot of the latest values per primary key, and that the online store supports primary key lookup but not point-in-time lookup. Even the time-series designation, which publishes all historical rows, is described as an exact-match lookup on a key, “not a point-in-time (as-of) or temporal range query”. Training sets come from the offline store, which keeps the history point-in-time joins need.
The Hopsworks paper (SIGMOD 2024) states that all existing commercial and open-source feature stores follow this dual-database design, because no single hybrid transactional/analytical database had yet been shown to handle both workloads. Tacnode, a feature store vendor, argues the market is moving towards a unified model where computation and serving sit in one boundary, while acknowledging the dual-store model is the more mature of the two.
What does point-in-time correctness mean, and how do as-of joins stop leakage?
Point-in-time correctness means every feature value in a training row could have been known when the label’s event happened. For each label at time T, the as-of join takes the most recent feature value whose event timestamp is strictly earlier than T, and never one stamped after it. Two further conditions make the rule hold. The timestamp must be event time, when the fact became true in the world, rather than ingest time. There must also be a stated maximum lookback, so a feature that stopped updating months ago does not pass a stale value off as current.
In Feast, that lookback is the TTL. The Feast documentation stresses that it is measured relative to each row’s timestamp in the entity dataframe, not relative to when the query runs. Databricks exposes the same idea as lookback_window, whose default of None uses every feature value regardless of age. That default deserves an explicit override.
The sources disagree on one detail. Precision Federal says strictly earlier, while the Feast-oriented and Databricks documentation describe “at or before” or “not later than” the label timestamp. We would hold to the strict rule wherever a feature and a label can share a timestamp, for example when the event that produced the label also updated the feature.
Getting this wrong inflates offline metrics. In a worked example published by Cloudmash, joining today’s feature snapshot to every training row scored 0.945 offline but 0.669 when served with real values.
Backtests that only see what the vendor had published at the time
A backtest must use only the information that existed on each simulated day. One point-in-time data guide gives the case: a company reports quarterly EPS of $1.00 and restates it to $0.85 two months later. A non-point-in-time database shows $0.85 for that quarter forever, while a point-in-time database shows $1.00 for the two months before the restatement. The rule for the backtest is to use, on any test date, the latest version published on or before that date.
That means every vendor value needs the date it became known alongside the value itself. The Azure ML team’s geo-distributed feature store paper (arXiv, 2023) builds this into the record itself, with an event timestamp and a creation timestamp that is always later than the event timestamp. Feast also tracks a created timestamp. By default its point-in-time join constrains only the event timestamp and uses the created timestamp to deduplicate. An optional filter adds created_timestamp <= entity_timestamp, which keeps backfilled values out of training data.
A point-in-time feed keeps every version of a number with the date it became known. A non-point-in-time feed shows only the restated value forever.
The backtest runs as-of joins against the offline history, and the live strategy reads the latest value from the online store, both through one feature definition. Divergence between backtest and production then comes from bugs rather than from separate data paths.
One definition feeding the nightly training job and the Kafka inference path
A feature definition is written once and applied to both stores, so the pipeline code does not split and only the destinations differ. Published designs show streaming paths writing to both stores. Uber’s Michelangelo publishes metrics to Kafka, runs Samza streaming jobs to compute aggregates, writes them to Cassandra for serving and logs them back to HDFS for training. Feast push sources push feature values to the online and offline stores in real time. Feast also supports streaming feature views over Kafka and Kinesis, with Redis, DynamoDB and Bigtable as online backends. SageMaker accepts Amazon MSK or Kinesis as streaming sources feeding the online store.
The batch side has to compute the same windows as the stream. In AWS’s fraud example, the batch job computes a 10-minute lookback over historical transactions so the training set stays consistent with the 10-minute streaming window used at inference. The streaming SQL is kept consistent with the query that builds the training set. Parity breaks the moment streaming and batch transforms become separate code with, say, different missing-value handling or window boundaries. As of January 2022, Chip Huyen noted that Tecton and Feast handled streaming feature computation only for data needing no joins. Check the current behaviour of whichever version you deploy.
How do streaming features reach the online store without adding serving latency?
Keeping feature computation off the request path keeps it out of user-facing latency. Near-real-time features are precomputed asynchronously by a stream processor, so their computation latency does not add to user-facing latency, at the cost of staleness in the order of seconds. Two different numbers matter here. One is read latency on the request path. The other is freshness, how long ago the value was materialised.
In Feast’s feature-server tuning, multi-worker configuration was the most impactful change, and precomputed feature vectors took single-row p99 below 2 ms. On freshness, Databricks reports that streaming aggregations from Kafka reach its online store with 200 ms p99 latency.
Freshness also depends on how writes are acknowledged. The Feast documentation warns that asynchronous writes can leave the online store stale. It recommends synchronous writes for critical operations such as calculating loan amounts.
How do we stop a hindsight-only field reaching the online feature view?
Point-in-time safety should be enforced by the schema and by CI, not by developer convention. Every feature definition should carry a declared availability time. The store should generate as-of joins from that declaration, and an automated gate should reject any feature whose underlying data postdates its label timestamp.
TechnoLynx does this work by first auditing existing feature code for places where prediction-time and hindsight-only fields are mixed, then defining the point-in-time contract and building the as-of join logic and CI checks that enforce it, as part of its MLOps engineering practice; the audit is the first step. After that, the checks are validated against known leakage scenarios, such as future SCADA or weather data and post-settlement fields, and wired into the production pipeline’s CI/CD.
Prevention by construction
The Azure ML feature store paper states a query rule designed to prevent data leakage. Given an observation at time ts0, the store looks only at feature values from the past of ts0, and finds the nearest one while “considering the expected delay of source and feature data”. A declared delay per feature is what we would use to turn “available after end-of-day settlement” into a property the join enforces. An intraday label row simply cannot match a settlement field stamped later.
TechnoLynx would treat the hindsight-only annotation as a data contract, for example an availability-time field on each feature. Making the annotation binding takes a lint or CI check at feature-definition time. Feast’s mechanism does the join correctly only if the team states the availability timestamp correctly. A wrong timestamp passes straight through a correct join.
Detection in CI, and what it cannot see
One small open-source leakage-checking library states that leakage detection “is a screen, not a proof”. It finds copies, proxies, identifiers and post-outcome timestamps, but “cannot see leakage that lives in how the features were computed”. A revised or corrected historical value can slip past such a screen, which is why the knowledge-time handling above is needed.
Parity is the second thing to test. A feature store guarantees schema parity but not freshness parity, and freshness skew goes undetected. One failure described in a practitioner wiki is weekend jobs that do not run: the model serves stale features while distribution checks calibrated on weekdays stay quiet. Materialisation timestamps should be monitored as a first-class SLO.
A numeric parity check works like this:
- Run the same feature definition through the batch and online paths on identical input, and compare outputs within a floating-point tolerance.
- Log the feature vectors actually served in production, as Google’s Rules of Machine Learning recommend, even if only for a fraction of requests.
- Recompute the logged vectors offline from the same definition at the same timestamps. Precision Federal targets 100% exact match on categorical and identifier features, relative difference under 1e-6 on floats, and treats anything below 99.9% as a defect.
- Check feature pairs as well as single features, since two individually valid features can be inconsistent together.
- Compare feature values, not only predictions, because shadow serving that compares predictions hides the root cause.
Spark offline, C++ online: parity across two runtimes
When training features are computed in Spark and inference features in C++, there are two options. Either both sides run one computation path, or the team maintains two implementations and proves them equal with parity tests on every release. Implementing feature logic independently for batch training and for low-latency services in different languages is a known source of diverging feature values. Small differences in library versions or edge-case handling are enough.
Embedded C++ inference against a Spark training pipeline
Uber’s approach within the JVM was a DSL, a subset of Scala, whose expressions are part of the model configuration and are applied identically at training and prediction time. Where it is ruled out, we would keep a dual implementation and a frozen set of golden fixtures. Both implementations must reproduce those fixtures before each firmware release, because a later revision can reintroduce drift that a one-time check never sees. Expect float-level differences from precision and summation order, which is why the tolerance has to be written down rather than eyeballed.
Python backtest and a low-latency C++ intraday service
For trading, reading both sides from one computation removes skew from code divergence. The research backtest reads as-of values from the offline store, and the C++ service reads precomputed values from the online store, so there is one computation path with two readers. Client access in C++ is available on at least one stack: the Hopsworks paper describes its RonDB online store accepting native C++ and Java API clients alongside MySQL and REST. For point-in-time joins against tick data at scale, push the join into the warehouse engine rather than running a pandas merge that runs out of memory. If exchange time is confused with capture or ingest time, point-in-time joins reintroduce leakage through the back door.
Feast, Databricks, SageMaker or a custom Redis/DynamoDB build
Choose by who computes the features, how fresh each feature must be and how strict your point-in-time contract is, rather than by vendor name. TechnoLynx matches the architecture to your freshness and backfill needs, starting by characterising data volume, sensor cardinality, freshness targets and backfill requirements.
| Option | How it runs both paths | What you still own |
|---|---|---|
| Feast | Thin, modular layer over your storage: offline on Parquet, BigQuery, Snowflake, Redshift or a lakehouse; online on Redis, DynamoDB, Cassandra or Postgres; push sources write to both stores | Feature computation, since Feast stores and serves precomputed features from external Spark or Airflow pipelines |
| Databricks Feature Store | Point-in-time joins offline; publish_table syncs to the online store in TRIGGERED, CONTINUOUS or SNAPSHOT mode; Unity Catalog required |
Workflow design around one sync per online table at a time; no point-in-time lookup online |
| SageMaker Feature Store | Feature groups online, offline or both; append-only Parquet history in S3; Standard, Standard_V2 or InMemory online tiers | External compute for features; InMemory tier is online-only with no offline replication and a default 50 GiB maximum |
| Tecton | Owns definitions, compute, offline and online stores and serving; streaming features in seconds | Platform cost; SystemOverflow estimates $50,000 to $200,000 a year |
| Custom Redis/DynamoDB with Kafka and Flink | Whatever you build; Redis as an in-memory store gives sub-millisecond reads in AWS’s ElastiCache reference design | The as-of join, the registry, and materialisation that must merge every batch into both stores or eventual consistency breaks |
SystemOverflow also estimates that a self-run Feast deployment needs two to four full-time engineers to maintain pipelines, monitoring and data quality. Treat both of its figures as rough planning numbers rather than measurements. Treat vendor benchmarks the same way. The Hopsworks paper, written by the vendor, reports 11, 10 and 17 times the point-in-time join throughput of SageMaker, Vertex and Databricks respectively. That is a vendor result on its own setup, not a forecast for your telemetry volume.
For sensor telemetry with sub-minute freshness and historical backfill, we would use the split Uber and AWS publish for their own systems. Streaming ingestion feeds the online store, and a batch path feeds history for training. AWS’s fraud example deliberately keeps a nightly batch group separate from a 10-minute streaming group, because streaming every feature adds complexity and can degrade data quality. Precision Federal’s rule follows from that: decide streaming per feature, build the batch path first, declare a freshness SLO for each feature, and stream only those the batch path cannot meet. A managed store may still not express your exact availability-time contract, in which case a thin custom enforcement layer sits on top of whichever product you pick.
Moving existing pipelines onto one store
Migration should be incremental. Start with the features two or more models share, move those definitions into the registry, and prove parity against the existing implementations before any traffic switches. Precision Federal warns that moving 200 features at once opens a long window where two systems disagree and nobody can tell which one is right. Whether consolidating per-region pipelines justifies that effort at all is a separate cost question, and it is worth answering before the first feature moves.
Keep the online side lean. Each materialised feature view is a recurring compute and storage cost and another thing that can go stale, so materialise only what an online model actually reads. Treat the store as a consistency mechanism with numbers attached. Name the targets for parity, leakage, freshness, latency and coverage, then choose the lightest architecture that meets them.
