Replication mode is not a storage checkbox. It is a statement about how much data your AI system may lose and how long it may take to serve traffic again — and the answer is usually different for the feature store than it is for the model registry sitting next to it.
The naive version of this decision runs on vibes. Synchronous sounds safer, so someone enables it globally. Or asynchronous sounds faster, so someone enables that globally. Either way the consequence stays invisible until a failover, a zone loss, or a region outage turns an abstract setting into a real incident. The divergence point between the two modes is the recovery objective, not the storage engine.
What is the difference between asynchronous and synchronous replication?
Synchronous replication acknowledges a write only after the replica has durably accepted it. The commit latency of every write is therefore bounded by the slowest participating replica, plus the network round trip to reach it. In exchange you get a zero-data-loss recovery point: whatever the primary confirmed, the replica has.
Asynchronous replication acknowledges the write locally and ships it onward afterwards. Write latency stays local — the replica’s distance stops mattering to the caller — but a lag window opens between “committed on the primary” and “present on the replica.” Anything inside that window can be lost if the primary disappears before the shipment lands.
Both statements are unremarkable on their own. What matters for AI data serving is that those two costs land on completely different parts of the system. Synchronous replication taxes the write path. Asynchronous replication taxes the correctness of the read path, because reads served from a lagging replica are reads of stale data.
Start from what an inference request actually reads
The useful entry point is not the storage tier. It is the request. For any AI system in production, work through three questions per data store:
- What does a single inference request read from this store?
- How stale can that data be before the prediction becomes wrong rather than merely old?
- What does one lost write cost — a retry, a silently wrong prediction, or a compliance problem?
Answer those and the replication mode usually falls out on its own. An online feature store backing a fraud model reads recent behavioural aggregates; a few seconds of staleness can be the difference between catching a transaction and missing it, so the tolerated lag is genuinely tight. An approximate-nearest-neighbour embedding index rebuilt on a batch cadence tolerates far more lag, because the index is already a snapshot of something older than itself. A model-metadata catalogue — which artefact is promoted, which version serves which traffic slice — writes rarely but must never lose a write, since a lost promotion record leaves the fleet disagreeing about what production is.
That is why a single global setting is the wrong shape of answer. The same system frequently needs different modes for different tables, and treating replication as one platform-wide switch forces the strictest requirement onto every write path, including the ones with no durability requirement at all.
RPO and RTO, stated in units you can test
Recovery point objective is the amount of data loss you accept, expressed in seconds. Recovery time objective is how long you accept being unable to serve, expressed in minutes. Synchronous replication is how you buy an RPO near zero; asynchronous replication is how you buy local write latency and accepts an RPO equal to your worst observed lag.
The measurable outcome of this whole exercise is a stated and tested pair of objectives per data store rather than one assumed globally. Three metrics make it testable:
- p99 write latency added by synchronous commit, measured separately across availability zones and across regions. These are different orders of magnitude and conflating them is where most bad decisions start.
- The observed replication lag distribution under peak write load — not the median, and not the idle-hours number. Lag is worst exactly when the system is busiest, which is also when the failover you are planning for is most likely.
- The proportion of inference reads served from a replica within the tolerated staleness window. If you cannot compute this figure, you do not currently know whether your asynchronous setup is safe.
Note what is not on that list: a vendor’s advertised durability tier. Durability guarantees describe the storage system’s behaviour, not your model’s tolerance for stale features.
Replication-mode decision table for AI data stores
| Data store | Typical read pattern | Loss cost of one write | Recommended mode | What to monitor |
|---|---|---|---|---|
| Online feature store | Per-request, latency-critical | Wrong prediction, hard to detect | Synchronous within region; asynchronous across regions | Lag distribution at peak; share of reads inside staleness window |
| Offline feature / training store | Batch, non-serving | Recoverable by re-derivation | Asynchronous | Lag at end of ingest window; backfill success |
| Embedding / vector index | Read-heavy, snapshot-based | Low — index is rebuilt | Asynchronous | Index build age; time since last successful sync |
| Model registry / metadata catalogue | Rare reads, control-plane | High — fleet disagrees on active version | Synchronous | Commit failure rate; failover promotion path |
| Prediction / audit log | Write-heavy, append-only | Depends on regulatory scope | Asynchronous unless audit-bound | Lag ceiling; gap detection on sequence numbers |
| Session / short-lived cache | Per-request | None — reconstructable | Asynchronous or unreplicated | Cache hit rate post-failover |
The table is a starting shape, not a verdict. The regulatory scope of your prediction log, and whether your feature store’s freshness requirement is seconds or minutes, will move rows.
When mixed is right, and where split-brain gets you
The pattern that survives contact with production most often is mixed: synchronous replication within a region across availability zones, asynchronous replication across regions. Intra-zone round trips are cheap enough that synchronous commit is affordable for the stores that need a near-zero RPO, while cross-region synchronous commit imposes a latency floor set by physics and one that every write pays whether it needs the guarantee or not.
The cost of mixed mode is failover complexity. Asynchronous cross-region replication means the secondary is, by construction, behind. Promoting it during a partition while the original primary is still accepting writes is how you get two divergent histories — split-brain — and reconciling divergent feature values after the fact is worse than the outage was. Guard it structurally: quorum-based or fenced promotion rather than manual DNS flips, explicit write-fencing of the demoted primary, and a documented answer to what happens to the writes inside the lag window. Managed services differ sharply here; PostgreSQL streaming replication, Kafka’s acks=all with a minimum in-sync replica count, and Redis replication all express “replicated” with different guarantees, and a Kubernetes operator that automates promotion will do so according to its semantics, not your intentions.
We see the monitoring side skipped more often than the design side. Replication lag and failover readiness belong in the same dashboards and alerting as model latency and accuracy drift, with a rehearsed failover on a schedule — otherwise the first real test of the recovery objective is the incident itself. The broader practice of treating these controls as part of the serving system rather than the storage layer is what we cover under production AI reliability engineering, which goes deeper into how these objectives get instrumented alongside model-level signals. That reliability engineering sits inside the wider cloud security and operations picture mapped out in AI in Cloud Computing: Boosting Power and Security.
Frequently Asked Questions
Asynchronous vs synchronous replication: what is the difference, and which should an AI-data-serving system use? Synchronous replication confirms a write only once a replica has durably accepted it, giving a near-zero data-loss recovery point at the cost of commit latency bounded by the slowest replica. Asynchronous replication confirms locally and ships afterwards, keeping write latency local but opening a lag window in which committed data can be lost. Most AI-serving systems should not pick one globally — they should pick per data store, driven by what an inference request reads and what a lost write costs.
How do RPO and RTO change under each replication mode, and how do you choose targets you can actually test? Synchronous replication is the mechanism for an RPO near zero; asynchronous replication sets your effective RPO to the worst replication lag you actually observe, not the lag you hope for. RTO is largely independent of mode and driven by promotion mechanics. Targets become testable only when you measure lag distribution at peak load and rehearse failover on a schedule.
What does synchronous replication cost in write latency across availability zones versus across regions? The two cases are different orders of magnitude and should never be reasoned about together. Intra-region, cross-zone synchronous commit is usually affordable for stores that need a near-zero RPO; cross-region synchronous commit imposes a latency floor set by the network round trip that every write pays, whether or not that write needs the guarantee. Measure p99 write latency added by synchronous commit separately for each topology before committing to either.
How much replication lag can a feature store, embedding index or model registry tolerate before predictions degrade? The tolerance differs by store, which is the whole argument for per-store modes. An online feature store feeding a fraud model may degrade within seconds of staleness; an embedding index rebuilt on a batch cadence is already a snapshot and tolerates far more; a model registry writes rarely but tolerates no lost write at all, because a missing promotion record leaves the fleet disagreeing about what production is.
When is a mixed strategy correct — synchronous within a region, asynchronous across regions — and how do you avoid split-brain on failover? Mixed is usually correct when some stores need a near-zero RPO but no write path can afford a cross-region commit tax. Split-brain is prevented structurally rather than procedurally: fenced or quorum-based promotion instead of manual DNS changes, explicit write-fencing of the demoted primary, and a documented decision about the writes stranded inside the lag window.
How do you monitor replication lag and failover readiness in a cloud/DevOps pipeline rather than discovering problems during an incident? Put replication lag distribution, staleness-window read coverage, and time-since-last-successful-sync into the same dashboards and alert routes as serving latency, then rehearse promotion on a schedule so the recovery objective is exercised before it is needed. If you cannot currently state what proportion of inference reads are served inside the tolerated staleness window, that number is the first thing to instrument.
Which leaves the question worth asking of your own platform: if the primary region vanished mid-peak right now, which of your data stores could tell you — in seconds — how much they lost?
