A team asks which database they should use for their machine learning platform, and the question already contains the mistake. There is rarely one database. The warehouse, the vector store, and the big-data operational store each answer a different question, and picking one as the “primary” store usually means the other two jobs get done badly inside it — slowly, expensively, or both. That is the failure we see most often when a data platform meant to serve ML starts to buckle. It is not usually a scaling problem. It is a layering problem: a single system asked to be a training-data lake, an analytics warehouse, and a low-latency retrieval index all at once. Three jobs that look like one The confusion is understandable. All three layers store data, all three are queried, and all three are marketed with overlapping language. But the access patterns pull in opposite directions, and hardware and cost structures follow those patterns. A data warehouse — BigQuery, Snowflake, Redshift, or a lakehouse layer like Databricks over Delta — is built for scan-heavy analytical queries over columnar data. It reads large slices of many rows, aggregates, and returns. This is where feature engineering, label auditing, and training-set assembly live. Warehouses are optimised for throughput on cold-ish data, not for millisecond point lookups. A vector store — Milvus, pgvector, Pinecone, Qdrant, or a FAISS-backed service — indexes high-dimensional embeddings for approximate nearest-neighbour search. It answers “which stored items are most similar to this query vector,” which is the retrieval half of retrieval-augmented generation and the backbone of semantic search and recommendation candidate generation. The dominant cost here is index memory and recall-vs-latency tuning, not scan throughput. A big-data operational database — Cassandra, HBase, DynamoDB, or a distributed key-value/wide-column store — serves high-volume, low-latency reads and writes on individual keys. This is the online feature store’s natural home: the model needs a user’s current features at inference time in single-digit milliseconds, at request volume, with no tolerance for a warehouse’s query planner. The core claim is simple. These are three distinct storage jobs — analytical scan, similarity search, and low-latency point access — and a mature ML platform runs all three, not one system pretending to cover all three. When you see one system carrying all the load, you are usually looking at accumulated technical debt rather than an elegant simplification. Why does putting embeddings in your warehouse eventually hurt? It is tempting, early on, to store embeddings as array columns in the warehouse you already have and compute cosine similarity in SQL. It works in a demo. It stops working when the corpus grows, because a warehouse has no ANN index — every similarity query becomes a full or near-full scan of the embedding table. Latency climbs with corpus size, and because warehouses bill by data scanned or by compute-seconds, so does the bill. In configurations we have worked with, the crossover point where a dedicated ANN index (HNSW or IVF-based, as implemented in Milvus or pgvector) beats in-warehouse brute force arrives well before a million vectors for interactive latency targets — this is an observed pattern from retrieval workloads, not a published benchmark, and the exact threshold depends on dimension count and recall target. The mechanism is not subtle: brute-force similarity is linear in corpus size; a graph or quantised index is sublinear. Once retrieval is on a hot path, that difference is the whole game. The reverse mistake also happens. Teams stand up a vector store and start treating it as their document database, storing full payloads and querying by metadata filters it was never designed to serve efficiently. Vector engines are getting better at hybrid filtering, but a wide-column or relational store still wins for structured, filter-heavy access. We cover the licensing and deployment side of one common choice in our breakdown of whether Milvus is open source and how its governance works. Which layer does your workload actually need? Use the access pattern, not the data type, to decide. The table below maps the question you are answering to the layer built for it. You need to… Access pattern Layer Representative tech Dominant cost Assemble training sets, audit labels, run feature engineering Scan-heavy analytical Data warehouse / lakehouse BigQuery, Snowflake, Databricks Scan throughput / compute-seconds Find items similar to a query embedding Approximate nearest-neighbour Vector store Milvus, pgvector, Pinecone, Qdrant Index memory, recall-vs-latency tuning Serve a model’s features at inference time Low-latency point read Big-data operational DB Cassandra, DynamoDB, HBase Read latency at request volume Track offline feature history for training Append-heavy analytical Warehouse / object store Delta, Parquet on S3/GCS Storage + scan The row that trips people up is the feature store, because it spans two layers at once. Offline features (the historical values used to build training sets) live in analytical storage; online features (the current values served at inference) live in the operational database. A feature store like Feast or Tecton is not a fourth database — it is a coordination layer that keeps the offline and online copies consistent so that training and serving see the same feature definitions. Skew between those two copies is one of the most common silent causes of a model that trains well and serves badly. The consistency problem nobody scopes for Once you accept that ML data lives across three layers, a new problem appears: keeping them coherent. The embedding in your vector store was generated by a model version; when you re-embed the corpus, the index and the source documents drift out of sync unless the pipeline versions them together. The online feature in your operational store must be computed by the exact transformation used offline, or you get training-serving skew. The warehouse table feeding your training run must be a reproducible snapshot, not a live table someone is still writing to. None of this is exotic, but it is rarely in the initial scope. Teams budget for the storage and forget the plumbing that keeps three stores telling the same story. In practice the pipelines — ingestion, transformation, embedding generation, and the versioning that ties them — are more work than any single database choice. That plumbing is where machine learning storage decisions play out in practice, and where the choice between orchestration tools like Azure Data Factory and Databricks actually gets made. The hardware layer matters here too, though less than teams expect. Vector index builds are compute-heavy and increasingly GPU-accelerated; warehouse scans are I/O- and memory-bound; operational point reads are latency-bound and benefit from data locality. Matching each layer to the right instance class is part of the same reasoning we apply to cloud platform selection more broadly, which we lay out in our AI and data workload selection guide across AWS, Azure, and GCP. A worked example: RAG-backed product search Say you are building semantic product search with a language model answering over your catalogue. Walk the layers. Assume a two-million-item catalogue, embeddings refreshed nightly, and an interactive latency target on the order of 100 milliseconds for retrieval. The warehouse holds the raw catalogue, transaction history, and the pipeline that generates training data for any ranking model — this is analytical, batch, and cost-sensitive to scan volume. The vector store holds the two million product embeddings behind an HNSW index and serves the nearest-neighbour lookup for each query; its sizing is driven by index memory and the recall target you set, not by catalogue byte size. The operational database holds live inventory, price, and per-user personalisation features that the ranking step reads at request time in single-digit milliseconds. Try to collapse this into one store and something breaks. Put it all in the warehouse and retrieval latency blows past the target. Put it all in the vector store and your inventory filters and price lookups fight an engine tuned for similarity. Put it all in the operational database and your nightly embedding pipeline has nowhere efficient to scan from. The three-layer split is not architectural fashion; it is what the access patterns force. FAQ Do I really need three separate databases for ML? Not three products necessarily, but three distinct storage jobs: analytical scan, similarity search, and low-latency point access. A small system can co-locate some of these (pgvector puts vectors inside PostgreSQL, for instance), but the access patterns remain distinct and will eventually pull the layers apart as scale grows. When should embeddings move out of the data warehouse? When similarity search is on an interactive hot path and the corpus is large enough that full-scan cosine similarity misses your latency target — often well before a million vectors, depending on dimension and recall target. That is an observed pattern from retrieval workloads, not a fixed benchmark; measure your own crossover. What is the difference between an online and offline feature store? Offline features are historical values in analytical storage, used to build reproducible training sets. Online features are current values in a low-latency operational database, served at inference time. A feature store coordinates the two so training and serving use identical definitions, preventing training-serving skew. Is a vector store a replacement for a document database? No. Vector stores are optimised for approximate nearest-neighbour search over embeddings, not for structured, filter-heavy queries over full payloads. Hybrid filtering is improving, but a relational or wide-column store still wins when access is dominated by metadata filters rather than similarity. The question worth asking is not “which database,” but “which access patterns does my ML workload actually generate, and does each one have a home built for it?” Answer that honestly and the layering falls out on its own — usually more layers than the first sketch admitted, and each one cheaper for having a clear job. If you are mapping this onto a specific cloud, our overview of AI in cloud and DevOps is the wider frame this sits inside.
