Teams evaluating a vector database usually treat Milvus as one black box: push in embeddings, run a similarity search, hope recall and latency hold as the collection grows. That mental model breaks at production scale, and it breaks in a specific, predictable way. The single-node deployment that felt fast during a proof of concept stalls once ingestion and query traffic compete for the same resources — and no amount of index tuning fixes a topology problem. The expert view starts one level down, from the architecture. Milvus is not a monolithic process that stores vectors and answers queries in the same place. It is a distributed system that deliberately separates compute from storage and splits responsibility across four kinds of layers: coordinators, a proxy tier, worker nodes, and an object-storage backend. Understanding those layers up front is the difference between a demo and a system that survives billions of vectors. What problem does the layered architecture actually solve? The naive assumption is that a vector database is “a database with an ANN index bolted on.” If that were true, you would scale it the way you scale a traditional database — bigger box, more RAM, faster disk. Vector workloads do not cooperate with that model. Two things pull in opposite directions inside a vector store. Ingestion wants to write and index new embeddings, which is memory- and CPU-heavy and bursty. Search wants stable, low-latency reads across the whole collection, ideally undisturbed by whatever ingestion is doing. On a single node those two workloads share the same CPU, the same memory, and the same page cache — so a heavy re-index run degrades query latency exactly when you can least afford it. Milvus solves this by refusing to keep the two coupled. Compute is stateless and disposable; durable state lives in object storage and a message log. That is the structural decision everything else follows from, and it is the reason the architecture matters more than any single index choice. The four layers, and what each one owns Milvus organizes itself into four functional tiers. The cleanest way to reason about a deployment is to ask, for any given bottleneck, which layer is saturated — because each scales on a different axis. Layer Components Owns Scales with Access Proxy Client connections, request routing, result aggregation Concurrent client load Coordination Root, Data, Query, Index coordinators Metadata, task scheduling, load balancing Cluster size / segment count Worker Query nodes, Data nodes, Index nodes Search execution, ingestion, index building The specific workload that is hot Storage Meta store (etcd), log broker (Pulsar/Kafka), object store (S3/MinIO) Durable state and streaming data Data volume and retention The proxy is the stateless front door. It validates requests, routes them, and merges partial results returned from the worker nodes. Because it holds no durable state, you scale it horizontally to absorb connection concurrency without touching anything below. The coordinators are the brain. Milvus splits coordination by concern — a root coordinator handles global metadata and timestamps, a data coordinator manages segment allocation and flushing, a query coordinator handles load balancing of searchable segments, and an index coordinator schedules index builds. They make decisions; they do not sit on the data path for individual queries. The worker nodes are where compute happens, and they are separated by job. Query nodes hold searchable segments in memory and execute similarity search. Data nodes consume the write log and persist data into segments. Index nodes build indexes as an asynchronous background job. Crucially, these are independent node types — you add query nodes to serve more search traffic without adding a single data node. The storage layer is the source of truth. Metadata lives in etcd, streaming writes flow through a log broker such as Pulsar or Kafka, and the actual vector segments and index files live in object storage like S3 or MinIO. Worker nodes are, in effect, caches over this durable tier. Why does Milvus separate compute from storage? Because it lets you right-size each layer independently instead of over-provisioning one monolithic node to satisfy the worst-case of every workload at once. Consider what happens when compute and storage are fused. To hold a growing collection in memory you buy larger nodes; to survive a node failure you replicate the whole thing including the storage; to re-index you pay with query latency because the index build runs on the same box that is serving reads. Every dimension of scaling is entangled with every other. The separated design decouples them. Query nodes can be scaled to match read concurrency and recall targets. Data nodes scale with ingest throughput. Object storage grows independently and cheaply, and — because the durable copy already lives there — a failed query node is not a data-loss event. It reloads its segments from object storage and rejoins. This is the same architectural instinct that shows up across modern data infrastructure: keep the durable state in one cheap, reliable place and treat compute as elastic and replaceable. A comparable philosophy of designing systems for the failure case rather than the happy path underlies our discussion of the risks that surface when AI systems are treated as black boxes — the architecture is a governance decision as much as a performance one. How the layers cooperate on a single query Following one request end to end makes the separation concrete. A client sends a search request to a proxy. The proxy validates it and assigns a timestamp so the query sees a consistent view of the data. The proxy fans the query out to the query nodes that hold the relevant segments. Each query node runs the approximate-nearest-neighbour search over its in-memory segments. Query nodes also scan a small buffer of freshly written, not-yet-flushed data streaming from the log broker, so recent inserts are searchable without waiting for a flush. Each query node returns its top-k partial results. The proxy merges them into a single ranked result set and returns it to the client. Nothing on that read path touches the coordinators, and nothing writes to object storage. The write path is separate: inserts land in the log broker, data nodes consume the log and flush segments to object storage, and index nodes later build indexes on those segments asynchronously. Reads and writes travel different roads, which is exactly why a heavy ingest run does not have to poison query latency. How does the architecture scale at large collection sizes? Data in Milvus is chunked into segments — the unit of both storage and search. A collection of hundreds of millions or billions of vectors is many segments, distributed across query nodes and balanced by the query coordinator. Scaling read capacity means adding query nodes so segments are spread thinner and more searches run in parallel; scaling write capacity means adding data nodes so ingestion keeps up. This independence is the operationally important property. In configurations we have reasoned through for high-ingest workloads, the failure mode of a monolithic deployment is almost always the same: one node is simultaneously the ingest bottleneck and the query bottleneck, and tuning one starves the other (observed pattern across infrastructure engagements; not a published benchmark). The layered design removes the coupling, so recall targets and latency targets can be defended by scaling the layer that is actually saturated. A quick diagnostic for which layer to scale: Query latency climbs under concurrent load, ingest is quiet → add query nodes; the search tier is saturated. Ingest backlog grows, search is fine → add data nodes; the write tier is behind. New data takes too long to become searchable → index nodes or flush cadence, not the search tier. Metadata operations (create collection, load, release) are slow → the coordination/etcd tier, not the workers. Getting the diagnosis right depends on measuring the right layer, which is the same discipline we describe for how explainability works in practice: the observable symptom rarely names its own cause. When is Milvus a good fit — and when is it overkill? The separated, distributed architecture is a liability at small scale. All that coordination — etcd, a log broker, coordinators, multiple worker types — is operational overhead you pay for whether or not you have the data volume to justify it. Milvus offers a lightweight embedded mode (Milvus Lite) precisely because the full cluster is the wrong tool for a prototype or a small collection. Situation Fit Prototype, single machine, < ~1M vectors Embedded / Lite mode, or a simpler store Steady collection, modest concurrency, no independent scaling need Consider a lighter-weight vector store Hundreds of millions to billions of vectors Distributed Milvus earns its complexity Ingest and query loads that must scale independently Distributed Milvus is the intended case Strict recall + latency SLOs under concurrent load Distributed Milvus, sized per layer The honest framing: adopt the full architecture when you need what it buys — independent scaling, durability decoupled from compute, and read/write isolation. If you do not need those, the complexity is a cost with no matching benefit. This decision sits inside a wider set of infrastructure trade-offs we cover across our [machine learning practice](our service packs), where the recurring lesson is that architecture choices are only justified by the failure modes they prevent. FAQ How should you think about Milvus architecture in practice? Milvus is a distributed system, not a single process. It separates compute from storage and splits work across a proxy access tier, coordinators, worker nodes, and a durable storage backend. In practice this means reads and writes travel separate paths and each layer scales on its own axis, so a heavy ingest run does not have to degrade query latency. What are the main components of Milvus — coordinators, proxy, worker nodes, and storage? The proxy is a stateless front door that routes requests and merges results. Coordinators (root, data, query, index) handle metadata, scheduling, and load balancing without sitting on the per-query data path. Worker nodes are split by job — query nodes serve search, data nodes handle ingestion, index nodes build indexes. Storage combines etcd for metadata, a log broker like Pulsar or Kafka for streaming writes, and object storage like S3 or MinIO for durable segments. Why does Milvus separate compute from storage, and what does that enable? Because it lets each layer be right-sized independently instead of over-provisioning one monolithic node. Durable state lives in object storage while compute nodes act as elastic caches, so query nodes scale with read demand, data nodes scale with ingest, and a failed worker reloads from storage rather than causing data loss. How does Milvus scale query and data nodes independently at large collection sizes? Data is chunked into segments distributed across query nodes and balanced by the query coordinator. Adding query nodes spreads segments thinner and runs more searches in parallel; adding data nodes raises ingest capacity. Because the two node types are separate, you scale read and write capacity independently as the collection grows past hundreds of millions of vectors. How does the architecture affect vector search recall and latency under concurrent load? Read/write isolation keeps ingestion and indexing off the search path, so query latency stays stable even during heavy writes. Spreading segments across more query nodes increases search parallelism, which lets recall and latency targets be defended by scaling the specific layer that is saturated rather than the whole system. When is Milvus a good fit versus a lighter-weight or embedded vector store? The full distributed architecture pays off at hundreds of millions to billions of vectors, or when ingest and query loads must scale independently under strict SLOs. For prototypes or small collections the coordination overhead is a cost with no matching benefit, which is why an embedded mode like Milvus Lite exists. The layers are not incidental implementation detail — they are the contract the system offers you. Before adopting Milvus, the useful question is not “is it fast?” but “which of my workloads will saturate first, and can I scale that layer without dragging the others down?” If the answer names a single layer, the architecture is working as designed; if scaling one layer forces you to scale all of them, you have deployed the topology wrong, not chosen the wrong database.
