The choice between a key-value store and a document store is not a choice between two flavours of NoSQL. It is a single question about whether the database needs to understand the shape of the value it stores. If every lookup arrives with a known key and the value is opaque to the engine, a key-value store is the cheaper and more predictable option. If the application needs secondary indexes, partial updates, or server-side filtering on nested fields, a document store earns its extra cost — and only then.
The naive version of this decision treats the two families as interchangeable and picks whichever managed service the cloud provider markets hardest. Six months later, every read fetches and parses a whole blob so the application can filter on one field. That is not a database failure. It is a data-model failure that the database faithfully executes at scale.
What is a key-value database, and what is it actually good at?
A key-value store maps an opaque key to an opaque value. The engine does not parse the value, does not index inside it, and cannot filter on it. That constraint is the feature: because the engine has no work to do beyond locating a partition and returning bytes, the read path is short and the latency distribution is narrow. Redis, DynamoDB in its simplest usage, and etcd all sit in this family, and all of them reward workloads where the caller already knows the key.
A key-value store is the right default when the access pattern is a single known key and the whole value is wanted every time. Session state fits this exactly: you have a session ID, you want the session. A feature cache keyed by entity ID fits it too — the inference path needs the full feature vector, not a subset of it, and it needs the answer in single-digit milliseconds.
The cost shows up the moment a second access pattern appears. Answering “which sessions belong to this tenant” against a pure key-value store means either maintaining a second key space by hand or scanning. Both are application code you now own and test.
What a document store buys, and what it charges
A document store parses the value. JSON or BSON documents can carry secondary indexes on nested fields, be updated in place field by field, and be filtered server-side so only matching documents cross the network. MongoDB, Couchbase, and Azure Cosmos DB’s document API all trade some read-path simplicity for that query capability.
That trade is worth it when the number of distinct query shapes is greater than one and likely to grow. Inference request logs are the clearest AI-side example we see: the write is a nested document with model version, latency, token counts, and outcome flags, and the reads are ad-hoc — errors by model version last hour, p99 by region, requests where a guardrail fired. Serving those from a key-value store means over-fetching everything and filtering in application memory.
The charge is real, though. Indexes consume write throughput and storage, flexible schemas drift unless validation is enforced somewhere, and per-operation cost in managed document services is typically higher than the key-value equivalent for the same logical read.
Comparison matrix: key-value vs document
| Axis | Key-value store | Document store |
|---|---|---|
| Value visibility to engine | Opaque bytes | Parsed structure (JSON/BSON) |
| Primary access | Get/put by known key | Query by key, index, or field predicate |
| Secondary indexes | Not native; hand-rolled key spaces | Native, on nested fields |
| Partial update | Read–modify–write whole value | Field-level update in place |
| Server-side filtering | None | Yes — reduces bytes on the wire |
| Latency profile | Narrow, predictable | Wider; depends on index selectivity |
| Per-read cost | Lower for full-value fetch | Higher, but avoids over-fetch |
| Schema drift risk | Owned entirely by the application | Managed by validation rules |
| Best fit | One known access pattern | Several evolving query shapes |
Which AI workloads sit where
The useful move is to stop asking “which database should we standardise on” and ask where each dataset actually lives.
Feature caches are key-value work. The key is an entity or request ID, the value is consumed whole, and the latency budget is tight. Session and conversation state is the same shape, with a TTL attached.
Embeddings metadata is the interesting case. The vectors themselves belong in a vector index; the surrounding metadata — source document, tenant, permissions, chunk offsets, ingestion timestamp — is queried by field, not by key, because retrieval needs pre-filtering before or alongside the similarity search. That is document-store shape, or a relational one.
Inference request logs are document-store work when queried interactively, and often belong in an analytical store instead once the retention window grows past a few weeks.
That splits into a short decision rubric:
- Can you name every read query the dataset will serve, and is it exactly one, by key? → key-value.
- Does anything need to filter on a field inside the value, server-side? → document.
- Does the write path update one field of a large value frequently? → document (avoid read–modify–write amplification).
- Is the dominant operation approximate nearest-neighbour search over vectors? → a vector index, with metadata alongside it.
- Do more than two of the above apply to one dataset? → the dataset is probably two datasets.
Where vector databases fit
A vector database is not a third option on the same axis. It is a specialised index for similarity search, and most deployments still need a key-value or document store beside it for metadata, source text, and audit records. Some document stores now ship vector indexes natively, which collapses the operational footprint at the cost of tuning depth. Whether a dedicated vector store is worth a separate system depends on filter complexity and recall requirements, not on corpus size alone. Our parent piece, AI in Cloud Computing: Boosting Power and Security, works through the open-source vector database landscape and the surrounding security trade-offs in more detail than belongs here.
The read-path numbers to watch
This decision is measurable, which is what makes it worth making deliberately rather than by provider default. Three signals tell you whether the model matches the access pattern:
- Bytes read per request. If the application reads a 40 KB document to use 200 bytes of it, the model is wrong for that path — server-side filtering or a narrower key would remove the waste.
- p99 read latency, not mean. Over-fetching shows up in the tail first, because deserialisation cost scales with payload size.
- Count of application-side filters. Every filter in application code that could have been an index predicate is a data-model mismatch you are paying for on every call.
The second cost is migration. Changing the data model after production launch means a dual-write window, a backfill, and a verification pass before the old path can be retired — an order of magnitude more work than the comparison exercise up front. We keep this on the pre-launch checklist for cloud data work precisely because the fix is cheap before traffic and expensive after; the same architectural discipline that shapes our wider engineering practice applies here. Pick the model that is easy to leave: keep the access pattern documented, keep serialisation behind an interface, and do not scatter provider-specific query syntax through the application layer.
None of this settles cleanly when a dataset genuinely has two access patterns of comparable volume. That is the case where we would rather run both stores, accept the duplication, and let the read paths stay simple — but the point at which that becomes cheaper than one compromised store is not something we would claim to know in the abstract.
Frequently Asked Questions
Key-value vs document database: what is the practical difference, and which should you pick for an AI/application workload? The practical difference is whether the database can see inside the value. A key-value store returns opaque bytes for a known key; a document store parses structure, indexes nested fields, and filters server-side. Pick key-value when every read is a full-value fetch by key, and document when more than one query shape exists or will exist.
What is a key-value database, and what access patterns is it actually good at? It maps an opaque key to an opaque value with no server-side understanding of the payload. It excels at single-key lookups where the caller already knows the key and wants the whole value — session state, feature caches, configuration, and short-lived tokens. Any second access pattern must be hand-built as a secondary key space.
What is a document database, and when do secondary indexes and partial updates justify the extra cost? A document database stores parsed JSON or BSON and can index and query nested fields. The extra cost is justified once the application needs to filter server-side, update individual fields of a large value without a read–modify–write cycle, or serve query shapes that were not known at design time.
Which AI/application workloads suit a key-value store, and which suit a document store? Feature caches and session or conversation state suit key-value: keyed access, whole-value reads, tight latency budgets. Embeddings metadata and inference request logs suit document stores, because both are queried by field — tenant, model version, permissions, timestamp — rather than by a single known key.
How do key-value and document databases differ on query capability, indexing, consistency and scaling? Key-value stores offer key-based access only, no native secondary indexes, and a narrow latency distribution that scales predictably by partitioning on the key. Document stores add field predicates and nested indexes, which widens the latency profile with index selectivity and consumes write throughput and storage to maintain those indexes.
Where do vector databases fit relative to key-value and document stores, and do you need a separate one? A vector database is a similarity-search index, not a competing general-purpose model, so it usually sits beside a key-value or document store that holds metadata and source content. Whether it needs to be a separate system depends on filter complexity and recall requirements — some document stores now carry native vector indexes at the cost of tuning depth.
What are the migration and cost consequences of picking the wrong data model, and how do you avoid locking yourself in? The visible cost is over-fetching: bytes read per request and p99 read latency inflate as payloads grow. The hidden cost is a post-launch model change, which requires a dual-write window plus backfill and verification. Reduce lock-in by documenting the access patterns, keeping serialisation behind an interface, and avoiding provider-specific query syntax scattered through application code.
