A p95 latency alert fires. The APM trace waterfall shows one red span: a call into your big data store. The obvious move is to scale or swap the database. It is also, very often, the wrong move — because a trace span tells you where time was spent, not why. This is the recurring pattern behind expensive, unnecessary re-platforming projects. A team sees a slow data-access call in the request budget, concludes “the database,” and starts evaluating replacements. Meanwhile, profiling across the serving boundary would have shown the query itself finished in single-digit milliseconds and the rest of that span was network round-trips, deserialisation, or the model server waiting on a batch to fill. The database gets blamed for work it never did. To reason about this correctly you have to hold two things apart: what a big data database actually does in an inference request, and what the model-serving stack does on the GPU. They live in the same trace, they show up in the same latency number, and they respond to completely different fixes. How does a big data database work in an inference serving path? A “big data database” is a loose term for a store built to hold and serve data at a volume that does not fit — or does not perform — on a single node. The defining traits are distribution and access-pattern specialisation: the data is partitioned across many machines, and the storage layout is tuned to one kind of read. In an AI serving path, these systems do one job at request time: feature and context retrieval. When a request arrives, the serving code looks up the features, embeddings, or documents the model needs before it can run. That lookup is a separate step from the model’s forward pass. It happens on CPU and network, not on the GPU. The database’s contribution to your p95 is entirely on the retrieval side of the request budget — a point we develop further in how the data layer shapes production-AI reliability. The reason this gets tangled with model-serving cost is that both happen inside one request. The client sees one latency number. The APM trace shows a stack of spans. But retrieval time and model-execution time are governed by different constraints, and a single trace waterfall rarely separates them cleanly. What types of big data databases matter for an inference path? Three broad families show up in feature stores and inference pipelines, and each is fast at a different thing. Confusing their access patterns is one way teams end up “scaling the wrong axis.” Family Layout Fast at Typical inference role Common examples Key-value / wide-column Row or partition keyed by an ID Single-key point lookups at high concurrency Online feature store, per-entity feature fetch Cassandra, DynamoDB, Redis, HBase Columnar / warehouse Data stored column-by-column Scanning and aggregating many rows over few columns Offline feature computation, batch scoring, analytics BigQuery, Snowflake, ClickHouse, Parquet on object store Distributed SQL / lakehouse Rows partitioned, SQL semantics Mixed transactional and analytical queries at scale Feature backfill, joins, hybrid online/offline CockroachDB, Spanner, Delta Lake, Iceberg The practical consequence: a columnar warehouse is superb for computing features across a training window and catastrophic for a per-request point lookup — its scan-oriented layout pays a fixed cost to open and read that a key-value store avoids. Teams that serve online features out of an analytics warehouse then blame “big data latency” are, in effect, using a floor scrubber to wash a teacup. The store is not slow; it is being asked to do the one thing it is worst at. Choosing the right engine for the feature-computation side of this is a related decision we cover in Spark vs Presto for AI feature pipelines. Why does the APM span blame the database? Here is the trap, stated plainly. An APM span that reads db.query and shows 40 ms of wall time is measuring the client-observed duration of a call, not the server-side query execution. Those are different numbers, and the gap between them is where most misattribution lives. That 40 ms can decompose into several very different costs: Query execution — the store actually finding and reading the data. This is the only part a database change or index tweak affects. Network round-trips — connecting, sending the request, receiving the response, possibly across availability zones. A cross-AZ hop alone can dominate a fast query. Serialisation and deserialisation — turning rows into objects your serving code can use. For wide feature vectors this is frequently the largest slice, and it runs in your process, not the database’s. Connection-pool contention — the request waiting for a free connection under load, which looks like slow “database time” but is a client-side queueing problem. Batch-fill wait — in a batched serving stack, the retrieval call may block while the model server assembles a batch, so the span absorbs time that belongs to the model path, not the data path. Only the first of these is fixed by touching the database. The rest are fixed in networking topology, serialisation format, pool sizing, or the batching strategy of your model server. A trace span cannot distinguish them because it stops at the boundary of the instrumented call; it does not follow the cost inward. This is the same trace blind spot that shows up when reading a machine learning architecture diagram of the serving path — the arrow between “feature store” and “model server” hides several distinct costs behind one line. We see this pattern regularly: a red span becomes a migration proposal without anyone measuring which of the five costs above actually dominated. That is a governance failure as much as a technical one. How do you measure the database’s share of inference latency? The fix is not a tool; it is a discipline. You attribute the request budget by profiling across the serving boundary rather than trusting the single span. The goal is a breakdown of where each millisecond of p95 went, split into data-retrieval time and model-execution time. A workable diagnostic sequence: Instrument the boundary, not just the call. Add timing on both sides of the retrieval — client-side around the whole call, and server-side around query execution alone. The difference is your network-plus-serialisation cost. This is the single most revealing measurement and most teams have never taken it. Isolate serialisation. Time the deserialisation step separately from the wire time. For wide feature vectors this often surprises people. Profile the model path independently. Measure GPU execution time with the model server’s own metrics, so you know how much of p95 is the forward pass versus everything before it. GPU profiling that isolates kernel time from data-access time is the discriminating step — our GPU performance work exists precisely to draw that line. Check for batch-fill wait. If retrieval spans grow under load but query execution stays flat, the time is queueing, not the database. Attribute the p95. Only now do you have a defensible split: X ms retrieval, Y ms model, Z ms overhead. A re-platforming decision made on this is evidence-based; one made on the raw span is a guess. This attribution is exactly what an AI inference cost audit produces. When an APM trace flags a slow database call, Inference Cost-Cut Pack profiles across the serving boundary to attribute latency between feature retrieval and model execution — and reports the result as p95 before/after and cost-per-request, so the finding is measurable rather than anecdotal (this is how we scope the audit in engagements, not a published benchmark). When is the fix a database change versus a serving-path change? Once the p95 is attributed, the decision usually resolves quickly. The table below maps the dominant cost to the fix that actually addresses it — and, just as importantly, to the fix that does not. If the dominant cost is… The fix is… Not… Query execution time Indexing, partitioning, or a store better suited to the access pattern — Network round-trips Co-location, connection reuse, or caching hot features Scaling the database Serialisation A leaner format (columnar wire, protobuf) or smaller feature payloads Scaling the database Connection-pool contention Right-sizing the pool, async retrieval Scaling the database Batch-fill wait Batching-window and runtime tuning on the model server Anything on the data layer GPU forward pass Runtime, quantisation, or batching changes on the model side Anything on the data layer Notice how few rows point at the database itself. That is the whole point. The intuitive fix — scale the store — addresses one row out of six. Acting on the trace span alone is how teams re-platform a database that profiling would have exonerated, absorbing a migration cost that the request budget never justified. The unit-economics view makes the stakes concrete: translating data-retrieval latency into a cost-per-request KPI shows exactly how much of your spend the database contributes, which is what turns “the database feels slow” into a number a VP of Engineering can act on. That translation is the bridge between a latency trace and a spend decision. FAQ How should you think about big data databases in practice? A big data database distributes data across many nodes and specialises its storage layout for one kind of read — point lookups, column scans, or SQL joins at scale. In an inference path its practical job is feature and context retrieval at request time: fetching what the model needs before the forward pass runs. That retrieval happens on CPU and network, entirely separate from the model’s work on the GPU. What types of big data databases (columnar, key-value, distributed SQL) matter for an AI inference serving path? Key-value and wide-column stores (Cassandra, DynamoDB, Redis) are fast at high-concurrency point lookups and suit online feature stores. Columnar warehouses (BigQuery, Snowflake, ClickHouse) are fast at scanning many rows over few columns and suit offline feature computation and batch scoring. Distributed SQL and lakehouse systems (Spanner, Delta Lake, Iceberg) handle mixed workloads. Using one where another belongs — serving point lookups from a scan-oriented warehouse — is a common source of avoidable latency. How does data retrieval from a big data store contribute to inference request latency, and how do I measure its share? Retrieval adds time on the data side of the request budget, distinct from model-execution time on the GPU. To measure its share, instrument both sides of the retrieval call, isolate serialisation from wire time, profile GPU execution independently, and check whether latency growth under load is queueing rather than query time. The output is a defensible split of p95 into retrieval and model portions. Why does my APM span blame the database when the real cost may be serialisation, network hops, or the model waiting on features? An APM span measures the client-observed duration of a call, not server-side query execution. That duration bundles query time, network round-trips, serialisation, connection-pool contention, and batch-fill wait into one number. Only the query portion is fixed by touching the database; the span cannot separate the others because it stops at the boundary of the instrumented call. When is the fix a database change versus a batching, runtime, or serving-path change? A database change is warranted only when attributed profiling shows query execution time dominates. Network cost points to co-location or caching; serialisation to a leaner format; contention to pool sizing; batch-fill wait to model-server batching; and a slow forward pass to runtime, quantisation, or batching on the GPU side. Most dominant costs sit outside the data layer. How does an inference cost audit attribute latency between big data retrieval and model execution? The audit profiles across the serving boundary rather than trusting a single trace span. It times both sides of the retrieval, isolates serialisation, measures GPU execution independently, and attributes the request budget into retrieval versus model portions. Results are reported as p95 before/after and cost-per-request, so a re-platforming decision rests on profiled evidence. What are examples of big data databases used in feature stores and inference pipelines, and where do they hit the same trace blind spot APM leaves? Cassandra, DynamoDB, Redis, and HBase back online feature stores; BigQuery, Snowflake, and ClickHouse back offline feature computation; Delta Lake and Iceberg back lakehouse pipelines. All of them share the same blind spot: an APM span around the retrieval call absorbs network, serialisation, and batch-wait time, so any of these stores can look slow when the real cost lives elsewhere in the boundary. The question worth ending on is not “which database should we move to.” It is “which millisecond of our p95 are we actually paying for” — and until the request budget is attributed across the serving boundary, that number does not exist. A red span is a starting point for a measurement, not a conclusion.
