A training job runs on eight A100s. The GPU dashboard reads 40% utilisation and the ML team is drafting a request for four more cards. Before that purchase clears, one question decides whether it helps at all: is the GPU actually the bottleneck, or is Cassandra failing to feed it fast enough?

That question rarely gets asked, because the two halves of the problem live in different teams. The database group tunes Cassandra read latency in isolation, chasing p99 numbers against a synthetic query mix. The ML group benchmarks GPU FLOPs in isolation, comparing cards on peak throughput. Both sides optimise their own layer and both sides can be individually correct while the system as a whole wastes half its compute budget. When Cassandra is the feature store or feature-vector store sitting upstream of a training or inference loop, its sustained read throughput sets the ceiling on how busy the GPU can be. You do not get to pick which layer is the bottleneck by intuition. You measure both, together, or you are guessing.

How does Cassandra database performance actually work here?

Cassandra is a wide-column, distributed database designed for high write throughput and horizontal scale across commodity nodes. Data is partitioned by a partition key, hashed onto a token ring, and replicated across nodes according to a replication factor. Reads resolve by locating the replica set for a partition, reading from one or more replicas depending on consistency level, and merging results across the in-memory memtable and on-disk SSTables. That last part — merging across SSTables — is where a lot of AI-pipeline surprises hide.

In a standard OLTP setting, “Cassandra performance” means: can it serve a point lookup or a small partition scan within a latency budget, under a given write load, at a given consistency level. That framing is fine for a web backend. It is the wrong framing when Cassandra sits in front of a GPU, because the relevant question is no longer per-query latency — it is sustained rows-per-second delivered to the training loop without the reader stalling. A pipeline that reads one partition per training step at 5 ms each caps you at 200 steps/second on data alone, no matter what the GPU can do. If each step needs a batch of feature rows and those rows are scattered across cold partitions, the effective feed rate collapses well below what a p99 latency chart would suggest.

The naive model treats the database and the accelerator as separate procurement problems. The correct model treats the data path as a pipeline where the slowest stage governs the whole. This is the same wasted-capacity pattern we describe across our GPU performance work: you pay for FLOPs you never use because something upstream — here, the feature store — cannot keep the pipe full.

When does Cassandra become the bottleneck that leaves the GPU idle?

The stall shows up during the data-transfer phase of each step. A training iteration is, roughly: fetch a batch, move it to the device, run forward and backward passes, apply the update. If the fetch stage cannot complete before the previous compute stage finishes, the GPU sits idle waiting for its next batch. On a well-fed pipeline that gap is hidden by prefetching and double-buffering. On a Cassandra-limited pipeline the prefetch queue drains faster than Cassandra refills it, and the double-buffer stops absorbing the variance.

A few concrete failure shapes we see when this issue arises in practice:

• Hot partitions. A partition key that concentrates reads on a small set of partitions (for example, keying features by day when most reads hit “today”) pushes disproportionate load onto the replica nodes owning those tokens. Those nodes saturate; the rest of the cluster idles. Adding cluster capacity does nothing because the load is not spread.
• Uncompacted SSTables. Under heavy write load, a partition’s data spreads across many SSTables. A read must touch each of them and merge, so read amplification climbs. The read latency chart looks acceptable at p50 and terrible at p99 — and the training loop feels the p99, because one slow batch stalls the whole step.
• Tombstone accumulation. Deletes and TTL expiries leave tombstones that a read must scan past before returning live rows. A partition with thousands of tombstones can turn a nominal point read into a multi-millisecond scan.
• Single-region reads under a distributed cluster. When the GPU nodes and the Cassandra replicas serving them sit in different availability zones or regions, the per-read network round-trip dominates. The database is “fast” and the feed is still slow.

None of these show up if you only watch GPU-busy percentage, because a stalled GPU that is polling for its next batch can still register as busy in coarse utilisation metrics. The distinction between GPU-busy percentage and useful-FLOP percentage is the whole game, and it is the reason the three pillars of observability applied to GPU utilisation matter as much on the data side as on the compute side.

Which Cassandra design choices most affect throughput for AI feeds?

The design decisions that govern feed throughput are made at schema-design time, long before anyone benchmarks a GPU. Three matter most, and they interact.

Partition key design is first because it determines whether load spreads or concentrates. A good partition key for an AI feed produces partitions that are large enough to amortise the read overhead but small enough that no single node becomes a hotspot, and it aligns with the access pattern of the training or inference loop — ideally one partition maps cleanly to one batch’s worth of features. A partition key chosen for the write path (natural for the application) is often wrong for the read-heavy feed path.

Replication factor and consistency level set the read cost. A replication factor of three with LOCAL_QUORUM reads gives durability and local-region resolution but requires two replicas to respond per read. Dropping to ONE for a training feed — where a slightly stale feature is often tolerable — can cut read latency materially, at a correctness trade-off you have to decide deliberately, not by default.

Compaction strategy governs read amplification over time. Size-tiered compaction favours write throughput but lets read amplification grow; leveled compaction bounds the number of SSTables a read touches at the cost of more write I/O. For a feature store that is written in bulk and read continuously by a GPU loop, the compaction strategy is a throughput decision, not a housekeeping detail.

Decision surface: which Cassandra lever to pull for a GPU-fed feed

Symptom in the feed	Most likely Cassandra cause	First lever to try	Evidence class
GPU idle in bursts, some batches slow	High read amplification (many SSTables)	Switch to leveled compaction; force compaction	observed-pattern
A few nodes saturated, rest idle	Hot partitions	Redesign partition key to spread load	observed-pattern
Steady low feed rate, all nodes moderate	Consistency/replication cost per read	Lower read consistency where staleness is tolerable	observed-pattern
Latency fine locally, feed slow overall	Cross-region reads	Co-locate replicas with GPU nodes	observed-pattern
Reads slow only on some partitions	Tombstone accumulation	Tune TTL/GC grace; audit delete patterns	observed-pattern

These are starting hypotheses, not guarantees — each is a pattern we have seen across data-pipeline engagements, not a benchmarked rate. The point of the table is to turn a vague “the database is slow” into a specific thing to measure next.

How do I tell whether the GPU is stalling on compute or on Cassandra?

You profile both layers on the same timeline and look for the anti-correlation. This is the single most useful diagnostic move, and it is skipped far more often than it should be.

On the GPU side, the tools are standard: nvidia-smi for coarse utilisation, and Nsight Systems or the PyTorch profiler for the fine-grained picture. What you are looking for is not average utilisation — it is the shape of utilisation over a step. A compute-bound step shows the GPU busy from batch-ready to update-complete. A data-starved step shows a gap: the GPU goes idle, then a batch arrives, then it works. If the profiler timeline shows the GPU waiting on the dataloader’s next-batch call, the bottleneck is upstream.

On the Cassandra side, the signals that map to lost GPU time are read latency percentiles (p99 especially), SSTables-per-read, tombstone-scanned counts, and per-node request rates. Cassandra exposes these through JMX and nodetool (nodetool tablestats, nodetool tablehistograms). A tablehistogram showing a partition touching a dozen SSTables per read, correlated in time with GPU idle gaps, is your answer: the feed is the bottleneck.

The reason to profile together rather than separately is that each layer looks acceptable in isolation. The database team’s p99 chart and the ML team’s utilisation chart can both pass their own thresholds while the combined pipeline wastes compute. The gap only appears when you overlay the two timelines. This is exactly what a GPU performance audit does when a stall traces back to the data layer — it isolates the data-feed as the wasted-capacity cause rather than blaming the GPU, and it stops a “buy more cards” purchase that would not have moved the number.

The same logic extends beyond Cassandra. If your features live in a vector index rather than a wide-column store, the mechanism is identical — retrieval latency, not compute, caps throughput. We walk through that variant in how retrieval latency becomes a GPU bottleneck for vector databases, and the storage-layer measurements that quantify it in storage benchmarks for Cassandra-fed GPU pipelines.

Where does the wasted spend appear on cloud versus on-prem?

The mechanism is the same in both environments; where the money leaks differs. On-prem, the GPU is a sunk capital cost — an idle A100 has already been paid for, so the loss is opportunity cost: fewer training runs per week, longer time-to-model, a case for expansion that is really a case for fixing the feed. The Cassandra cluster is likewise fixed hardware, so the tuning levers (compaction, partition keys, node placement) are the whole story.

In the cloud the loss is direct and metered. You rent the GPU by the hour, so every idle second during a data stall is billed at the full instance rate. Worse, cross-region reads that were “free” on a flat on-prem network now carry inter-zone data-transfer charges and higher latency, so a single-region-read misconfiguration costs you twice — once in stalled GPU time, once in transfer fees. The economic framing we use elsewhere applies directly here: the gap between TCO per purchased FLOP and TCO per useful FLOP is where a Cassandra-limited feed shows up on the invoice. If you are weighing where GPU-bound cost hides across managed cloud tooling, cost per useful FLOP on AWS versus Azure covers the compute-instance side of the same accounting.

FAQ

What should you know about cassandra database performance in practice?

Cassandra is a distributed wide-column database that partitions data by a partition key across a token ring and resolves reads by merging a partition’s data across the in-memory memtable and on-disk SSTables. In practice, “performance” for an AI pipeline is not per-query latency but sustained rows-per-second delivered to the training or inference loop without the reader stalling — a different measurement than the p99-latency framing used for OLTP backends.

How does Cassandra read/write throughput become a bottleneck that leaves GPU capacity idle during data transfer?

During each training step the GPU fetches a batch, transfers it to the device, and computes. If Cassandra cannot refill the prefetch queue before the previous compute stage finishes, the GPU goes idle waiting for its next batch. Coarse utilisation metrics can still read “busy” while the GPU polls, which is why a data-fed stall hides from anyone watching only GPU-busy percentage rather than useful-FLOP percentage.

Which Cassandra design choices — partition keys, replication, compaction — most affect throughput for AI data feeds?

Partition key design determines whether read load spreads across the cluster or concentrates on hot partitions; replication factor and consistency level set the per-read cost; and compaction strategy governs read amplification over time. These are set at schema-design time and interact — a partition key chosen for the write path is often wrong for a read-heavy GPU feed.

How do I tell whether my GPU is stalling on compute or on Cassandra reads, and how do I profile both together?

Profile both layers on the same timeline and look for anti-correlation. Use Nsight Systems or the PyTorch profiler to see whether the GPU idles waiting on the dataloader’s next-batch call, and correlate that with Cassandra’s read-latency percentiles and SSTables-per-read from nodetool tablehistograms. Each layer can look acceptable in isolation, so the stall only becomes visible when you overlay the two timelines.

What Cassandra-side metrics (read latency, tombstones, hot partitions) map to lost GPU useful-FLOP time?

The signals that map to lost GPU time are p99 read latency, SSTables-per-read, tombstone-scanned counts, and per-node request rates — all exposed via JMX and nodetool. A partition touching many SSTables per read, or a small set of nodes saturated while the rest idle, correlated in time with GPU idle gaps, points the finger at the feed rather than the accelerator.

How does Cassandra performance behave differently on cloud versus on-prem, and where does the wasted spend appear?

The stall mechanism is identical, but on-prem the loss is opportunity cost against sunk hardware, while in the cloud each idle GPU-second is billed at the full instance rate. Cross-region reads add both latency and inter-zone transfer charges in the cloud, so a single-region-read misconfiguration costs twice — the gap between TCO per purchased FLOP and TCO per useful FLOP is where a Cassandra-limited feed appears on the invoice.

Before you buy more compute, profile the path

The instinct to add GPUs when utilisation looks low is understandable and often exactly backwards. If the accelerator is stalling on a Cassandra feed, doubling the cards doubles the idle time and the bill along with it. The question that resolves it is not “how fast is my database” or “how fast is my GPU” in isolation — it is whether the two timelines, overlaid, show the compute waiting on the data. When the stall traces to poor partition keys, uncompacted SSTables, or cross-region reads, the fix lives in the data layer and costs nothing in silicon. A GPU performance audit exists to draw that line precisely: to name the wasted-capacity cause before a procurement decision assumes it away.