A disk that hits 3 GB/s sequential in fio can collapse to a fraction of that the moment Cassandra starts serving a real training loop. The datasheet number is not the number your GPU sees. When infrastructure teams benchmark storage before standing up a Cassandra-fed GPU pipeline, they almost always measure the wrong thing — and the gap between what they measured and what the disk delivers under load is exactly where the GPU sits idle, waiting for rows. That idle time is expensive. If a training loop stalls waiting for data, the GPU is billed at full rate while doing no useful FLOPs. The question that matters is not “how fast is this disk?” but “how fast is this disk under the specific access pattern Cassandra generates when it reads rows to feed the loop?” Those are different questions with different answers, and the second one is the only one that maps to recovered GPU time. Why synthetic sequential benchmarks overstate what you can use The default reflex is to reach for fio or dd, point it at the target device, and report peak sequential throughput plus a headline IOPS figure. Those numbers are real — the hardware genuinely can push that bandwidth under a large-block sequential read. The problem is that a Cassandra read path never issues that workload. Cassandra stores data in immutable SSTables. A single-partition read that misses the row cache and the page cache has to consult a bloom filter, seek into a partition index, and pull the relevant data blocks — often from more than one SSTable, because a partition’s data can be spread across files that haven’t been compacted together yet. The resulting disk traffic is a stream of small, effectively random reads, not a long contiguous scan. On top of that, compaction runs in the background rewriting SSTables (write amplification), and read-repair traffic adds more competing I/O. Everything lands on the same device at the same time. This is why the sequential figure is misleading rather than merely optimistic. A disk rated at 3 GB/s sequential can deliver an order of magnitude less throughput under a 4 KB random-read profile, and that collapse is precisely the operating point Cassandra pushes it into (observed pattern across storage-bound pipelines we have profiled; not a single named benchmark). The datasheet is describing a workload shape that does not occur on the read path feeding your model. You cannot infer the useful storage ceiling from a spec sheet — you have to benchmark the pattern that actually reaches the disk. What access pattern should you actually benchmark? The honest answer is: the one Cassandra produces on your data, at your partition sizes, with your compaction state. There are two ways to get there, and they answer slightly different questions. The first is to reconstruct the pattern synthetically. In fio this means abandoning the sequential defaults and configuring a random-read job with a small block size (Cassandra’s chunk length, often 4 KB or 16 KB depending on chunk_length_in_kb), a realistic queue depth, and a mix of concurrent read and write streams to stand in for compaction. This is portable and repeatable, and it gets you close — but it is still a model of the workload, not the workload. The second is to benchmark Cassandra directly and watch the disk underneath. Run cassandra-stress (or replay a captured production read profile) against a cluster loaded with representative data, then measure block-device throughput, IOPS, and latency with iostat, blktrace, or biolatency from the BCC tools while the load runs. This captures the real interaction — bloom filter misses, SSTable fan-out, compaction contention — that a synthetic job approximates. This is the pattern that decomposes a data-feed stall one layer below query design, which is the same wasted-capacity story we trace in Cassandra database performance for GPU-fed AI pipelines, pushed down to the physical disk. Neither approach is complete on its own. The synthetic job tells you the ceiling the hardware can hit under the right shape; the Cassandra-driven run tells you how much of that ceiling the database’s own overhead leaves for your reads. Which storage metrics map to lost GPU time? Not every storage number connects to a GPU stall. The ones that do are the ones that describe latency and sustained random throughput under contention — not peak anything. Here is the mapping we use when a GPU performance audit traces a stall to the storage layer. Storage metric How to measure it What it costs the GPU Sustained random read throughput (MB/s) fio random-read job or iostat under cassandra-stress Sets the raw rate rows can arrive; below the loop’s demand, the GPU underfeeds Random read IOPS Same, small block size Small-partition reads are IOPS-bound, not bandwidth-bound; a low ceiling caps samples/sec p99 read latency floor biolatency, or Cassandra client-side p99 Tail latency, not mean, sets the effective batch-assembly time; a high p99 stalls the slowest reads in every batch Write amplification under compaction Compaction throughput vs. logical write rate Steals read IOPS on the shared device; a stall that appears only during compaction windows points here Read-repair / hint traffic Cassandra metrics + iostat correlation Adds background I/O contention that inflates read latency intermittently The metric people most often miss is the p99 latency floor. Mean latency looks fine while the tail quietly sets the ceiling — a training batch is only assembled when its slowest row arrives, so the 99th-percentile read is what governs samples/sec, not the average. When we see a GPU that is 40% utilised despite a “fast” disk, the p99 read latency under load is usually the first place the answer shows up (observed pattern; not a benchmarked rate). How do you tell whether the stall is query design or the disk itself? This is the diagnostic that separates a Cassandra tuning problem from a storage procurement problem, and getting it wrong is expensive in both directions — you can spend weeks re-partitioning a schema when the real fix was a faster disk tier, or upgrade to provisioned IOPS when the real fix was a data-model change. Work through it in order: Is the GPU actually stalling on data? Confirm the bottleneck is upstream before touching storage at all. Framework-level input pipeline timers (the PyTorch DataLoader wait time, or nvidia-smi/DCGM utilisation dips correlated with fetch) tell you the GPU is waiting on rows, not on compute. If it isn’t, storage is not your problem yet. Is the disk near its measured random-read ceiling? Compare live iostat throughput and IOPS during the stall against the random-read ceiling you benchmarked. If the disk is saturated at its real (not datasheet) limit, the storage layer is the constraint — no query rewrite recovers throughput a saturated device cannot deliver. Is the disk idle while Cassandra is slow? If the device has headroom but reads are still slow, the stall lives above the disk: tombstone-heavy partitions, unbounded partition scans, missing row-cache hits, or a data model that fans a single read across many SSTables. That is query and schema work, not a hardware swap. Does the stall track compaction windows? If read latency spikes only when compaction runs, the write-amplification contention on the shared device is the culprit — addressable by compaction throttling, a separate commit-log device, or faster storage, in roughly that order of cost. The point of the sequence is to localise the cost before spending money. Disk choice — NVMe versus network-attached, and which provisioned-IOPS tier — is a real cost lever, but it is a separate lever from GPU procurement, and it only pays off when steps 1 and 2 confirm the disk is the binding constraint. Treating storage and compute cost as one line item is how teams end up over-provisioning both; the same separation-of-cost-levers logic runs through AWS vs Azure for GPU workloads. A worked example: reading the numbers Suppose you benchmark a cloud NVMe volume and the datasheet promises 3.0 GB/s sequential read and 500,000 read IOPS. You then run cassandra-stress with a read-heavy profile against a representative dataset with 16 KB chunks and measure the block device with iostat. Under the Cassandra random-read pattern, sustained throughput settles at roughly 350–500 MB/s, and IOPS peak well below the headline figure because each logical row read touches multiple SSTable blocks (illustrative figures — treat as a shape to reproduce, not a portable benchmark). Your training loop demands, say, 900 MB/s of row data to keep the GPU fed at target batch size. The gap — demand above the sustained random-read ceiling — is your stall. The GPU is billed while it waits, and no query tuning closes a gap this size because the device is already at its real limit. Now the decision is concrete: a higher provisioned-IOPS tier, a local NVMe cache in front of network-attached storage, or a memory-tiering layer to keep hot partitions out of the disk read path entirely — the trade-off space we walk through in Intel Optane explained: memory tiering and GPU data-feed bottlenecks. Each carries a different cost, and you can only price them because you benchmarked the pattern that actually reaches the disk instead of the one on the datasheet. FAQ What does working with storage benchmarks involve in practice? Storage benchmarking measures how a disk or volume performs under a defined access pattern — block size, read/write mix, queue depth, and concurrency. In practice the result is only meaningful if the pattern matches the real workload; a sequential benchmark and a random-read benchmark of the same device produce very different numbers, and only the pattern your application actually issues predicts what you will get in production. Why do synthetic sequential-throughput benchmarks overstate the storage a Cassandra-fed GPU pipeline can actually use? Because Cassandra reads rows from immutable SSTables as small, effectively random I/O — not the long contiguous scans that sequential benchmarks measure. A device rated for multi-GB/s sequential throughput can drop by an order of magnitude under a small-block random-read profile, and that lower operating point is the one Cassandra pushes it into, so the datasheet figure describes a workload that never reaches the disk. What access pattern should I benchmark to reflect how Cassandra reads rows to feed a training or inference loop? Benchmark small-block random reads (matching Cassandra’s chunk_length_in_kb, often 4–16 KB) with realistic concurrency and a concurrent write stream to stand in for compaction. Better still, drive real load with cassandra-stress or a replayed production read profile and measure the block device underneath with iostat or biolatency, so the SSTable fan-out and compaction contention are captured rather than approximated. Which storage metrics — random read IOPS, p99 latency, write amplification under compaction — map to lost GPU useful-FLOP time? Sustained random-read throughput and IOPS set the raw rate rows arrive; below the loop’s demand the GPU underfeeds. The p99 read-latency floor is the most-missed one — a batch assembles only when its slowest row arrives, so the tail, not the mean, sets the samples/sec ceiling. Write amplification during compaction steals read IOPS on the shared device, showing up as stalls confined to compaction windows. How do I benchmark NVMe versus network-attached or provisioned-IOPS storage under a realistic Cassandra load? Run the same Cassandra-driven read profile against each option and measure sustained random-read throughput, IOPS, and p99 latency under load — not peak sequential. Network-attached and provisioned-IOPS tiers behave very differently under small random reads than local NVMe, and the tier’s guaranteed IOPS number is a ceiling you must confirm holds under the SSTable access pattern before treating it as usable capacity. How do I tell whether the read stall lives in Cassandra query design or in the underlying storage layer? First confirm the GPU is waiting on data (input-pipeline timers or utilisation dips), then compare live disk throughput against your measured random-read ceiling. If the disk is saturated at its real limit, storage is the constraint; if the disk has headroom while reads are slow, the problem is above the disk — data model, tombstones, or partition fan-out — and no hardware swap fixes it. When a data-feed stall traces back to Cassandra, the useful question is never “is the disk fast?” — it is “is the disk fast under the pattern Cassandra hands it, and is that pattern the thing starving the GPU?” A GPU performance audit that stops at query tuning leaves the storage read stall in place; decomposing it to the disk is what turns a vague “the pipeline is slow” into a priced, one-line decision about which storage tier buys back the most GPU time.
