A training run stalls at 40% GPU utilization and the team blames the model. Nine times out of ten, the accelerators are idle because storage cannot feed them fast enough. Machine learning storage is not a single tier you provision once — it is a chain of stores, each with a different access pattern, and the chain is only as fast as the link the GPU is currently waiting on.

That is the point most storage discussions miss. People treat ML storage as a capacity question (“how many terabytes of training data do we have?”) when the operationally relevant question is a movement question: can the data reach the compute in the shape and at the rate the compute needs it? Capacity is cheap. Sustained throughput to a saturated accelerator is not.

Why the GPU sits idle when storage is the bottleneck

The naive mental model treats a training loop as compute-bound: load the batch, run the forward and backward pass, repeat. In that model, storage is a background concern — data lives somewhere, the framework reads it, the math happens. The reality is that a modern accelerator can consume batches far faster than a general-purpose object store can serve them, especially for datasets of small files: millions of JPEGs, short audio clips, tokenized shards.

When this mismatch appears in practice, the symptom is low and jittery GPU utilization while CPU and network wait times climb. The nvidia-smi reading hovers well below 100%, the data-loader workers in a PyTorch DataLoader spend their time blocked on I/O, and adding more GPUs makes the per-GPU efficiency worse because they all contend for the same read path. This is an observed pattern across the data-heavy training engagements we take on, not a benchmarked constant — but the shape is consistent enough to plan around.

The correct frame is to treat storage as a bandwidth budget that must match the accelerator’s appetite. If a training node’s GPUs can process, say, several gigabytes per second of decoded samples, then every layer between the raw bytes and the GPU — the object store, the network, the local cache, the decode step — has to sustain at least that rate on average, with enough buffering to absorb bursts. Provisioning for average capacity while ignoring peak throughput is the single most common way ML infrastructure underperforms its hardware.

The tiers of machine learning storage

There is no one storage system for machine learning because the workload is not one workload. Raw ingestion, feature computation, training reads, checkpoint writes, and low-latency serving each stress storage differently. Treating them as one tier is how teams end up paying block-storage prices for cold archival data, or serving features out of a data lake that was never built for millisecond lookups.

A useful way to decompose it is by the job each store does and the access pattern that defines it.

Storage tier	Primary job	Access pattern	Typical technology	What breaks if you skip it
Object store (data lake)	Durable, cheap bulk storage of raw and processed data	High-throughput sequential reads, write-once	Amazon S3, Google Cloud Storage, Azure Blob	Nothing — this is the foundation; the risk is treating it as a serving layer
Training read path	Feed batches to accelerators at sustained bandwidth	Massively parallel, latency-sensitive on small files	S3 + local NVMe cache, Lustre/FSx, WebDataset shards	GPUs idle, training cost inflates 2–3×
Feature store	Consistent features for training and serving	Batch reads (offline) + point lookups (online)	Feast, Tecton, Vertex AI Feature Store	Training/serving skew, silent accuracy loss
Vector store	Similarity search over embeddings	Approximate nearest-neighbor queries	Milvus, pgvector, managed vector DBs	Retrieval latency spikes, RAG quality drops
Checkpoint / artifact store	Durable model and run artifacts	Large sequential writes, infrequent reads	S3 + MLflow, versioned buckets	Lost runs, no reproducibility, no rollback

The rows are not interchangeable. The most expensive mistake we see is collapsing the training read path into a direct read from the object store, with no local caching tier. Object stores are optimized for durability and sequential throughput on large objects, not for the millions of tiny random reads a shuffled image dataset generates. The fix is almost always structural — reshard the data into larger sequential-read containers (the WebDataset .tar shard pattern, or TFRecord/Parquet files) and stage hot data on local NVMe — rather than buying a faster object store.

For a deeper treatment of how these stores fit together across warehouses, vector databases, and big-data systems, see our breakdown of data infrastructure for ML across warehouses, vector stores, and big-data databases, which covers the warehouse and query-engine side this article deliberately leaves out.

Throughput or IOPS: which one actually matters?

Storage vendors quote two numbers that get conflated constantly: throughput (bytes per second) and IOPS (operations per second). For machine learning, which one binds depends entirely on file size and access pattern, and getting this wrong leads to buying the wrong tier.

Large sequential reads — streaming a 1 GB Parquet shard or a TFRecord file — are throughput-bound. Here you want raw MB/s, and object stores with parallel range reads do well. Small random reads — grabbing individual samples from a shuffled dataset of small files — are IOPS-bound and latency-bound, and this is where object stores fall over. A single small-file read incurs request overhead that dwarfs the payload, so a dataset of ten million 30 KB images can be slower to train on than a dataset ten times its size stored as large shards.

The practical rule we apply: never let the training loop issue one storage request per sample. Batch the samples into shard files so one read amortizes across thousands of examples, then shuffle at the shard-and-buffer level rather than the individual-file level. This is exactly what the WebDataset and TFRecord formats are designed for, and it is why the same dataset can go from I/O-bound to compute-bound with no hardware change at all.

The failure that hides for months: training/serving skew

The subtlest machine learning storage failure has nothing to do with speed. It happens when the features used to train a model come from one path — say a batch job over the data lake — and the features used to serve it come from another — a real-time lookup against a different store, computed by different code. The transformations drift apart. A timestamp is truncated differently, a null is filled with a different default, a unit is off by a factor. The model trained on clean, consistent features now sees subtly different inputs in production, and its accuracy quietly degrades.

This is what a feature store exists to prevent. A feature store like Feast or Tecton provides a single definition of each feature, materialized once, and served to both the offline training path and the online inference path from the same logical source. The early warning sign is a model that validates well offline but underperforms in production for reasons no one can explain — and the reason is almost always that the two data paths never agreed on what the features were. Because this failure is silent and slow, it is far more expensive than an idle GPU; it corrupts the model’s decisions before anyone measures it.

FAQ

Why does storage cause GPU idle time during training?

Modern accelerators can consume training batches faster than a general-purpose object store can serve them, especially for datasets of many small files. When the data-loader can’t sustain the accelerator’s read appetite, the GPU sits waiting on I/O — showing up as low, jittery utilization. The fix is usually restructuring data into large sequential shards and adding a local NVMe cache, not buying faster hardware.

What is the difference between throughput and IOPS for ML storage?

Throughput measures bytes per second and binds on large sequential reads like Parquet or TFRecord shards; IOPS measures operations per second and binds on many small random reads. Datasets of many small files are IOPS- and latency-bound, which is why object stores struggle with them. Batching samples into shard files converts an IOPS problem into a throughput problem the object store handles well.

Do I need a feature store for machine learning?

You need one when the same features are used for both training and low-latency serving and are at risk of being computed by two different code paths. A feature store gives a single feature definition served to both paths, preventing training/serving skew — a silent accuracy loss that is hard to diagnose. For batch-only or single-path workloads, it may be unnecessary overhead.

Can I just train directly from object storage like S3?

For large sequential files, yes — object stores serve those efficiently with parallel range reads. For shuffled datasets of many small files, reading directly from the object store per-sample makes GPUs idle. Reshard into large containers and stage hot data on local NVMe so one read amortizes across thousands of samples.

Storage decisions for ML are ultimately about matching each store’s access pattern to the job it serves — and the moment your training reads, feature serving, and archival needs are pulling one tier in three directions, the question stops being “how do we store this?” and becomes “how do we run the whole platform?” That platform view, spanning ingestion, orchestration, and cost, is where our work on AI in cloud and DevOps picks up the thread.