Look at any MLPerf Storage result table and the temptation is immediate: sort by the biggest number, buy that. That instinct is wrong, and it is wrong in a way that costs money after the hardware ships. MLPerf Storage does not tell you which storage system is fastest in some absolute sense. It tells you whether a storage system can keep a specified number of simulated accelerators busy while they train a specified model — and that framing changes everything about how the number should be read. The benchmark exists because a storage array’s raw bandwidth spec, quoted in gigabytes per second, says almost nothing about whether your GPUs will sit idle during training. The relevant question is never “how much bandwidth does this array have.” It is “can this array sustain the samples-per-second that my model at my batch size demands, under the messy random-read access pattern that training actually produces.” MLPerf Storage was built to answer the second question, and reading it as an answer to the first is where teams go wrong. What Does the MLPerf Storage Benchmark Actually Measure? MLPerf Storage, published by MLCommons, works by emulating the accelerators rather than requiring them. It uses a component called DLIO to replay the I/O access pattern of a real training workload — the read sizes, the shuffling, the epoch structure — and a “sleep” model that stands in for the time an accelerator would spend computing on each batch. The storage system feeds this emulated cluster, and the benchmark reports the maximum number of accelerators the storage can keep above a utilization threshold, typically around 90% for the training workloads in the suite. That last point is the one most easily missed. The headline result is not a bandwidth figure. It is an accelerator count at a sustained utilization target. A result that reads “supports 40 emulated H100s at 90%+ utilization on the 3D U-Net workload” is a statement about a specific model, a specific access pattern, and a specific accelerator profile. Change the model and the number changes, because different models impose radically different I/O intensity per unit of compute. This is the same measurement philosophy that runs through the rest of the suite. If you have read our explanation of what the MLPerf Training benchmark measures and how to use it for sizing AI infrastructure, the pattern is familiar: the number is only meaningful relative to a named workload. Storage inherits that discipline directly. A MLPerf Storage result is a proxy for one thing — can storage keep accelerators fed — and it is only valid for the workload profile it was measured against. Why Raw Bandwidth Misleads Consider two access patterns with identical aggregate bandwidth. A large-language-model pretraining job streaming tokenized shards is largely sequential, tolerates prefetching well, and can be satisfied by a system tuned for large sequential reads. A vision or diffusion training job randomly sampling millions of small image files produces a punishing random-read pattern where metadata operations and small-object latency dominate. The same array that comfortably feeds the LLM job can starve the vision job — despite identical bandwidth on paper. The variables that actually decide whether storage keeps accelerators fed are: Access pattern. Sequential streaming versus random small-file reads. Random small reads are far harder to sustain and expose metadata and IOPS limits, not bandwidth limits. Sample size and count. Millions of small samples stress the metadata layer; a smaller number of large samples stresses throughput. Batch size and per-step compute time. A larger batch or a heavier model spends more time computing per I/O, which relaxes the storage demand per accelerator. A light model per accelerator tightens it. Data-loading pipeline. Prefetch depth, decode CPU budget, and caching all sit between storage and the accelerator. A poor loader can starve GPUs even when the underlying storage is adequate — which is why CPU specifications for GenAI workloads belong in the same feasibility conversation as storage. None of these are captured by a bandwidth number. They are captured by matching the benchmark’s workload to yours. How Do I Read MLPerf Storage Results Against My Own Model? The correct procedure is a mapping exercise, not a ranking exercise. Start from your workload, then find the closest benchmarked profile, then adjust. A decision rubric for reading a result Step Question to answer What it tells you 1. Match the workload Which benchmarked model has an access pattern like mine — large sequential (LLM-style) or random small-file (vision/diffusion-style)? Whether the published number is even relevant to you 2. Match the accelerator Was it measured against emulated accelerators comparable to what I will deploy (e.g. H100-class)? Whether the per-accelerator I/O demand assumption holds 3. Read the count, not the bandwidth How many accelerators does it sustain at the utilization threshold? The real capacity figure for planning 4. Scale to your cluster Does my planned accelerator count fit under that sustained number, with headroom? Whether storage will bottleneck at your scale 5. Adjust for your loader Is my data-loading pipeline (prefetch, decode, cache) as efficient as the benchmark’s DLIO replay? Whether real-world utilization will fall below the benchmark’s Step 5 is where most planning error hides. The benchmark uses an idealized, well-tuned I/O replay. Your production loader — with real image decoding, augmentation, and a CPU budget shared against other work — is usually less efficient. Treat the benchmark’s sustained accelerator count as an optimistic ceiling and plan with margin below it. The ROI framing follows directly. Storage that cannot sustain your required samples-per-second leaves accelerators idle, and accelerator utilization is the number that justifies the hardware spend. In configurations we have reviewed, the difference between a storage tier sized correctly and one sized on headline bandwidth is the difference between GPUs running above 90% utilization and GPUs stalling into the 60s waiting on I/O (observed pattern across our data-pipeline scoping work; not a published benchmark). At that point the effective cost-per-training-run climbs regardless of how fast the accelerators themselves are. Why Storage I/O Becomes a Hidden Bottleneck in GenAI Pipelines Here is the failure that connects the infrastructure layer to a pattern we see repeatedly in GenAI projects. A prototype is built and validated against a small, curated dataset — a few thousand carefully chosen samples that fit comfortably in page cache or on a laptop’s local SSD. The pipeline looks fast. Utilization looks fine. The model trains. Everyone signs off on the architecture. Then production data arrives: millions of samples, no longer cache-resident, read randomly across epochs from shared network storage. The access pattern the prototype never exercised is suddenly the dominant cost, and the GPUs that looked saturated in the prototype now spend a large fraction of each step blocked on I/O. This is the storage-layer version of a problem we describe in why GenAI systems fail when they move from curated data to production data. The mechanism is the same: a prototype validated on a small curated set never exercises the sustained pressure that production scale imposes. On the data-quality side it shows up as accuracy collapse. On the infrastructure side it shows up as idle accelerators. Both are invisible until the moment the system meets real data, and by then the spend is committed. Retrieval-augmented workloads add a second flavor of the same trap. A RAG or agentic-retrieval pipeline that hits a vector store and object storage for context at inference time produces its own sustained random-read load. Validated on a toy corpus, it never surfaces the tail latency that a production-scale index produces — the same curated-to-production gap, one layer down. What Storage Checks Belong in a Data-Readiness Review? Because the bottleneck is invisible until production, the only reliable defense is to test the access pattern before committing the hardware budget. A storage-throughput check belongs in the data-readiness pass that precedes prototyping, not after it. The checklist is short and concrete: Estimate demanded samples-per-second from your target model, batch size, and per-step compute time. This is the number storage must sustain per accelerator, multiplied by your accelerator count. Characterize the real access pattern — sample size distribution, sequential versus random, metadata operation rate — from the actual production dataset, not the curated prototype set. Map to the closest MLPerf Storage workload and read the sustained accelerator count for storage comparable to your candidate tier. Discount for loader efficiency. Assume your production data-loading pipeline is less efficient than the benchmark’s replay and plan headroom accordingly. Test at production scale, or at least production access pattern, before purchase — replay a representative random-read load against candidate storage rather than trusting the curated prototype’s behavior. We treat storage-throughput sizing as one of the data-readiness checks in a generative AI feasibility assessment, precisely so that an I/O bottleneck is caught while it is still a planning decision rather than a committed cost. The same discipline extends beyond GenAI — storage sizing is a general AI-project readiness concern, and it sits inside the broader question of whether a project is ready to build at all. For teams weighing that broader readiness question, an R&D engagement scoped to your problem is where storage sizing joins the rest of the feasibility picture. FAQ What should you know about MLPerf Storage in practice? MLPerf Storage, from MLCommons, emulates a cluster of accelerators using the DLIO tool to replay a real training workload’s I/O access pattern, with a sleep model standing in for accelerator compute time. It reports the maximum number of accelerators a storage system can keep above a utilization threshold (typically around 90%) for a given workload. In practice it answers a single question: can this storage keep accelerators fed for a specific model, not how much raw bandwidth it has. What does the MLPerf Storage benchmark actually measure — throughput, accelerator utilization, or both? It measures both, but the headline result is expressed as an accelerator count at a sustained utilization target, not as a bandwidth figure. The benchmark’s job is to find how many emulated accelerators a storage system can hold above the utilization threshold for a named workload. Bandwidth is an input to that; the sustained accelerator count is the output you should plan against. How do I read MLPerf Storage results against my own model and batch size rather than taking the headline number? Match your workload to the closest benchmarked access pattern (large sequential LLM-style versus random small-file vision-style), confirm the emulated accelerators are comparable to what you will deploy, then read the sustained accelerator count rather than the bandwidth. Fit your planned cluster under that number with headroom. Finally, discount the result because your production data loader is usually less efficient than the benchmark’s idealized replay. Why does storage I/O become a hidden bottleneck in GenAI training and retrieval pipelines? Prototypes are validated on small curated datasets that fit in cache, so the pipeline never exercises the sustained random-read pressure that production-scale data imposes. When production data arrives — millions of samples read randomly from shared storage — accelerators start stalling on I/O. Retrieval-augmented pipelines hit the same trap when a toy corpus hides the tail latency a production-scale index produces. How does the curated-data vs production-data gap show up at the storage layer, and how do I catch it before spend? At the storage layer the gap appears as idle accelerators: a pipeline that ran GPUs near saturation on curated data stalls on I/O once production-scale random reads dominate. It is the infrastructure-side version of the data-quality failure where a model that worked on curated data collapses on production data. Catch it by characterizing the real production access pattern and testing it against candidate storage before purchase, not after. What storage-throughput checks belong in a data-readiness review before prototyping? Estimate the demanded samples-per-second from your model, batch size, and per-step compute time; characterize the real production access pattern including small-file and metadata pressure; map it to the closest MLPerf Storage workload; discount for loader inefficiency; and test a representative random-read load against candidate storage before committing budget. These checks turn an otherwise invisible bottleneck into a planning decision. The question worth answering first The instinct to sort a result table by the largest number survives because bandwidth feels like a property of the storage system. It is not — sustained throughput is a property of the storage system and the workload and the loader together, which is exactly why the benchmark emulates a specific model rather than reporting a single figure. Before you compare vendors, answer the harder question: what samples-per-second does your actual model at your actual batch size demand, and have you exercised that access pattern at production scale? That question, resolved on paper, is the difference between accelerators that stay fed and accelerators that sit idle burning the budget you spent to buy them.
