Intel Optane worked because 3D XPoint was bit-addressable. That single mechanical property — no page-erase cycle, no block-granular rewrite — is why Optane returned low queue-depth random reads in single-digit microseconds, and why it did not degrade under small-block random I/O the way NAND flash does. Everything operationally useful about Optane follows from that sentence, and almost nothing useful follows from the marketing line (“non-volatile media, faster than NAND”), which tells you a direction without telling you which access pattern actually improved.

The distinction matters because it determines what you do next. If you know the mechanism, you go and measure your own data loader’s block size, queue depth and read randomness before concluding you need faster media at all. If you only know the direction, you buy the newest NVMe device, watch GPU utilisation stay flat, and eventually discover the stall was single-threaded JPEG decode on the host. We see that sequence often enough that it is worth writing the profiling procedure down.

What made 3D XPoint mechanically different from NAND flash?

NAND flash is organised in pages and erase blocks. You read at page granularity, you write at page granularity, and you erase at block granularity — which means a small random write is a read-modify-write against a much larger unit, and the controller has to garbage-collect behind you. That structure is why NAND’s random-read latency sits well above its sequential figures and why small-block random I/O queues behave badly as the device fills.

3D XPoint dropped the erase block. The cell could be addressed and rewritten in place, so the controller was not amortising a page-erase cycle across your request. Two consequences that are worth stating on their own:

• Optane’s advantage was latency at low queue depth, not peak sequential bandwidth. A NAND NVMe drive with a deep queue and large sequential reads can match or beat Optane on throughput specs; the gap opens at queue depth 1–4 with small random blocks.
• Optane’s random-read latency stayed roughly flat as block size shrank and randomness rose, whereas NAND degrades along both axes. This is the property that made the media interesting for AI data paths, because shuffled sample reads during training are precisely small-block, low-queue-depth, randomly addressed reads.

Optane also shipped in two operating modes, and only one of them was interesting for a data pipeline. In memory mode, the persistent memory modules presented as a large volatile-looking address space with DRAM acting as a cache — transparent to software, no application changes, and no persistence guarantee exposed. In app-direct mode, the application addressed the persistent region explicitly and could rely on its durability semantics. For AI pipelines, app-direct was the mode that mattered, because a loader or cache layer had to know it was talking to a byte-addressable persistent tier to exploit it. Memory mode mostly bought capacity, not a changed access profile.

The parent article covers where this tier sat in the memory hierarchy and why Intel retired the product line — our analysis of the GPU memory hierarchy and storage tiers is adjacent reading on where the cost of that idle time actually lands. This piece stays on the measurement procedure.

Which loader access patterns actually benefit from microsecond random reads?

Not all of them. The honest answer is that a minority of training pipelines are device-latency-bound, and the ones that are share a recognisable shape.

Loader pattern	Benefits from microsecond random reads?	Why
Shuffled reads of many small samples (individual images, short audio clips, per-record files)	Yes	Small blocks, queue depth often 1–4 per worker, fully random addressing
Sequential reads of packed shards (WebDataset, TFRecord, tar streams)	No	Large sequential reads; any NVMe saturates the link first
Random access into a memory-mapped index or embedding table larger than DRAM	Yes	Byte-addressable random reads are the whole workload
Re-reading a working set that fits in page cache after epoch 1	No	Device is out of the path after warm-up
Heavy per-sample CPU work (JPEG decode, resize, tokenise, augment)	No	Host compute is the constraint, not the device
Checkpoint writes during long training runs	Partly	Write latency matters, but this is bursty and rarely the step-time stall

The table is the fast filter. If your pipeline sits in the “No” rows, faster media will not move GPU utilisation regardless of what the media is, and no amount of 3D XPoint mechanics changes that.

Profiling the attribution split

The measurable outcome you want is an attribution split: what percentage of step time is data wait versus kernel execution, and of that wait, what share is device read latency versus host-side decode and collate. Getting that split right is the difference between a loader change costing engineer-days and a storage-tier purchase costing capital.

The procedure we use, in order:

1. Establish step-time composition first. Instrument the training loop with the PyTorch profiler (or the framework equivalent) and record time-to-first-batch, per-step wall time, and CUDA kernel time. If kernel time is close to wall time, stop — you have a compute-bound job and storage is irrelevant.
2. Separate wait from compute at the loader boundary. Time the next(iterator) call independently of the forward/backward pass. A synthetic-data run (same tensor shapes, no I/O, no decode) gives you a clean upper bound on achievable step rate; the gap between synthetic and real is your entire data-path cost.
3. Split host decode from device read. Re-run with the decode and augmentation stages replaced by no-ops while still reading the real bytes from disk. If the gap closes, the stall was CPU-side transform work. If it stays open, the bytes themselves are late.
4. Characterise the actual I/O, not the assumed I/O. Use blktrace/iostat or nvidia-smi dmon alongside biolatency (eBPF) to capture the real distribution of request sizes, queue depths and completion latencies your loader generates. Most teams are surprised here: the loader’s effective block size is often much larger than the sample size because of readahead, and the effective queue depth is much higher than 1 because of worker parallelism.
5. Only then reason about media. A device-latency bottleneck looks like this: request sizes at or below 16 KB, queue depth persistently in the single digits, completion latency dominating the loader’s wait, and the wait surviving step 3. Anything else points elsewhere.
6. Re-measure after every change. One variable at a time, same dataset, same shuffle seed.

Steps 2 and 3 are where the value is. In our experience, the majority of “we need faster storage” conclusions dissolve at step 3 because the real constraint was a single-threaded decode path, an under-provisioned num_workers setting, or a collate function doing per-sample Python work. That is an observed pattern across engagements, not a benchmarked rate — the point is where to look, not how often the answer lands.

If the bottleneck is real, what replaces Optane?

Assume profiling holds up: small blocks, shallow queues, device latency dominating. Optane is discontinued, so the question is what provides that access profile now.

• Restructure the access pattern. Pack samples into shards and shuffle within a large in-memory buffer. This converts random small reads into sequential large reads and is usually cheaper than any hardware change.
• Cache the working set in DRAM or in a host-side page cache you actually size for the dataset. If the dataset fits, the media question disappears after the first epoch.
• High-endurance low-latency NVMe with a shallow-queue focus. Modern enterprise NVMe has narrowed the gap at queue depth 1 considerably; measure candidates at your block size and queue depth rather than at the vendor’s sequential figures.
• CXL-attached memory tiers, where the platform supports them, are the architectural successor to the byte-addressable persistent-memory idea, though the durability semantics differ from app-direct Optane.
• Keep the GPU fed asynchronously. Overlapping H2D copies on a separate CUDA stream, pinned memory, and prefetch depth greater than one hide latency you cannot remove.

One caveat on the older numbers. Because 3D XPoint was retired, Optane benchmark results you find online were produced on platforms and driver stacks that are no longer current, and they were frequently published as latency-at-QD1 comparisons against consumer NAND rather than against contemporary enterprise NVMe. Treat them as evidence about the mechanism — bit-addressable media gives flat small-block random-read latency — and not as a live procurement comparison.

If you inherited a machine with Optane memory enabled as a caching tier, the same procedure answers whether it is doing anything: run step 4 with the cache enabled and disabled and compare the completion-latency distribution your loader actually sees. A cache that never shows up in the latency histogram is not contributing to your workload, whatever the BIOS reports. Our GPU performance and optimisation work starts from exactly this kind of attribution measurement rather than from a hardware shortlist.

The uncomfortable part of all this is that the attribution split is cheap to measure and rarely measured. Which raises the question worth carrying into your next infrastructure review: of the last three hardware purchases your team justified with “the data path is too slow”, how many were preceded by a synthetic-data run?

Frequently Asked Questions

What made 3D XPoint mechanically different from NAND flash, and why did that matter for random-read latency?

3D XPoint cells were bit-addressable and rewritable in place, so there was no page-erase cycle and no block-granular read-modify-write behind a small request. That removed the main source of NAND’s random-read latency penalty, which is why Optane returned single-digit microsecond reads at queue depth 1 and held roughly flat latency as block size shrank and addressing became more random.

How did Optane behave differently in memory mode versus app-direct mode, and which one mattered for AI data pipelines?

Memory mode presented the persistent modules as a large volatile-looking address space with DRAM in front as a cache — transparent to applications, but effectively a capacity play with no exposed durability. App-direct mode let the application address the persistent region explicitly with durability semantics, and that is the mode that mattered for data pipelines, because a loader or cache layer has to know it is talking to a byte-addressable persistent tier to exploit it.

Which access patterns in a training data loader actually benefit from microsecond-scale random reads?

Shuffled reads of many small individual samples and random access into a memory-mapped index or embedding table larger than DRAM both benefit, because they are genuinely small-block and shallow-queue. Sequential reads from packed shards, warm working sets served from page cache, and pipelines dominated by per-sample decode or augmentation do not — in those cases the device is not on the critical path.

How do I profile whether my GPU stalls come from device read latency, host-side decode, or the kernel itself?

Compare three runs: the real pipeline, a synthetic-data run with identical tensor shapes and no I/O, and a run that reads real bytes but replaces decode and augmentation with no-ops. The synthetic run bounds achievable step rate, the no-op-transform run separates host CPU work from byte arrival, and biolatency or blktrace alongside the framework profiler tells you the request sizes, queue depths and completion latencies your loader is actually generating.

Why was 3D XPoint discontinued, and does that change how I read Optane benchmark numbers I still find online?

Intel retired the product line, so published Optane results were produced on platforms and driver stacks that are no longer current, and many compared latency at queue depth 1 against consumer NAND rather than contemporary enterprise NVMe. Read them as evidence about the mechanism — bit-addressable media gives flat small-block random-read latency — not as a live comparison against devices you can buy today.

When is the correct answer “change the data pipeline” rather than “buy faster media”?

Whenever the profiling gap closes once decode and augmentation are stubbed out, or whenever the loader’s effective requests turn out to be large and sequential rather than small and random. Sharding plus in-buffer shuffling, more loader workers, pinned memory and deeper prefetch are engineer-days; a storage tier is capital, and it only pays back when completion latency at your real block size and queue depth is the thing dominating the wait.

Profile wear leveling under write amplification

3D XPoint cells endure fewer program cycles than NAND; track write volume per dimm and monitor for early retirement before your swap tier degrades. The teams that do tend to ship the boring, correct version first.