A DGX Spark spec sheet shows a headline memory bandwidth figure, and the natural instinct is to treat throughput as something that scales with it. It doesn’t work that way. On unified-memory hardware, bandwidth is a shared budget that weights, activations, and data staging all draw from at once — and many AI workloads exhaust that budget long before the compute cores get busy. The number on the slide tells you the ceiling. It says almost nothing about whether your workload will ever reach it. That gap is where teams overspend. A GPU can report a high busy percentage while it is, in practice, idling on memory transfers — waiting for the next tile of weights to arrive rather than doing useful arithmetic. If you profile only the busy counter, the hardware looks fully utilised. If you profile effective throughput per useful FLOP, you often find you are paying for compute that never gets fed. Understanding this distinction is what separates a bandwidth figure on a slide from a system that actually uses the silicon you bought. How does NVIDIA DGX Spark memory bandwidth work in practice? Memory bandwidth is the rate at which data can move between the memory system and the processing units. On a discrete GPU, that rate is set by the on-package HBM or GDDR stack and its bus width, and the CPU sits across a PCIe link with its own, much slower, path to system DRAM. On DGX Spark, the architecture is different: CPU and GPU share a single coherent memory pool. There is no separate “GPU memory” you copy into before a kernel runs — both processors address the same physical bytes. This coherent, unified design removes an entire class of explicit host-to-device copies, which is genuinely useful for workloads that were previously bottlenecked on PCIe transfers. But it does not create bandwidth out of nothing. The same pool now serves CPU-side data preparation, GPU kernel reads and writes, and any staging in between. The headline bandwidth figure is the aggregate capacity of that shared pool — not a private allocation the GPU gets to itself. When the CPU is busy decoding a batch of inputs while the GPU is streaming model weights, both draw from the same budget, and the effective bandwidth available to either one is lower than the spec number implies. The practical consequence: on DGX Spark, the bottleneck lands in a different place than it does on a discrete-GPU box. A workload that was compute-bound on a system with dedicated HBM can become bandwidth-bound on unified memory, because the contention model changed. This is the same reasoning we walk through for capacity in what a 128GB GPU means in practice — a big number on the spec sheet answers a different question than “will this feed my kernels.” What is unified memory bandwidth, and how does it differ from discrete-GPU bandwidth? The word “unified” is doing a lot of work, and it’s worth being precise about it. Unified memory on DGX Spark means one coherent address space shared by CPU and GPU, backed by one physical memory system. That is architecturally distinct from the software-managed unified virtual memory abstraction on discrete GPUs, where CUDA migrates pages between separate physical pools behind the scenes — a mechanism we cover in how unified virtual memory works for GPU inference. On DGX Spark the sharing is physical, not a migration illusion. Here is the difference that matters for planning. Dimension Discrete GPU (HBM/GDDR + system DRAM) DGX Spark unified memory Physical pools Two separate pools, two bandwidth budgets One shared pool, one bandwidth budget CPU→GPU data path PCIe copy (explicit or migrated) Coherent access, no copy Who contends for bandwidth GPU kernels dominate the device pool CPU prep and GPU kernels share the pool Where a copy-heavy workload stalls PCIe transfer Aggregate memory bandwidth Failure mode to watch Transfer overhead hiding real throughput Contention starving the GPU while busy% stays high The upshot is that unified memory trades one problem for another. It eliminates the PCIe bottleneck that plagues copy-heavy pipelines on discrete hardware, which is a real win for pipelines that shuffle a lot of data between host and device. In exchange, it puts every consumer of memory on the same budget line. If your data preparation is heavy — video decode, tokenisation, feature staging — that work now competes directly with your model’s weight and activation traffic. How do I tell whether my workload is memory-bandwidth-bound or compute-bound? This is the question that actually changes decisions, and it has a concrete answer. The roofline mental model is the cleanest way to reason about it: every kernel has an arithmetic intensity — FLOPs performed per byte moved from memory. If a kernel’s arithmetic intensity is below the hardware’s compute-to-bandwidth ratio, it is memory-bound; above it, compute-bound. The hardware ratio is roughly its peak FLOP rate divided by its memory bandwidth, both from published specifications. You do not have to derive this by hand. Profilers surface it directly. In practice the diagnostic sequence looks like this: Check GPU busy percentage against effective throughput. If busy% is high but tokens-per-second or images-per-second is well below what the peak FLOP rate would predict, that divergence is the first signal of a memory bottleneck (observed pattern across our GPU audit engagements; not a single benchmarked figure). Read the memory-throughput counters. Nsight Compute and nvidia-smi dmon expose achieved DRAM/bandwidth utilisation. A kernel pinned near peak memory throughput while its compute pipes sit half-idle is bandwidth-bound by definition. Look at arithmetic intensity per kernel. Nsight Compute’s roofline view places each kernel above or below the ridge point. Attention and elementwise kernels usually sit left of the ridge (memory-bound); large dense GEMMs usually sit right of it (compute-bound). Vary batch size and watch what moves. If doubling batch size roughly doubles throughput, you had spare compute and were latency-limited; if throughput barely moves, you have hit the bandwidth wall. The reason this matters more on DGX Spark than on a discrete box is that the ridge point shifts. Sharing bandwidth with the CPU effectively lowers the bandwidth the GPU sees, which moves the ridge to the right — meaning more of your kernels fall into the memory-bound region than they would on dedicated HBM. The same model can be compute-bound on one platform and bandwidth-bound on another. Why can a GPU show high busy percentage while bandwidth is the real bottleneck? Because “busy” is not “productive.” The busy percentage reported by nvidia-smi counts intervals during which at least one kernel was resident and executing — it does not distinguish a kernel doing dense arithmetic from a kernel stalled waiting for operands to arrive from memory. A memory-bound kernel is still technically running; its warps are just waiting on load instructions. The busy counter ticks up all the same. This is one of the most common misreadings we see when a team hands over a dashboard and says the GPUs are “at 95%.” The number is real, but it is answering the wrong question. High utilisation on that counter can coexist with the compute units being starved most of the time. The metric that actually reflects value is throughput per useful FLOP: how much real work completed relative to the theoretical peak the hardware could sustain if it were fed. We unpack the broader version of this trap in reading DGX Spark utilisation, not just peak FLOPs, and it’s the exact gap a proper GPU performance audit is designed to expose. Naming this correctly is not pedantry. If your dashboard says 95% and you conclude you need more GPUs, you buy compute that will be starved the same way the current compute is. The bottleneck follows you to the new hardware. Which workload patterns saturate DGX Spark memory bandwidth before compute? Some workloads are structurally bandwidth-hungry. Recognising the pattern early tells you what to expect before you profile. Autoregressive LLM decoding. Generating tokens one at a time means re-reading the entire model’s weights (and the growing KV cache) for every single token, with very little arithmetic per byte. Decode is almost always memory-bound; this is why techniques like KV-cache reuse target bandwidth, not FLOPs. Small-batch inference. At batch size one or two, there isn’t enough parallel work to amortise the cost of streaming weights. The arithmetic intensity is low and the GPU spends its time fetching rather than computing. Attention and elementwise-heavy layers. Softmax, layer norm, activation functions, and unfused attention move a lot of data relative to the FLOPs they perform. Kernel fusion (FlashAttention being the canonical example) exists precisely to cut the memory traffic these layers generate. Heavy CPU-side preprocessing running concurrently. On unified memory this is the DGX-Spark-specific trap: video decode or tokenisation on the CPU draws from the same pool the GPU is reading weights from, so the GPU’s effective bandwidth drops even though its own workload didn’t change. By contrast, large-batch training with big dense matrix multiplies, or convolution-heavy vision backbones at high resolution, tend to be compute-bound — they perform enough arithmetic per byte to keep the tensor cores fed. The dividing line is arithmetic intensity, and it is workload-specific, not hardware-specific. What levers recover throughput when bandwidth is the ceiling? Once you know a workload is bandwidth-bound, the optimisation levers are different from the ones you’d reach for if it were compute-bound. Throwing more or faster compute at a bandwidth wall does nothing. These do: Batch sizing. Larger batches raise arithmetic intensity by amortising each weight read across more work — provided the batch fits in the memory budget and doesn’t blow your latency target. There is a sweet spot, and it moves with the model. Quantisation. Lower-precision weights (FP8, FP4) move fewer bytes per parameter. Because decode is bandwidth-bound, quantisation frequently buys throughput roughly in proportion to the byte reduction — for a memory-bound kernel, halving weight width can approach halving read time. We cover the trade-offs of aggressive quantisation in what FP4 means for GPU utilisation. Kernel fusion. Fusing elementwise and attention operations keeps intermediates in registers or shared memory instead of round-tripping through DRAM. FlashAttention, torch.compile, and TensorRT graph fusion all attack the same problem: fewer bytes moved. Data staging discipline. On DGX Spark specifically, scheduling CPU-side preprocessing so it doesn’t contend with peak GPU read phases can recover bandwidth the GPU was silently losing. This is a placement problem the unified model makes visible. The point of naming the bottleneck first is that it selects the lever. Quantising a compute-bound workload wastes effort; adding compute to a bandwidth-bound one wastes budget. Get the diagnosis right and the fix is often a configuration change rather than a hardware purchase. How does memory bandwidth relate to cost per useful FLOP? Total cost of ownership per useful FLOP is the metric that ties all of this together. You pay for a fixed amount of peak compute when you buy DGX Spark hardware. If bandwidth caps your effective throughput at a fraction of peak, your real cost per unit of useful work is that same fraction higher than the spec-derived figure suggests — you are amortising the full hardware cost over a workload that only reaches part of the machine’s capability. This is the same cost-per-useful-FLOP lens we apply to cloud instances in our GPU landing page and across the utilisation-cost cluster. The team that profiles bandwidth utilisation before procuring avoids the most expensive version of this mistake: buying compute they will never feed. If your workload is autoregressive decoding at small batch sizes, the constraint is bandwidth, and the questions that matter are memory bandwidth per dollar and whether quantisation can shrink your byte budget — not peak TFLOPs. The spec sheet’s headline number is the wrong figure to anchor a purchase decision on when your workload lives left of the ridge point. FAQ How does NVIDIA DGX Spark memory bandwidth work? DGX Spark uses a coherent unified memory pool shared by CPU and GPU, so the headline bandwidth figure is the aggregate capacity of that shared pool, not a private budget for the GPU. In practice it means CPU-side data preparation and GPU kernel traffic draw from the same budget, and the effective bandwidth available to a kernel can be lower than the spec number when both are active. What is unified memory bandwidth on DGX Spark, and how does it differ from discrete-GPU memory bandwidth? Unified memory on DGX Spark is one physical, coherent pool addressed by both CPU and GPU, eliminating the explicit PCIe copies that discrete GPUs need between separate host and device pools. The trade-off is that a discrete GPU gives kernels a dedicated device-memory budget, whereas on DGX Spark every consumer — CPU prep and GPU compute alike — contends for a single shared bandwidth budget. How do I tell whether my workload is memory-bandwidth-bound or compute-bound on DGX Spark? Compare GPU busy percentage against effective throughput, then read the memory-throughput counters in a profiler like Nsight Compute: a kernel near peak memory throughput while its compute pipes sit idle is bandwidth-bound. The roofline view places each kernel above or below the ridge point, and varying batch size confirms it — if throughput barely moves when you double the batch, you’ve hit the bandwidth wall. Why can a GPU show high busy percentage while memory bandwidth is the real bottleneck? The busy counter registers any interval where a kernel is resident and executing, without distinguishing productive arithmetic from warps stalled waiting on memory loads. A memory-bound kernel is still technically running, so busy% stays high even while the compute units are starved — which is why throughput per useful FLOP, not busy percentage, is the metric that reflects real value. Which workload patterns are most likely to saturate DGX Spark memory bandwidth before compute? Autoregressive LLM decoding, small-batch inference, attention and elementwise-heavy layers, and heavy CPU-side preprocessing running concurrently are the usual culprits, because they perform little arithmetic per byte moved. On unified memory the concurrent-preprocessing case is the DGX-Spark-specific trap: CPU work draws from the same pool as GPU weight reads, lowering the bandwidth the GPU effectively sees. What levers — batch sizing, quantisation, data staging — recover throughput when bandwidth is the ceiling? Larger batches raise arithmetic intensity by amortising weight reads; quantisation (FP8, FP4) moves fewer bytes per parameter and often buys throughput roughly in proportion to the byte reduction for memory-bound kernels; kernel fusion keeps intermediates off DRAM; and scheduling CPU preprocessing to avoid contending with peak GPU read phases recovers bandwidth on unified memory. Adding compute does nothing to a bandwidth wall — the diagnosis selects the lever. How does memory bandwidth relate to total cost of ownership per useful FLOP on DGX Spark? You pay for peak compute up front, so if bandwidth caps effective throughput at a fraction of peak, your real cost per unit of useful work rises by that same fraction. Profiling bandwidth utilisation before procuring avoids buying compute you can never feed — for a bandwidth-bound workload the questions that matter are memory bandwidth per dollar and whether quantisation shrinks your byte budget, not peak TFLOPs. Where this leaves a procurement decision The honest summary is that DGX Spark’s unified memory changes where the bottleneck lands, and the only way to know where it lands for your workload is to measure arithmetic intensity and effective throughput, not to read the spec sheet. Autoregressive decoding will be bandwidth-bound; large dense training may not be. The platform that looks best on paper depends entirely on which side of the ridge point your kernels sit — and that is a property of your model, not the marketing. If your dashboard shows high GPU busy percentage but throughput you can’t explain, the failure class is a memory-bandwidth ceiling masquerading as full utilisation, and a GPU performance audit is what surfaces the gap between busy silicon and useful work.
