A DGX Spark spec sheet gives you two numbers people latch onto: a large unified-memory figure and a headline TFLOPS rating. Neither one tells you how fast your model will actually serve. The number that matters is the one your workload is bound by — and that is not something a spec sheet can tell you without knowing your model, your batch size, and your latency target. This is the recurring mistake with any desktop-class AI system built on Grace Blackwell silicon. The large unified memory pool is genuinely useful: it changes what fits locally. A model plus its KV cache that would have spilled off a consumer GPU can now sit resident in one address space, shared between the Grace CPU and the Blackwell GPU. But fitting a model and serving it fast are two different questions. Unified memory capacity does not automatically improve tail latency for a given model. The bottleneck that governs latency lives somewhere specific in your pipeline, and until you find it, the spec sheet is a distraction. What matters most about DGX Spark performance in practice? Think of DGX Spark as three resources that interact: compute (the GPU’s arithmetic throughput), memory capacity (how much fits without spilling), and memory bandwidth (how fast data moves between memory and the compute units). A given inference workload leans hard on one of these and is comparatively indifferent to the others. Reading DGX Spark performance means identifying which one your workload is pinned to. For most transformer decoding — the token-by-token generation phase of an LLM — the binding constraint is memory bandwidth, not compute. Each generated token requires reading the entire model’s weights out of memory at least once. At batch size one, the GPU’s arithmetic units sit largely idle, waiting on the memory system. This is why a device with abundant unified memory can hold a large model comfortably yet still generate tokens slowly: capacity solved the “does it fit” problem and did nothing for the “how fast” problem, because those are governed by different hardware resources. The prefill phase behaves differently. Processing a long prompt is compute-heavy and can saturate the arithmetic units, so prefill throughput tends to track the TFLOPS figure more closely. A workload that is dominated by short generations after long prompts, and one dominated by long generations after short prompts, will read the same hardware in opposite ways. We see this split constantly when profiling serving stacks — the two phases have such different resource signatures that treating “inference performance” as a single number is misleading. The SGLang prefill-decode disaggregation work exists precisely because these two phases want different treatment. What inference workloads does DGX Spark’s unified memory capacity actually make feasible locally? The honest answer is: workloads where fit was the blocker, not speed. A large unified-memory pool lets you run models locally that would otherwise require sharding across multiple discrete GPUs or renting a data-centre instance. That is a real capability change, and it maps onto a clear set of use cases. Large models at low concurrency. A model whose weights plus KV cache exceed a consumer card’s memory but fit in the unified pool becomes viable for single-user or low-request-rate serving — local development, a private assistant, an on-premise agent handling occasional queries. Long-context workloads. The KV cache grows linearly with context length and can dwarf the model weights at long contexts. Capacity headroom lets you hold long conversations or large documents resident without eviction. Multi-model residency. Keeping several models loaded simultaneously — a router model, a couple of specialists, an embedding model — is a capacity problem first. Unified memory lets them coexist without constant load/unload churn. What unified memory does not buy you is high sustained throughput under heavy concurrency. Once you push batch sizes up to serve many concurrent users, you re-enter a bandwidth- or compute-bound regime, and a desktop-class device’s ceiling arrives well before a data-centre GPU’s does. The capacity that made the model fit does nothing to raise that throughput ceiling. If you want the general framing of why capacity and the real bottleneck diverge, what a 128GB GPU means in practice covers the same trap from the memory-capacity angle. When is DGX Spark memory-bandwidth bound versus compute bound for AI inference? This is the diagnostic that decides everything downstream. Rather than guessing from specs, profile the actual pipeline. The rubric below is how we approach the classification in practice; it is an observed pattern from profiling inference stacks, not a benchmarked threshold table. Bottleneck diagnostic rubric Signal Likely bandwidth-bound Likely compute-bound Likely transfer/host-bound Phase Token generation (decode) Prompt processing (prefill) Pre/post-processing, data feed Batch size Small (1–4) Large Any GPU arithmetic utilisation Low despite high wall-clock time High and sustained Low, with GPU idle gaps What raising batch size does Improves throughput sharply Marginal or negative No change Where time is spent Memory reads of weights/KV Matrix multiplies PCIe/host copies, Python glue First thing to measure Achieved memory bandwidth vs peak Achieved TFLOPS vs peak Copy time and CPU-side latency The practical move: measure achieved memory bandwidth and achieved arithmetic throughput as fractions of the device’s published peaks. If decode is spending its time at high memory-bandwidth utilisation and low arithmetic utilisation, you are bandwidth-bound, and no amount of extra compute or capacity helps — quantisation (which shrinks the bytes read per token) is your lever. If prefill saturates the arithmetic units, you are compute-bound there. And if the GPU shows idle gaps while the host does work, you have a transfer or host-side bottleneck that lives outside the GPU entirely. The NVIDIA DGX Spark memory bandwidth breakdown goes deeper on why the bandwidth ceiling is usually the operative one for local LLM decode. What model sizes and quantisation levels fit within DGX Spark’s memory before latency degrades? Fit and latency degrade for different reasons, so treat them as two calculations. Fit is arithmetic you can do before you buy anything. Model weight footprint is roughly the parameter count times the bytes per parameter: FP16 is 2 bytes, INT8 is 1 byte, and 4-bit formats are half a byte. A model at 4-bit floating point (FP4) quantisation occupies roughly a quarter of its FP16 footprint. On top of the weights you must budget for the KV cache, which scales with the number of concurrent sequences and their context length — at long contexts this can rival or exceed the weight footprint. A worked example: a model with roughly 70 billion parameters at FP4 needs on the order of 35 GB for weights alone (illustrative, from the bytes-per-parameter arithmetic above), leaving the rest of the unified pool for KV cache and activations. Latency degrades before you hit the capacity wall, for two reasons. First, quantisation is a bandwidth lever and an accuracy trade-off: dropping to 4-bit roughly halves the bytes read per token versus 8-bit, which speeds up bandwidth-bound decode, but at some quantisation level the model’s output quality drops below what your application tolerates. Second, as you push batch size up to raise throughput, per-request latency climbs — the p99 you promised users erodes long before memory runs out. The right fit is the largest model at the highest quantisation that still meets your accuracy bar, served at the batch size that holds your p99 target, with capacity to spare. Capacity is rarely the first thing to break. How does DGX Spark inference latency compare to data-centre GPU serving for the same model? For a single request or low concurrency, a well-matched local device can be competitive on latency — sometimes better, because you have removed the network round-trip to a cloud endpoint. The gap opens under concurrency. A data-centre GPU carries far more memory bandwidth and compute, so it sustains higher throughput and holds lower tail latency as request volume climbs. Below is the decision surface we use when the question is “local device or scale out to server GPUs.” Local DGX Spark class vs data-centre GPU serving Factor DGX Spark class (local) Data-centre GPU serving Best at Low-concurrency, model-fits-locally, latency-sensitive single-user High sustained throughput, many concurrent users Latency at low load Competitive; no network hop Competitive; add network round-trip Latency under concurrency Degrades earlier (lower bandwidth/compute ceiling) Holds longer Data residency Stays on-premise Leaves your environment Cost shape Fixed capital, low marginal per-inference Variable; pay per hour/token Breaks down when Concurrency rises past the throughput ceiling Utilisation is low (paying for idle) The economic crossover is a utilisation question. A local device is fixed capital: if you keep it reasonably busy, cost-per-inference is low. A rented data-centre GPU is variable cost that is efficient at high, steady utilisation and wasteful when idle. Sporadic or private workloads often favour local; spiky high-volume workloads with strict tail-latency SLAs favour scaling out. This is the same reasoning we apply in the broader HGX vs DGX platform choice — the platform follows the workload’s concurrency and residency profile, not the other way round. When should I use a DGX Spark class device for inference rather than scaling out to server GPUs? Reach for the local device when three things are true at once: the model fits in unified memory at an acceptable quantisation, your concurrency is low enough that you stay under the device’s throughput ceiling at your p99 target, and either data residency or cost-at-low-utilisation makes the fixed-capital shape attractive. Miss any one of those and the case weakens. If your concurrency is high and your SLA is strict, you are buying a throughput ceiling you will hit, and scaling out to server GPUs is the correct call. If the model does not fit even at aggressive quantisation, capacity has ruled the device out before latency enters the conversation. And if the device would sit mostly idle, the fixed-capital advantage evaporates. The decision is workload-shaped, and it belongs upstream of the hardware purchase. Our GPU engineering practice treats this as a profiling problem first, a procurement problem second. How do I benchmark DGX Spark inference so the numbers hold up? The numbers that hold up are the ones measured under your real serving conditions, not vendor-favourable ones. A few conditions decide whether your benchmark is trustworthy: Warm up first. Discard the first runs. Cold-start effects — kernel compilation, cache population, radix / KV-cache priming — distort the numbers if you count them. Report percentiles, not just the mean. p50 hides tail behaviour; p99 is what your users feel. A serving decision made on mean latency is a serving decision made blind to the worst experiences. Sweep realistic batch sizes. Measure at the concurrency you actually expect, and at a couple of levels above it, so you can see where the throughput/latency curve bends. Match prompt and generation lengths to production. Prefill-heavy and decode-heavy traffic stress different resources; a benchmark with unrepresentative lengths measures the wrong bottleneck. Report tokens/sec (LLM) or images/sec (vision) at a stated batch size and percentile — a throughput number without those conditions is not comparable to anything. If you are running local LLMs specifically, the Ollama benchmark walkthrough shows how to turn these conditions into repeatable measurements you can trust for a serving decision. FAQ How does DGX Spark performance work? DGX Spark performance is governed by three interacting resources — compute, memory capacity, and memory bandwidth — and any given workload is pinned to one of them. In practice this means the headline TFLOPS and unified-memory numbers describe potential, not delivered latency; token generation is usually bound by memory bandwidth while prompt processing is bound by compute. Reading the spec sheet correctly means first identifying which resource your workload is bound by. What inference workloads does DGX Spark’s unified memory capacity actually make feasible locally? Workloads where fit was the blocker rather than raw speed: large models at low concurrency, long-context workloads whose KV cache would otherwise spill, and keeping several models resident at once. The unified pool changes what runs locally without sharding or renting cloud instances. It does not raise the sustained-throughput ceiling, so heavy-concurrency serving still hits a wall well before a data-centre GPU would. When is DGX Spark memory-bandwidth bound versus compute bound for AI inference? Token generation (decode) at small batch sizes is typically memory-bandwidth bound — the arithmetic units idle while weights are read from memory. Prompt processing (prefill) and large batches tend to be compute bound and track the TFLOPS figure. Classify by profiling: measure achieved memory bandwidth and achieved arithmetic throughput as fractions of peak, and see which one saturates while the other has slack. How does DGX Spark inference latency compare to data-centre GPU serving for the same model? At low concurrency a well-matched local device is competitive, and skipping the network round-trip can make it faster than a cloud endpoint. Under rising concurrency the data-centre GPU holds lower tail latency because it carries more bandwidth and compute; the local device’s throughput ceiling arrives sooner. The economic crossover is a utilisation question — fixed local capital wins when kept busy, variable cloud cost wins for spiky high-volume traffic. What model sizes and quantisation levels fit within DGX Spark’s memory before latency degrades? Estimate weight footprint as parameter count times bytes-per-parameter (2 for FP16, 1 for INT8, 0.5 for 4-bit), then add KV-cache budget that scales with concurrency and context length. Fit is arithmetic you can do before buying; latency, though, degrades earlier — quantisation trades bandwidth savings against accuracy, and raising batch size erodes p99 before capacity runs out. The right choice is the largest model at the highest quantisation that still clears your accuracy bar and holds your p99. When should I use a DGX Spark class device for inference rather than scaling out to server GPUs? Use it when three conditions hold together: the model fits at an acceptable quantisation, concurrency stays under the device’s throughput ceiling at your p99 target, and data residency or low-utilisation economics favour fixed capital. If concurrency is high with a strict SLA, if the model will not fit even quantised, or if the device would sit idle, scale out instead. The decision is workload-shaped and belongs upstream of the purchase. How do I benchmark DGX Spark inference so the numbers hold up — which batch sizes, percentiles, and warm-up conditions matter? Warm up and discard cold-start runs, report p50 and p99 rather than the mean, sweep batch sizes at and above your expected concurrency, and match prompt and generation lengths to production traffic. Always report throughput (tokens/sec or images/sec) at a stated batch size and percentile — an unconditioned number is not comparable. These conditions are what separate a decision-grade measurement from a vendor-favourable one. Where teams get this wrong is treating the purchase as the decision. The device is a consequence; the bottleneck is the decision. Profile the pipeline, name whether capacity, bandwidth, or compute is the binding constraint for your serving target, and only then ask whether a DGX Spark class device or a data-centre GPU is the tuple that meets it — which is exactly the question a GPU performance audit exists to answer before the hardware is bought.
