A recommendation model looks, on paper, like any other feed-forward network. So when latency creeps up, the reflex is to reach for the same tools you would use on a vision or language model — kernel fusion, a faster runtime, a lower precision. On a Deep Learning Recommendation Model (DLRM), that reflex usually misses the target. Most of the wall-clock time in a DLRM inference call is not spent in the dense matrix multiplies the GPU is good at. It is spent gathering sparse embedding rows from memory, assembling features on the host, and moving data across the interconnect. That single fact reshapes almost every optimisation decision that follows. If your latency lives in memory-bound embedding lookups, then a faster kernel or a lower-precision MLP touches the wrong constraint — and the port or accelerator you were about to buy may not move the number that matters. Understanding what a DLRM is made of is the precondition for attributing its latency to the right layer. It is not a substitute for measuring, but you cannot measure meaningfully without the map. How does DLRM work in practice? DLRM is Meta’s open reference architecture for click-through-rate and ranking-style recommendation. Its job is to take a mix of dense inputs (continuous numbers — a user’s age, a session dwell time) and sparse inputs (categorical IDs — which product, which advertiser, which user cohort) and produce a single relevance score. The dense features go through a small bottom multi-layer perceptron (MLP). The sparse features do something quite different: each categorical ID is used as an index into an embedding table, pulling out a learned vector for that specific category. Those embedding vectors, plus the processed dense features, are then combined in a feature interaction step — typically pairwise dot products — before a top MLP produces the final score. In practice this means a DLRM is not one homogeneous compute graph. It is three distinct kinds of work stitched together, and each one stresses a different part of the machine. Treating the whole thing as “a neural network” is exactly how teams end up optimising the 10% that was never the problem. The same failure class shows up whenever a workload’s cost is assumed rather than measured — we cover the general version in what “computationally expensive” actually means in an inference path. What are the components of a DLRM? Three parts, three very different hardware profiles. This is the map you attribute latency against. Component What it does Dominant cost Typical bottleneck Embedding tables Gather a learned vector per categorical ID Random memory access (gather) Memory capacity + bandwidth; often host RAM or interconnect Feature interaction Pairwise dot products across embeddings + dense features Small, irregular compute Layout / data movement, not FLOPs Bottom + top MLPs Dense matrix multiplies GPU dense compute GPU tensor throughput — the part CUDA/cuDNN handle well The embedding tables are the surprising part for anyone coming from vision or language work. They can be enormous — production recommendation tables routinely run to tens or hundreds of gigabytes, far larger than the model’s weight matrices. And the access pattern is a sparse gather: for a given request you touch a tiny, scattered subset of rows. That is a random-access memory workload, not a throughput-friendly matrix multiply. The MLPs, by contrast, are exactly the dense linear algebra a GPU exists to accelerate — and they are usually a small share of the total time. Why is DLRM inference typically memory-bound rather than compute-bound? Because the arithmetic intensity of an embedding gather is close to zero. You read a vector, and you do almost no math per byte read. Compute-bound work — a large matmul — does many floating-point operations per byte fetched from memory, so it can hide memory latency behind arithmetic. An embedding lookup cannot: there is nothing to hide behind. Two things follow. First, embedding tables that exceed GPU HBM capacity have to live in host DRAM (or be sharded across devices), which means every lookup can pay a PCIe or NVLink crossing. Second, even when a table fits in HBM, the random-access gather does not saturate the bandwidth the way a streaming kernel would. The GPU’s advertised peak FLOPs are irrelevant to this phase — what matters is memory capacity, bandwidth, and interconnect topology. This is the same reasoning behind why DGX Spark’s memory bandwidth, not its peak FLOPs, sets the ceiling for many inference bottlenecks, and it is why capacity-versus-bandwidth trade-offs deserve their own analysis. The shape of that ceiling is a structural property of the workload, not of a particular chip (observed pattern across recommendation-serving stacks we have profiled; not a single benchmarked figure). If you know the model is memory-bound, you already know that a compute upgrade will disappoint before you run it. How do you tell whether DLRM latency lives in embedding lookup, dense compute, or host glue? You measure the three layers separately. The trap is that a single end-to-end latency number tells you nothing about which layer to touch. Here is a diagnostic ordering that isolates each contributor. DLRM latency-attribution checklist Instrument the request boundary. Wrap the full inference call and record wall-clock time. This is your denominator — everything below is a share of it. Time the embedding gather in isolation. Feed a fixed feature vector and time only the lookup + interaction. If this dominates, you are memory-bound and no MLP optimisation will help. Time the MLPs on-device. Run the bottom and top MLPs on synthetic dense input. If dense compute is a small slice, GPU kernel tuning is not your lever. Measure the host feature-assembly path. Time the Python (or C++) code that parses the request, hashes categorical IDs, and marshals tensors before the model runs. This “glue” is invisible to GPU profilers and frequently the largest single share. Watch the interconnect. If tables live off-device, use nvidia-smi / NSight to see PCIe or NVLink traffic per request. Crossings here are pure overhead. Step 4 is where most teams are surprised. A great deal of recommendation-serving latency lives in host-side glue — request parsing, hashing, tensor construction — that never shows up in a CUDA profiler because the GPU is idle while it runs. If you are staring at a low GPU utilisation number and concluding the model is “efficient,” check the host first. Our walkthrough on profiling the Python inference path before a port covers how to catch exactly this class of hidden cost. Why does treating DLRM like a standard feed-forward network mislead a port or acceleration decision? Because the port decision is really a question about which layer you are moving and whether it was the bottleneck. A vision or language model is compute-dominated, so porting it to a faster accelerator or a fused-kernel runtime like TensorRT tends to pay off directly. A DLRM is memory- and host-dominated, so the same move can produce almost no end-to-end improvement while the embedding gather and the Python glue sit untouched. Consider a worked example. Suppose a team measures 12 ms per request and, treating the model as a feed-forward network, plans to move the MLPs to a lower-precision TensorRT engine expecting a large win. If the attribution breakdown shows the MLPs are, say, roughly 2 ms of the 12 — with the embedding gather and host assembly owning the rest — then even halving MLP time buys about 1 ms, under 10% end-to-end. That is the cost of porting the wrong layer: real engineering spent on the part that was never the constraint (illustrative figures; the point is the ratio, which you must measure per model, not the specific numbers). The correct frame is to let the attribution decide the port. If embedding lookup dominates, the levers are memory-side: table sharding, quantised embeddings, better caching of hot rows, or a memory-tiering strategy — the same tiering logic we discuss in the context of memory tiering and GPU data-feed bottlenecks. If host glue dominates, the lever is the serving path — moving feature assembly out of Python, batching, or precomputing. Only when dense compute genuinely dominates does a kernel-level or precision port move the number. How does DLRM’s structure map onto the inference-engine layers? It helps to think of any served model as three stacked engines: model compute (the tensor math a framework like PyTorch dispatches to CUDA/cuDNN), runtime (the serving layer, batching, scheduling, device transfers), and host glue (everything in application code before and after the model call). DLRM spreads its cost across all three in a way that vision models do not. Engine layer Where DLRM work lands What “optimising” it means Model compute MLPs; part of feature interaction Kernel fusion, precision, TensorRT — helps only if this layer dominates Runtime Embedding gather, device transfers, batching Table sharding, caching, batch shaping, interconnect layout Host glue Request parsing, ID hashing, tensor assembly Move off Python, precompute, vectorise — often the biggest untapped win The reason this matters for a decision-maker is that the three layers have different owners, different tools, and different cost profiles. An accelerator vendor sells you model-compute speedups. A DLRM that is runtime- or host-bound will not feel them. Getting the mapping right before the port conversation is the whole point — it is the profiling baseline that a serious cost-reduction effort starts from, and it feeds directly into the work we scope in the inference cost-cut sprint. The broader GPU-engineering context lives on our GPU practice page. FAQ What’s worth understanding about DLRM first? DLRM takes dense continuous features through a small MLP and sparse categorical IDs through embedding-table lookups, combines them in a feature-interaction step, and produces a relevance score via a top MLP. In practice it is three distinct kinds of work — memory-bound gathers, irregular interaction, and dense compute — not one homogeneous network, which is why it cannot be optimised like a single feed-forward model. What are the components of a DLRM — embedding tables, feature interaction, and the dense MLP? Embedding tables gather a learned vector per categorical ID (a random-access memory workload, and often tens to hundreds of GB in production). Feature interaction does pairwise dot products across those vectors and the dense features. The bottom and top MLPs are the dense matrix multiplies the GPU accelerates well — usually a small share of total time. Why is DLRM inference typically memory-bound rather than compute-bound? Because an embedding gather has near-zero arithmetic intensity: it reads a vector and does almost no math per byte, so there is nothing to hide memory latency behind. Large tables spill to host DRAM and pay PCIe or NVLink crossings, and even in-HBM gathers do not saturate bandwidth like a streaming kernel. Peak GPU FLOPs are largely irrelevant to this phase. How do you tell whether DLRM latency lives in embedding lookup, dense compute, or host feature-assembly glue? Measure the three layers separately rather than reading one end-to-end number. Time the embedding gather in isolation, time the MLPs on synthetic dense input, and — critically — time the host-side feature-assembly path, which is invisible to GPU profilers and often the largest share. Watch interconnect traffic if tables live off-device. Why does treating DLRM like a standard feed-forward network mislead a port or acceleration decision? A vision or language model is compute-dominated, so porting to a faster accelerator or fused-kernel runtime pays off directly; a DLRM is memory- and host-dominated, so the same move can barely change end-to-end latency while the gather and Python glue sit untouched. Porting the MLPs when they are a small slice of total time spends real engineering on the part that was never the constraint. How does DLRM’s structure map onto the inference-engine layers — model compute, runtime, and host glue? The MLPs and part of feature interaction land in model compute; the embedding gather, device transfers, and batching land in the runtime layer; request parsing, ID hashing, and tensor assembly land in host glue. DLRM spreads cost across all three, whereas vision models concentrate it in model compute — which is why accelerator speedups aimed at model compute often disappoint on recommendation stacks. What can understanding DLRM’s structure let you measure before committing to a port? It lets you attribute measured serving latency to the correct layer: the share owned by memory-bound embedding gather, the fraction spent in Python feature assembly, and the slice that is genuinely dense GPU compute. With that breakdown you can avoid porting a compute path that was never the bottleneck in a memory-bound model. The uncomfortable part is that none of this is knowable from the architecture diagram alone. The structure tells you where latency can hide; only measurement tells you where it actually sits for your traffic, your table sizes, and your serving path. When a recommendation stack goes to production — or gets ported to a new accelerator — the release-readiness question is whether the deployed path still behaves like the profiled one, since memory-bound embedding lookups and host-side assembly are precisely the behaviours that shift under real load. Treat DLRM’s layer map as the checklist you profile against, not the answer.
