A GPU feeder thread pinned to “the right socket” is not the same thing as a feeder thread pinned to the right proximity domain. On modern CPUs the two can diverge, and the gap shows up as host-to-device bandwidth you paid for but never see. The ACPI System Resource Affinity Table (SRAT) is where the machine tells you how fine-grained that map really is — and on current server parts, it is finer than most schedulers, and most engineers, assume. The naive mental model is simple: a NUMA domain is a CPU socket, and the memory it owns is “local” to every core on that socket. Pin your DMA-issuing thread to the socket that hosts the GPU, allocate your staging buffer there, and you are done. That model was roughly true a decade ago. It is not true on a modern EPYC or Xeon with sub-NUMA clustering enabled, where a single socket is carved into several proximity domains — and where the L3 cache itself can appear as its own domain in the topology tables. What the ACPI SRAT actually describes The SRAT is a firmware-provided table that maps hardware resources — processors, memory ranges, and increasingly, cache and memory-side buffers — to proximity domains. A proximity domain is an abstract identifier: “resources tagged with the same domain number are close to each other.” The table does not say how close. That is the job of a second table, the System Locality Information Table (SLIT), which gives a distance matrix between every pair of proximity domains. A distance of 10 is the ACPI convention for “local”; larger numbers are relative penalties for reaching across. The distinction matters because the two tables answer different questions. The SRAT answers “which domain does this resource belong to”; the SLIT answers “how expensive is it to reach from one domain to another.” You need both to place a buffer correctly. Reading only the SRAT tells you a buffer is in domain 3; only the SLIT tells you that domain 3 is two hops away from the core issuing your DMA. On a socket with sub-NUMA clustering (Intel calls it SNC; AMD’s NPS partitioning does the analogous thing on EPYC), the firmware emits several proximity domains per socket. Each cluster owns a subset of the memory controllers and, critically, a subset of the last-level cache. When the platform also exposes memory-side caches or CXL-attached tiers, the SRAT can carry additional domains for those too. The result is that a two-socket box no longer has two NUMA domains — it can easily have eight or more. Why would a CPU expose an L3 slice as its own NUMA domain? Because on a chiplet or tiled design, the L3 is physically distributed. On EPYC, each core complex die (CCD) carries its own L3, and cross-CCD L3 access travels the Infinity Fabric. On a monolithic-but-tiled Xeon, sub-NUMA clustering splits the shared L3 into slices bound to specific memory controllers. In both cases the firmware is being honest: a core in cluster 0 that touches data resident in cluster 1’s L3 pays a real, measurable latency and bandwidth penalty. Modeling that penalty as a NUMA distance is the only way the OS scheduler and memory allocator can make locality-aware decisions. This is the same locality reasoning that governs how heterogeneous CPU/GPU/WASM targets divide inference work — the boundary just moves inside the host CPU instead of between accelerators. The physics is not new; the granularity is. Where this bites GPU data movement Here is the mechanism, stated plainly. A host-to-device transfer over PCIe (or NVLink-C2C on a Grace-class part) is a DMA operation. The GPU’s DMA engine pulls from a pinned host staging buffer. That buffer sits in some physical memory range, which belongs to some proximity domain. The PCIe root complex serving the GPU is itself attached to a specific domain — on a multi-die CPU, one CCD or one SNC cluster physically owns the lanes that reach that slot. When the staging buffer lives in a domain that is remote from the root complex, every byte the DMA engine pulls crosses the on-package fabric before it reaches the PCIe lanes. That crossing has finite bandwidth. If the fabric segment is narrower than the PCIe link, the link starves — you have a Gen5 x16 slot nominally capable of roughly 64 GB/s per direction (per the PCI-SIG Gen5 specification), fed through an inter-die hop that cannot sustain it. The observed effect, in configurations we have profiled, is that misplaced staging buffers cap effective host-to-device throughput well below the link’s nominal ceiling, and the GPU shows input-starvation stalls — kernels waiting on data that the transfer path cannot deliver fast enough (observed pattern across GPU data-movement engagements; not a published benchmark). The kernels themselves are untouched and correct. The bottleneck is entirely host-side placement. There is a second-order version of this that catches people even after they get the socket right. If the DMA-issuing thread and the staging buffer are in the same socket but different sub-NUMA clusters, they may not share an L3 slice. The thread’s writes into the buffer, and any last-minute repacking it does, now bounce across the intra-socket fabric — a smaller penalty than a cross-socket hop, but still enough to shave meaningful bandwidth off a saturated link. This is exactly the memory-tiering problem discussed in the context of Optane memory tiering and GPU data-feed bottlenecks: the feed path, not the compute, decides the ceiling. How do I read the SRAT/SLIT topology on Linux and place buffers correctly? The kernel parses the SRAT and SLIT at boot and exposes the result through sysfs and standard tooling. The practical path is a short sequence, not a single command. Diagnostic checklist: locating the GPU’s local domain Enumerate NUMA nodes and their distances. numactl --hardware prints the node list and the SLIT distance matrix. On a sub-NUMA-clustered box you will see more nodes than sockets, with intra-socket distances (often ~11–12) distinct from cross-socket distances (~20+). These numbers are firmware-reported, not measured by you — treat them as relative hints, not absolute bandwidth. Find the GPU’s PCIe attachment. lspci -D gives the GPU’s bus address; cat /sys/bus/pci/devices/<addr>/numa_node reports which proximity domain owns that root complex. A value of -1 means the firmware declined to tag it — a red flag that the topology is incomplete or SNC metadata was dropped. Confirm the CPU cores in that domain. /sys/devices/system/node/node<N>/cpulist lists the cores local to the GPU’s node. Your DMA-issuing thread belongs here. Pin thread and memory together. Use numactl --cpunodebind=<N> --membind=<N> for whole-process binding, or mbind() / set_mempref plus pthread_setaffinity_np() for per-thread control inside a larger process. The pinned staging buffer must be allocated after the memory policy is set, or it lands wherever the default first-touch policy put it. Verify with a transfer microbenchmark. Run your actual host-to-device copy and compare sustained throughput against the link’s nominal ceiling. A gap of more than a few percent that closes when you change the bind is a placement problem, not a hardware limit. The step people skip is (5). Reading the tables tells you what the firmware claims; the microbenchmark tells you what the silicon delivers. When the two disagree — and they sometimes do, because SLIT distances are coarse integers, not calibrated measurements — trust the measurement and adjust the placement. When host-side layout interacts with build settings, the GCC and host-code compiler flags that control pinned-memory allocation and alignment become part of the same tuning surface. A worked example, with explicit assumptions Assume a dual-socket EPYC host, NPS=2 (two NUMA domains per socket, four total), with a single GPU in a slot physically wired to the second CCD of socket 0. Call that domain node 1. Assume a Gen5 x16 link. Baseline (naive): feeder process launched without binding. First-touch places the staging buffer on node 0 (where the launching thread happened to start). The DMA path crosses from node 0 to node 1’s root complex. Sustained host-to-device throughput sits noticeably under the Gen5 ceiling, with periodic GPU input stalls. Socket-correct but cluster-wrong: process bound to socket 0 broadly, but memory lands on node 0 rather than node 1. Better than crossing sockets, still an intra-socket fabric hop. Some bandwidth recovered, not all. Domain-correct: thread and buffer both bound to node 1 with numactl --cpunodebind=1 --membind=1. The DMA engine now pulls from memory local to the root complex it feeds. Sustained throughput approaches the link ceiling; input-starvation stalls largely disappear. The numbers here are illustrative of the shape of the result, not a benchmark — the point is the ordering and the mechanism, which hold across the boxes we have characterized (observed pattern; your absolute figures depend on the specific CPU, fabric width, and PCIe generation). How does this relate to writing performance-portable GPU code? This is the host-side twin of a point that shows up everywhere in cross-vendor GPU work: performance does not travel with a portable API, and it does not travel with a naive NUMA assumption either. A kernel written once and compiled for two vendors can be perfectly correct on both and slow on both if the host feed path is misplaced. The API abstracts the compute; it does not abstract the memory topology underneath it. That is why host-side locality characterization belongs in the same audit as cross-platform performance work — it is one dimension of a GPU performance audit, alongside kernel-level and interconnect-level analysis. The uncomfortable part is that the correct placement is not portable across machines. A bind that saturates the link on one server is wrong on another with a different SRAT layout, a different NPS/SNC setting, or the GPU in a different slot. The durable artifact is not the magic node number — it is the procedure for reading the topology and validating the placement on each target. Bake that into deployment, not a hardcoded affinity mask. FAQ What should you know about acpi srat l3 cache as numa domain in practice? The ACPI SRAT maps hardware resources — processors, memory, and on modern CPUs, cache and memory-side buffers — to proximity domains, abstract “these are close together” identifiers. Because the L3 is physically distributed across chiplets or tiles, firmware can expose an L3 slice or sub-NUMA cluster as its own domain. In practice this means a single socket presents several NUMA domains, so “local” memory is finer-grained than the socket, and placement decisions have to account for it. What is the ACPI SRAT, and how does it differ from the SLIT distance table when describing memory affinity? The SRAT answers which proximity domain a resource belongs to; the SLIT answers how far apart two domains are, as a distance matrix where 10 is the local convention and larger numbers are relative penalties. You need both to place a buffer well: the SRAT tells you a buffer sits in domain 3, and the SLIT tells you domain 3 is two hops from the core issuing your DMA. Reading one without the other gives you an incomplete picture of memory affinity. Why would a CPU expose an L3 cache slice or sub-NUMA cluster as its own NUMA proximity domain? Because the L3 is physically distributed on chiplet and tiled designs, and cross-slice access pays a real latency and bandwidth penalty over the on-package fabric. Modeling that penalty as a NUMA distance is the only way the OS scheduler and allocator can make locality-aware decisions. The firmware is being honest about a cost that already exists in the silicon. How does host-side NUMA placement affect PCIe/NVLink host-to-device transfer throughput to a GPU? A host-to-device transfer is a DMA operation pulling from a pinned staging buffer; that buffer lives in some proximity domain, and the GPU’s PCIe root complex is attached to a specific domain too. When the buffer is remote from the root complex, every byte crosses the on-package fabric before reaching the PCIe lanes, and if that segment is narrower than the link, the link starves. The observed effect is effective throughput capped below the nominal ceiling with GPU input-starvation stalls — with the kernels themselves untouched. How do I read the SRAT/SLIT topology on Linux and place GPU staging buffers in the domain closest to the accelerator? Use numactl --hardware for the node list and SLIT distances, cat /sys/bus/pci/devices/<addr>/numa_node to find the domain owning the GPU’s root complex, and the node’s cpulist to identify local cores. Bind thread and memory together with numactl --cpunodebind=N --membind=N (or mbind() plus pthread_setaffinity_np() per-thread), allocating the buffer after the policy is set. Then verify with an actual transfer microbenchmark — the tables tell you what firmware claims, the measurement tells you what the silicon delivers. What performance loss should I expect if a GPU feeder thread runs in the wrong NUMA or sub-NUMA domain? Enough to matter: in configurations we have profiled, misplaced staging buffers cap sustained host-to-device throughput well below a Gen4/Gen5 link’s nominal ceiling and introduce GPU input-starvation stalls (observed pattern; not a published benchmark). A cross-socket hop is the worst case; a same-socket, wrong-cluster placement is a smaller but still measurable penalty. Absolute figures depend on the CPU, fabric width, and PCIe generation, so measure on your target rather than assuming a fixed percentage. How does this host-side locality concern relate to writing performance-portable GPU code across vendors? It is the host-side twin of the same lesson: performance does not travel with a portable API, and it does not travel with a naive NUMA assumption either. A correct, portable kernel can run slowly on every target if the host feed path is misplaced, because the API abstracts compute, not the memory topology underneath. The durable answer is a procedure for reading the topology and validating placement on each machine, not a hardcoded affinity mask. The question that decides whether any of this pays off is not “which node do I bind to” — it is whether your deployment reads the SRAT/SLIT topology on each target and validates the placement against a real transfer, or whether it ships a fixed mask that was correct on exactly one machine. Host-side NUMA characterization is one dimension of a GPU performance audit, and it is the cheapest bandwidth you will ever recover, because it costs nothing but the discipline to measure before you assume.
