A multi-GPU serving deployment stalls at high concurrency. The model is quantized, the kernels are fused, batch size is tuned — and tokens-per-second still plateaus below what the GPUs should deliver. The reflex is to blame the runtime. Often the real ceiling is the wire between the cards. Direct Attach Copper (DAC) cables are the short-reach interconnect that links GPUs, NICs, and top-of-rack switches inside a serving rack. They are cheap, low-latency, and passive over short distances — and they quietly shape the throughput ceiling of an inference deployment. Read as a facilities detail, they are invisible until concurrency exposes them. Read correctly, interconnect choice is a determinant of achievable batch size and sustained tokens-per-second, and therefore of cost-per-request. That last clause is the whole point of this article. DAC only matters here insofar as it moves the cost-per-request number — this is not a networking-hardware guide. What matters most about a DAC cable in practice? A DAC cable is a fixed-length assembly with the transceiver-equivalent electronics bonded directly to each end of a copper twinax cable. Instead of converting an electrical signal to light (as an optical transceiver does), it carries the signal as electricity end to end. Passive DACs do no signal conditioning; active DACs add a small amount of equalization to reach slightly further. Because there is no electrical-to-optical conversion, a DAC adds very little latency and draws very little power at the port. The practical consequence for a serving cluster is a reach limit. Passive copper at the higher line rates used for GPU fabrics is generally usable only over a few metres — which is why DAC is the default inside a rack and rarely between racks. When your GPUs, NICs, and top-of-rack switch all live in the same rack, DAC is typically the lowest-cost, lowest-latency way to wire them. Stretch the topology across rows and copper stops being an option; that is where Active Optical Cables (AOC) and pluggable optical transceivers take over. The reason this belongs in a cost conversation rather than a cabling manual is simple: the interconnect sets an upper bound on how fast tensors, KV-cache pages, and activations move between devices. When multi-GPU inference relies on collective operations — think NCCL all-reduce or all-gather across tensor-parallel shards — that traffic runs over exactly these links. The wire is in the critical path of every token. What is the difference between DAC, AOC, and optical transceivers in a GPU serving rack? The three options are not competitors so much as answers to different reach questions. Choosing among them is a distance-and-budget decision, and getting it wrong shows up as either wasted capital or a fabric that cannot span your topology. Interconnect Typical reach Relative port cost Port power Latency added Best fit in a serving cluster Passive DAC ~1–3 m Lowest Negligible Lowest Intra-rack GPU↔NIC↔ToR switch links Active DAC ~3–7 m Low Low Very low Slightly longer intra-rack or adjacent-rack runs Active Optical Cable (AOC) ~tens of metres Medium Medium Low Rack-to-rack within a row Pluggable optical transceiver Long (row-to-row, hall-scale) Highest Highest Low Spine/leaf uplinks, cross-hall fabric (Reach and cost bands are directional and depend on line rate, cable gauge, and vendor implementation — treat them as planning heuristics, not a spec sheet. Confirm against your switch and NIC vendor’s supported-optics matrix before ordering.) The engineering takeaway is that DAC wins on cost and power only within its reach envelope. The failure mode is buying DAC for a topology that will grow beyond a rack, then re-cabling under production pressure — or, conversely, provisioning optical everywhere and paying an interconnect tax you never needed. Our sibling article on when direct-attach copper beats optics on cost-per-request works the trade-off numerically; this piece stays with the mechanism and the diagnostic. When does interconnect become the bottleneck for multi-GPU inference throughput? Interconnect becomes the bottleneck when the volume of cross-device traffic per forward pass grows faster than the fabric can move it. Three parallelism patterns drive that traffic, and each stresses the wire differently. Tensor parallelism splits individual layers across GPUs, which means an all-reduce on every layer’s output — a chatty, latency-sensitive pattern that punishes any interconnect weakness at high concurrency. Pipeline parallelism passes activations between stages, a lighter but bursty load. Disaggregated serving, where prefill and decode run on separate device pools, moves KV-cache between them over the fabric; the prefill/decode split that cuts cost-per-request is a real win precisely because it isolates two very different traffic profiles, but it only pays off when the cache-transfer link can keep up. The tell is a specific shape in your metrics. As you raise concurrency, tokens-per-second should climb until the GPUs saturate. If instead throughput plateaus while GPU compute occupancy sits well below its ceiling — the cards are waiting, not working — the stall is upstream of the kernels. When that plateau coincides with rising p95 latency and NCCL collective time growing as a share of step time, the fabric is a leading suspect. This is an observed diagnostic pattern from serving-path profiling, not a single benchmarked threshold; the numbers depend on model, parallelism degree, and batch shape. How do DAC reach and bandwidth limits affect achievable batch size and tokens-per-second? Batch size is the lever operators reach for to amortize fixed per-step costs across more requests. Larger batches raise throughput and lower cost-per-request — up to the point where something saturates. When the interconnect saturates first, raising batch size stops helping and can hurt: the collective traffic per step grows, the fabric queues, and per-request latency climbs while tokens-per-second flatlines. DAC’s reach limit interacts with this indirectly but decisively. If reach forces a topology change — spreading tensor-parallel shards across racks over optical links with higher latency, or landing on a lower-bandwidth path — the effective bandwidth-per-collective drops, and the batch size at which the fabric saturates drops with it. You end up serving fewer concurrent requests per node before latency targets break, which is the same as saying your cost-per-request floor moved up. A worked illustration (assumptions stated) Suppose a two-GPU tensor-parallel node targets a p95 latency budget and you sweep batch size: Assumptions: fixed model, fixed quantization, NCCL all-reduce per layer, DAC intra-rack fabric, concurrency raised by increasing batch size. Observed shape: tokens-per-second rises with batch size up to batch B, then plateaus while GPU compute occupancy stays below its ceiling and NCCL collective time rises as a share of step time. Interpretation: batch B is a fabric-limited plateau, not a compute-limited one. Cost-per-request is minimized at B, not beyond it, and the floor is set by the interconnect — not the model. This is illustrative framing, not a measured result — the value of B is entirely deployment-specific. The point is the diagnostic logic: a plateau with idle compute and rising collective time reads as fabric, and the fix lives in the wire or the topology, not the model. How does fabric saturation show up in cost-per-request for a production AI feature? Cost-per-request is, roughly, the cost of a node-hour divided by the requests that node serves in an hour at your latency target. Fabric saturation attacks the denominator. Degraded p95 latency forces you to lower concurrency to stay inside the budget, and lower sustained tokens-per-second means fewer requests per node-hour — both inflate cost-per-request at a given concurrency without changing a single line of model code. That is the silent tax the naive approach never sees. A cost programme that only ever asks “is the model too big?” or “is the kernel slow?” will optimize the compute while the wire caps the unit economics. The three quantities worth watching are effective GPU utilization at your target batch size, tokens-per-second per node before and after any fabric change, and the resulting cost-per-request delta. If a topology or cabling change moves tokens-per-second per node and effective utilization together, the fabric was the tax. If neither moves, it was not — and you have ruled out an expensive rabbit hole. For the broader arithmetic of turning throughput into a per-request figure, our LLM token calculator walkthrough lays out the cost model this diagnostic plugs into. We treat this the way we treat any bottleneck claim: measure it on the deployed serving path before acting. The [inference cost-cut pack](Inference Cost-Cut Pack) profiles that path end to end, and cabling is one of the layers it isolates when the bottleneck turns out to be fabric rather than model or kernel. For teams running SaaS inference at scale, this sits inside the wider AI infrastructure engagement where fabric, runtime, and model choices are traded against one unit-economics number. When should a team investigate cabling versus model or kernel optimisation to cut inference cost? The honest answer is: investigate cabling after profiling, not before, and only when the profile points there. Model and kernel work — quantization, compiler-driven kernel fusion, batching strategy — is usually the higher-yield first pass and is where most cost-per-request wins live. Fabric investigation earns its place when the profile shows idle compute at a throughput plateau. Use this rubric to decide where to spend the next optimization hour: Is GPU compute occupancy near its ceiling at target batch size? If yes, the bottleneck is compute — pursue model/kernel work, not cabling. Does throughput plateau while compute occupancy stays low? If yes, the stall is upstream of the kernels — a fabric candidate. Does NCCL (or equivalent) collective time grow as a share of step time as concurrency rises? If yes, cross-device traffic is a leading suspect. Does the topology span beyond a single rack, or is it about to? If yes, reach limits and the DAC-vs-optical decision are now part of the cost model. Have you measured tokens-per-second per node before and after a fabric change? If not, you cannot attribute a cost-per-request delta to the wire — measure first. Steps 1–3 tell you whether the fabric is the problem. Steps 4–5 tell you what to do and how to prove it worked. Profiling the serving path is the discipline that separates a real fabric bottleneck from a guess — the same GPU serving-path profiling practice that governs which layer of the stack deserves your next hour. FAQ How does a DAC cable work? A DAC cable carries the signal as electricity end to end over copper twinax, with transceiver-equivalent electronics bonded to each end — no electrical-to-optical conversion. That makes it low-latency and low-power but reach-limited to roughly a few metres at high line rates. In practice it is the default interconnect inside a serving rack, linking GPUs, NICs, and the top-of-rack switch. What is the difference between DAC, AOC, and optical transceivers in a GPU serving rack? They answer different reach questions. Passive and active DAC are lowest-cost and lowest-power but reach only within or just beyond a rack; Active Optical Cables span rack-to-rack within a row; pluggable optical transceivers reach row-to-row and hall-scale. DAC wins on cost and power only inside its reach envelope — beyond that, optical is the only option. When does interconnect become the bottleneck for multi-GPU inference throughput? When cross-device traffic per forward pass grows faster than the fabric can move it — driven by tensor parallelism (per-layer all-reduce), pipeline parallelism, or disaggregated prefill/decode (KV-cache transfer). The tell is a throughput plateau while GPU compute occupancy stays well below its ceiling and collective time rises as a share of step time. How do DAC reach and bandwidth limits affect achievable batch size and tokens-per-second? When the interconnect saturates before compute, raising batch size stops helping: collective traffic queues, latency climbs, and tokens-per-second flatlines. If DAC’s reach forces a topology change onto higher-latency or lower-bandwidth paths, the batch size at which the fabric saturates drops — so you serve fewer concurrent requests per node and your cost-per-request floor rises. How does fabric saturation show up in cost-per-request for a production AI feature? It attacks the denominator of cost-per-request. Degraded p95 latency forces lower concurrency to stay inside the budget, and lower sustained tokens-per-second means fewer requests per node-hour — both inflate cost-per-request without any change to model code. Watch effective GPU utilization at target batch size, tokens-per-second per node before and after fabric changes, and the resulting cost-per-request delta. When should a team investigate cabling versus model or kernel optimisation to cut inference cost? Investigate cabling after profiling, not before. Model and kernel work is usually the higher-yield first pass; fabric investigation earns its place only when the profile shows idle compute at a throughput plateau with rising collective time. Measure tokens-per-second per node before and after any fabric change so you can attribute a cost-per-request delta to the wire rather than guess. The uncomfortable version of the lesson: you can ship a perfectly optimized model onto a fabric that quietly caps its economics, and no amount of further model work will move the number. The wire is a first-class variable in cost-per-request, and the only way to know whether it is your bottleneck is to profile the deployed serving path before you touch anything.
