Choosing between tensor+pipeline parallelism and FSDP comes down to measurement. Sweep NCCL all-reduce, all-gather, reduce-scatter and all-to-all on your own fabric across message sizes and topology tiers, then split each training step into compute, exposed communication and data loading for every candidate configuration. Megatron-LM’s rule, tensor parallelism inside a server and pipeline parallelism across servers, is a sound default. FSDP training of 7B models, though, was measured going communication-bound beyond 128 H100 GPUs, after which small tensor-parallel degrees matched or beat it.

TechnoLynx runs this as a profiling and benchmarking engagement for AI infrastructure and production SaaS teams. The first step is an NCCL sweep on your own fabric, done before anyone touches a parallelism setting. The order matters because a best practice only makes sense in a context, and profiling supplies that context in a way reading guidance cannot.

What bandwidth does a 512-GPU fabric actually deliver, compared with the spec sheet?

The all-reduce and all-to-all bandwidth of a 512-GPU fabric has to be measured by running the collectives on that exact topology. NVIDIA notes that NCCL’s default tuning can become suboptimal depending on network switch vendor, virtualisation, CPU or PCI configuration. TechnoLynx scopes which collectives, message sizes and level of detail (per node, per switch tier) you need. It then instruments the cluster with an nccl-tests harness and per-step timeline tooling, runs the sweep across the full fabric and reports the gaps between measured and theoretical bandwidth, tagged by topology.

LynxBenchAI publishes how each device runs the public PyTorch stack, which gives a single-GPU reference to set beside the fabric sweep, and the first step is to look up your GPU in its published results.

Start from the spec. A DGX SuperPOD H100 design gives fourth-generation NVLink at 900 GB/s per GPU and NDR InfiniBand at 400 Gbps per direction, in a rail-optimised full fat-tree with eight NDR400 connections per system. Divide 400 Gbps by eight bits per byte and you get 50 GB/s per NDR port, the inter-node figure to compare against. Elice Cloud runs both InfiniBand and RoCEv2 fabrics, and its view is that real training results vary enough between environments that field measurements matter more than spec sheets.

Published H100 all-reduce figures differ widely. SF Compute reports 112.6 GB/s NCCL bus bandwidth for a 1 GiB all-reduce across 16 GPUs. Another write-up reports about 483 GB/s bus bandwidth for intra-node all-reduce on 8 H100s over NVLink. Yobitel’s knowledge base quotes 40-50 GB/s bus bandwidth per GPU for cross-rack AllReduce over NDR. These come from different GPU counts, payloads and links, so none of them is your number.

Reading nccl-tests output: algbw, busbw and the correction factor

nccl-tests reports the average operation time and two bandwidths: algorithm bandwidth (algbw), which is simply size divided by time, and bus bandwidth (busbw). Algbw for a collective changes with the number of ranks. So the tests apply a correction factor per collective, which gives a figure you can compare with the hardware peak regardless of rank count.

Collective	busbw = algbw ×	Typical use in training
AllReduce	2(n−1)/n	Gradient sums in data parallelism; activations in tensor parallelism
ReduceScatter	(n−1)/n	FSDP and ZeRO gradient updates
AllGather	(n−1)/n	FSDP and ZeRO parameter rematerialisation
AlltoAll	(n−1)/n	MoE expert routing, sequence parallelism
Broadcast, Reduce	1

Bus bandwidth is meant to reflect the speed of whichever link is the bottleneck: NVLink, PCI, QPI or the network. Before you compare a measurement with a spec, you have to know which of those links your measurement actually crossed.

Worked example: converting a measured all-reduce to bus bandwidth

SF Compute’s production validation on two eight-GPU H100 nodes reported 60.1 GB/s algorithm bandwidth and 112.6 GB/s bus bandwidth for a 1 GiB all-reduce across 16 GPUs. With n = 16 the all-reduce factor is 2 × 15 / 16 = 1.875, and 60.1 × 1.875 ≈ 112.7 GB/s, which matches the reported figure to rounding. Busbw is the figure you place next to the bottleneck link’s peak. The same run confirmed that all eight InfiniBand adapters per machine carried traffic and that NCCL’s data channels used NET/IB.

Where measured throughput falls short, and at which message sizes

Plot busbw against message size and you should get an S-curve: latency dominates at small sizes, then the curve climbs and levels off at the hardware’s line rate. NVIDIA calls any dip as message size grows (its example is 2 MB to 4 MB) a strong signal of a bad tuning transition. Plateaus when the size doubles, such as 4 MB to 8 MB or 128 MB to 256 MB, point to NCCL choosing the wrong algorithm and protocol at those sizes. The recommended diagnosis is to rerun nccl-tests while sweeping NCCL_PROTO and NCCL_ALGO across their valid values.

Part of the shortfall comes from how NCCL picks algorithms and protocols. LL and LL128 are usually chosen for small messages to cut latency, and Simple for large messages to maximise throughput. On 16 GH200 nodes over Slingshot, analysed on NCCL 2.19.1, LL and LL128 were best below 64 KiB between nodes but fell sharply against Simple at gigabyte sizes. Ring was best for large messages and tree for small ones. NCCL disables LL128 on systems that cannot guarantee atomic 128-byte writes, so the platform itself limits which protocols you can get. Message size also changes what a tier boundary costs. In one test, crossing between servers made a 1 MB all-reduce 3.4 times slower but an 8 GB one only 7% slower.

A quiet failure is NCCL picking the wrong transport. One engineer’s write-up describes NCCL choosing an ordinary Ethernet interface over InfiniBand with no error or warning, so every sync ran about 200 times slower. Yobitel describes a similar silent fallback to PCIe, with full GPU utilisation but collective bandwidth at 1/20th of what was expected, caused by stale topology XML on a mixed NVLink and InfiniBand fabric. NCCL reports which transport it picked only when you ask it to, and nvidia-smi topo -m shows whether two ranks really share NVLink. Tuning also matters a great deal. On Alps and LUMI, telling NCCL to ignore Slurm’s CPU affinity gave up to 1.6 times on alltoall and 6 times on allreduce from two nodes, and raising NCCL_NET_GDR_LEVEL to 3 added a further 2 times and 3 times.

Breaking results out by node and by switch tier

NCCL uses different transports inside a node and between nodes, so intra-node and inter-node figures describe different paths. One experiment shows how large the gap can be. An 8 GB all-reduce took 31.31 ms on eight NVLink GPUs and 33.47 ms on sixteen GPUs across NVLink and InfiniBand. It took 142.51 ms on eight GPUs with NVLink disabled, and 6,412 ms on sixteen GPUs with InfiniBand disabled, 192 times the sixteen-GPU time.

Switch distance needs its own row in the report. On Leonardo, two GPUs in different Dragonfly+ groups lost 17% of goodput (395 Gb/s down to 328 Gb/s) compared with two GPUs under the same switch, while on Alps and LUMI the loss was 1%. NVIDIA found bandwidth degraded somewhat when tree all-reduce crossed L3 switches in an InfiniBand fabric. For per-rank attribution in real jobs, NCCL Inspector (shipped with NCCL 2.28) records message size, execution time, algbw and busbw for every collective. NIXT builds on those records to classify the physical interconnect each collective crossed and to localise a straggler to a specific rank. A tier breakdown is only as good as the topology metadata behind it, which can be stale or incomplete.

All-to-all needs separate treatment. NCCL’s low-level primitives are optimised for rings and trees and handle all-to-all’s N-to-N pattern less efficiently. Its PXN feature (NCCL 2.12) routes data over NVLink to a GPU close to the destination NIC, which allows up to eight messages to be aggregated into one and reduces traffic through the second-tier spine switches. On Alps and Leonardo, NCCL all-to-all held about 75% efficiency up to 1,024 GPUs, but the same study saw instability at large node counts for alltoall in both NCCL and RCCL.

Two limits apply to every number above. Network noise also changes results, cutting performance at scale by up to 50% on some HPC networks. Microbenchmarks also differ from training: on 2,048 H100s, mean nccl-tests bus bandwidth was within 4-8% of the production mean, but production variation was two to five times larger.

How much of each training step goes to compute, NCCL communication and data-loading stalls?

Per-step attribution comes from profiler traces of the training job, not from nccl-tests, which runs no model. You total CUDA and NCCL kernel time from PyTorch execution traces and measure how much communication is exposed rather than overlapped with computation. TechnoLynx instruments the training loop to capture compute, communication and data-loading time for each candidate configuration, then runs the decomposition across a sweep of GPU counts and strategies on the target cluster. From that it recommends the configuration with the best scaling efficiency, as part of GPU Performance Engineering.

Three published techniques do the attribution. The first is to query Kineto profiles from the PyTorch profiler with PerfettoSQL, which separates NCCL kernels from CUDA compute kernels. The second is to look at the NCCL stream in Nsight Systems and see where communication sits against computation. Our GPU profiler workflow with Nsight Systems and Nsight Compute covers the tooling itself. The third is a throughput subtraction. NERSC’s SC21 tutorial measures data loading alone, compute on synthetic data and the full pipeline. In its first case, IO ran at about 89 samples/s against 377 samples/s synthetic, and the full run, at about 90, was bounded by the slower of the two. Turning off DDP communication on synthetic data isolates communication the same way. At 32 nodes the overhead was about 25%, and it fell to about 12% when the local batch size went from 64 to 128.

Data-loading stalls show up as GPUs sitting idle between iterations. Imbalance between devices shows up when you compare per-phase durations across GPUs. If you need communication records during real runs, NIXT’s authors measured NCCL Inspector profiling at under 2% end-to-end overhead from 16 to 2,048 GPUs. Communication takes a large share of the step in published work. In ByteDance’s MegaScale-MoE work on Hopper GPUs, communication took 43.6% of the forward pass and 32% of total training time.

At what GPU count does scaling efficiency start to fall off for FSDP versus tensor parallelism?

The fall-off point has to be measured separately for each strategy, because each one degrades for a different reason. FSDP relies on AllGather and ReduceScatter, which NCCL supports only with ring algorithms and which quickly become latency-bound as device count grows. In the diminishing-returns study, FSDP training of 7B models became unavoidably communication-bound above 128 H100 GPUs. Beyond that point, tensor parallelism of degree 2 or 4 matched or beat it, and at 2,048 GPUs adding tensor parallelism raised throughput by 52.60%. The PyTorch FSDP paper, on A100s over RoCE, measured a milder 7% per-GPU TFLOPS regression from 8 to 512 GPUs. The two are different hardware and models, which is exactly why one team’s threshold should not become another team’s plan.

Tensor parallelism degrades differently. In a fitted speedup model its gains saturate quickly. Exposed communication climbs once tensor or pipeline groups span more than eight devices and land on InfiniBand. Moving the same workload from A100 to H100, with the best configuration on each platform, cut MFU from 59.67% to 40.77%, because compute improved far more than NVLink and HBM bandwidth. Two caveats apply to the decomposition. Hold caching and prefetch settings constant between runs, or the data-loading share moves for reasons unrelated to the strategy. And the breakdown is only valid for the model, interconnect and data pipeline it was measured on.

Does 8-way tensor+pipeline or FSDP give better throughput for our model size?

Whichever one sustains higher measured throughput at your target scale before communication dominates. There is no reliable generic answer, because Megatron-LM’s authors found that sub-optimal combinations of tensor and pipeline parallelism could cut throughput by up to 2 times even on high-bandwidth links. TechnoLynx confirms your model size, cluster shape and the failure scenarios you care about. It then implements representative tensor+pipeline and FSDP configurations on the target hardware, benchmarks both with fault injection for node dropout, and recommends the configuration to lock in.

LynxBenchAI shows how each candidate device runs the public PyTorch stack on its own, a broadly indicative baseline before either parallelism strategy adds communication, and the first step is to check your GPU in its published results.

The tier rule is where to start. Megatron-LM saw peak performance when tensor-parallel size equalled the GPUs per node (8 on DGX A100), because tensor parallelism’s all-reduce is expensive. Pipeline parallelism’s point-to-point traffic can cross nodes without bottlenecking the computation. When the GPU count doubled at a fixed batch size, its PTD-P combination beat ZeRO-3 by 70%, because it needed less cross-node communication. More recent work shows the older advice to minimise model parallelism does not hold when training with FSDP or ZeRO-3 alone. Small tensor or pipeline degrees shrink the FSDP collective groups and raise throughput. PyTorch also supports a 2D mesh that keeps tensor-parallel traffic inside the node and runs FSDP between nodes.

Serving shows the same boundary. In one H100 test, tensor parallelism across two nodes ran 2 times slower than two-stage pipeline parallelism, and single-node TP=8 beat every two-node configuration. On Perlmutter, vLLM tensor parallelism stopped improving latency for Llama 3.1 70B beyond 16 GPUs.

What your measurements show	Lean towards
Model fits under FSDP and exposed AllGather/ReduceScatter stays small at target GPU count	FSDP, possibly hybrid sharding to keep traffic local
FSDP exposed communication grows with node count (seen above 128 H100s for 7B models)	Add tensor parallelism of 2-4 inside the node, FSDP or data parallel across
Model does not fit within one node’s memory	Tensor parallelism up to GPUs per node, pipeline parallelism across nodes
Tensor-parallel degree would exceed one NVLink domain	Avoid; move the extra degree to pipeline parallelism
Throughput similar but one strategy’s overhead curve is flatter with scale	The flatter curve, if it also tolerates node loss better

Which configuration degrades more gracefully if a node drops out mid-run?

No source we have compares tensor+pipeline and FSDP under node loss on the same cluster, so what we can offer here is a method rather than a verdict. Hybrid-parallel training has functional redundancy across data-parallel pipelines, because GPUs serving the same stage hold identical parameters, and ReCycle uses that redundancy along with pipeline bubbles to keep training through failures. In 3D parallelism with sharded optimiser state, a permanent node failure leaves the parameters recoverable from healthy ranks, but the optimiser state shards are lost. In DeadPool’s measurements of recovery, rebuilding the NCCL communicator dominated recovery latency. The DeadPool authors also cite a 32K-GPU pre-training deployment in which HBM faults, PCIe failures and NCCL watchdog timeouts together caused 49.9% of interruptions.

TechnoLynx’s method is to inject node failures into short runs of each candidate. Checkpoint frequency and the software stack can confound the comparison between strategies, so hold them fixed. Benchmark runs are also shorter than real training and may not capture slow effects such as memory fragmentation.

At what GPU count does NCCL all-reduce start to bottleneck on our InfiniBand fabric?

No single GPU count marks where InfiniBand saturates. The point depends on fabric topology, rail configuration, job placement, NCCL’s ring-versus-tree choice and message size, so the reliable evidence is a pilot at or near your intended GPU count on your interconnect. TechnoLynx runs that pilot. TechnoLynx characterises your workload’s communication pattern and sweeps nccl-tests across increasing GPU counts on the actual cluster, testing ring and tree separately. TechnoLynx then tunes NCCL variables and placement, and reports the count at which achieved bandwidth and scaling efficiency fall away from theoretical.

LynxBenchAI supplies the single-device reference for that pilot, showing how each device runs the public PyTorch stack before the fabric comes into it, and the first step is to read its published results for your GPU.

Algorithm choice is a big part of the answer. Ring all-reduce latency grows linearly with GPU count. NCCL 2.4’s double binary trees keep full bandwidth with logarithmic latency, improving latency by up to 180 times at 24,576 GPUs on Summit. NCCL switches back to rings where they deliver more bandwidth. On Perlmutter, NCCL moved from ring to tree beyond 16 GPUs for 256 KB messages.

Published scaling curves disagree across fabrics. In NIXT’s Nemotron-4 runs, bus bandwidth for the dominant AllGather and ReduceScatter collectives barely changed between 16 and 2,048 GPUs. Elice’s RoCEv2 fabric held AllReduce flat to 16 nodes, then dropped 7-13% at 24 and 30 nodes. A published ring-versus-tree all-reduce figure for a DGX SuperPOD at 200-500 GPUs would not transfer to yours. Anyone quoting one is quoting someone else’s fabric.

The same applies to expected interconnect utilisation during an all-reduce-heavy risk run such as VaR. The commonly cited 20-40% of time spent in collectives is a figure for distributed training, not for your simulation. A pilot also has to use the job’s real message sizes and compute overlap rather than a synthetic sweep alone. NCCL implementation details can change between releases, so re-validate after driver, firmware or NCCL upgrades.

Is our networking layer being compared fairly against NCCL, or hiding overhead?

A custom networking or MPI layer is fairly compared with NCCL only when both run the same workload on the same topology at matched GPU counts. Latency also has to be split into compute, communication and scheduling, and the NCCL baseline has to be tuned rather than left on defaults. TechnoLynx instruments the existing pipeline into those phases, builds a head-to-head harness that runs the same workload over NCCL collectives and the custom layer, and produces scaling-efficiency curves. The report includes the configuration details needed to reproduce and defend the result.

TechnoLynx delivers this head-to-head comparison through GPU Performance Engineering, and the first step is instrumenting your existing pipeline into compute, communication and scheduling phases.

An untuned baseline skews such comparisons. De Sensi and colleagues found that default NCCL and GPU-aware MPI settings are not always optimal, and that manual tuning can improve performance by up to an order of magnitude. Neither library wins everywhere. In that study NCCL was generally faster on collectives and MPI on point-to-point transfers, depending on the system. A separate Perlmutter study found NCCL 1.5-2 times slower than MPI for cross-node all-reduce of 512 KB to 1 MB. The harness also needs real computation between collectives. A microbenchmark that launched all-reduces back to back reported slowdowns that disappeared once representative compute was interleaved.

One way to report multi-GPU scaling efficiency is measured throughput against an ideal of single-GPU throughput multiplied by GPU count. NERSC’s tutorial, for example, reports about 78% efficiency at 32 nodes. For the single-device reference before the fabric comes into it, LynxBenchAI’s published results show how each device runs the public PyTorch stack, which makes them broadly indicative for custom PyTorch models. LynxBenchAI does not run or measure your own workload or networking layer.

Who can benchmark our fabric and settle the parallelism choice before we lock it in?

TechnoLynx measures your NCCL collectives on your own fabric, breaks down training steps for each candidate configuration and recommends tensor+pipeline, FSDP or a 2D mix, with the measurements behind the recommendation. The risk of doing it without experience in reading S-curves, Kineto traces and NCCL logs is acting on a microbenchmark that misrepresents training. When choosing a partner, ask to see raw sweeps tagged by tier, the tuning variables tried, and how node loss will be tested. The engagement sits under GPU Performance Engineering.

The engagement is not the right fit for everyone. A team whose only question is which accelerator to buy is better served by published device results than by a fabric study. A team with an established performance group may only need its existing sweeps and traces reviewed. And a cluster vendor’s acceptance testing will confirm the fabric is healthy, but it does not tell you which parallelism strategy your model should run.

From fabric numbers to a cluster plan

Tier-tagged busbw curves and per-step breakdowns are the inputs for deciding node count, placement and scheduling. Our guide to sizing and validating multi-GPU clusters against real workloads picks up there. Two uncertainties remain even after a careful study. Production collectives vary more than isolated tests do, and we know of no published head-to-head of how tensor+pipeline and FSDP survive node loss. Both are reasons to keep NCCL telemetry running after the configuration is locked.

FAQ

What are the differences between all-reduce, all-gather and reduce-scatter?

All-reduce combines data across ranks, for example by summing, and gives every rank the result. It can be built from a reduce-scatter, which leaves each rank one reduced shard, followed by an all-gather, which reassembles the shards everywhere. FSDP uses all-gather to rebuild parameters and reduce-scatter for gradients.

Does NCCL use RDMA?

Yes, where the hardware supports it. NCCL’s InfiniBand Verbs transport can use GPUDirect RDMA so the NIC reads and writes GPU memory directly, and it falls back to a socket transport staged through host memory when RDMA is unavailable.

Is NVLink scale-up or scale-out?

NVLink is the scale-up path between GPUs inside a node or NVLink domain, and NCCL prefers it for intra-node traffic. InfiniBand or RoCE handles scale-out between nodes.