GPU cluster sizing for training comes down to three numbers for each candidate weight format: the bytes each GPU must hold, the bytes each step must move between GPUs, and the bytes each checkpoint must write and read back. The provisioning decision should rest on what those numbers measure on your own H100, NVLink and InfiniBand topology, because published FLOP and link figures are peaks that distributed training does not reach. This matters most when you are choosing between an FP16 worst case and an early commitment to ternary weights, because the saving on paper and the saving during pretraining are different numbers.
GPU cluster sizing for training starts with a memory floor and a throughput budget
Two constraints set the number of GPUs you need. Memory sets the minimum: after sharding, the model states and activations have to fit in each GPU’s HBM. Throughput then sets how many GPUs beyond that minimum you need to finish in your time budget.
The Megatron-Turing NLG 530B paper shows both constraints at work. Its authors note that the model needs over 10 terabytes of aggregate memory for weights, gradients and optimizer states alone. Each model replica spans 280 A100 GPUs, with 8-way tensor slicing inside a node and 35-way pipeline parallelism across nodes (8 × 35 = 280), and data parallelism then scales out to thousands of GPUs.
On the speed side, theoretical specs mislead. On 280, 350 and 420 DGX A100 servers, the same paper measured 126, 121 and 113 teraFLOP/s per GPU, against an A100 16-bit peak of 312 teraFLOP/s. Narayanan et al. (2021) trained a 1-trillion-parameter model at 502 petaFLOP/s on 3,072 GPUs, 52% of theoretical peak per GPU. Meta reports 38-43% BF16 model FLOPs utilisation for Llama 3 on H100. Sizing node count from peak FLOPs overstates what the cluster will deliver. Size it from achieved throughput at the node count you plan to run.
How do you estimate per-GPU training memory, and why measure it anyway?
For mixed-precision training with Adam, start from model states. The 2024 survey of distributed LLM training (arXiv 2407.20018) counts 16 bytes per parameter: 4 bytes for 16-bit parameters and gradients, plus 12 bytes for the 32-bit parameter copy, momentum and variance. DeepSpeed’s documentation uses 18 bytes, because it counts 16 bytes of fp32 state and adds 2 bytes for the fp16 parameters. The ZeRO-Infinity paper uses 20 bytes. Each describes a different accounting, so a sizing document should say which one it uses.
Worked example: model states under ZeRO-3
Assume a 70B-parameter model, DeepSpeed’s 18-byte accounting and ZeRO-3 with no offload. DeepSpeed gives the per-GPU need as the largest layer’s memory plus 18 × parameters ÷ total GPUs. Excluding the largest layer, that is 70 × 10⁹ × 18 = 1.26 TB of model states, about 19.7 GB per GPU across 64 GPUs or 39.4 GB across 32. These figures cover model states only, before activations and temporary memory.
Those remaining terms make the formula a floor. DeepSpeed warns that for long sequences, activations and temporary memory can take as much memory as parameters, gradients and optimizer states combined. Gradient accumulation changes the activation picture. In the Megatron-Turing case, splitting a batch of 1,920 into 1,920 micro-batches cut peak activation memory from 16.9 terabytes to 8.8 gigabytes without changing the effective batch size. The survey lists temporary buffers and memory fragmentation alongside model states and activations as separate consumers.
Some memory is also hard to see. PyTorch’s documentation notes that NCCL allocates GPU memory outside PyTorch’s allocator, so the memory profiler cannot see it, and it points to torch.cuda.device_memory_used() for raw device usage. We would run a few steps per candidate format at the real micro-batch and sequence length, take an allocator snapshot, and compare it with raw device usage. The gap between the two is the headroom the formula never shows you.
How do tensor, pipeline and data parallelism and ZeRO stages change the node count?
Parallelism choices decide how the memory floor maps onto nodes, and each puts a different traffic pattern on the interconnect. Tensor parallelism splits individual layers and reduces memory in proportion to the number of workers. Its communication is hard to overlap with compute, though, so it needs high-bandwidth links and normally stays inside one node. Pipeline parallelism only exchanges tensors at stage boundaries, which makes it suitable for crossing nodes on lower-bandwidth links. It pays for this with a bubble. Efficiency is MB ÷ (MB + PP − 1), so 4 or 8 micro-batches per pipeline stage give 81% or 90% parallel efficiency.
Megatron computes the data-parallel size as world size ÷ (TP × PP × CP). Meta ordered the Llama 3 dimensions as [TP, CP, PP, DP], with the innermost dimension needing the highest bandwidth and lowest latency, so it sits inside a server. Tensor parallelism works best within a single node. Data parallelism then scales out from the replica, as in the Megatron-Turing setup.
ZeRO changes the memory side of that arithmetic, because it partitions the state that data parallelism would otherwise replicate. More sharding costs more traffic. Full sharding has the lowest memory but about 1.5 times the communication of vanilla data parallelism. EleutherAI reports that it finds ZeRO-3 too communication-heavy on its hardware at large scale, so it trains with tensor parallelism inside nodes, pipeline parallelism across nodes and ZeRO-1. EleutherAI also notes that DeepSpeed forbids combining pipeline parallelism with ZeRO-2/3 gradient sharding. Megatron’s distributed optimizer shards optimizer state and master parameters, using a reduce-scatter of gradients and an all-gather of updated parameters in place of an all-reduce.
So a ZeRO stage can fit the memory budget and still lose on a fabric that cannot carry its traffic. That is a sizing outcome you only see by running it.
How much NVLink and InfiniBand bandwidth does multi-node training need?
You need enough bandwidth that the dominant collectives, at the message sizes your parallelism layout actually produces, do not stall compute. You find that level by measuring bus bandwidth, not by reading the link rate off a spec sheet. The nccl-tests documentation explains why raw algorithm bandwidth misleads for collectives. Algorithm bandwidth for collectives varies with the number of ranks. Bus bandwidth applies a correction (2(n−1)/n for AllReduce) so the result reflects the hardware bottleneck, whether that is NVLink, PCI or the network.
AWS lists expected values for p5.48xlarge: 3,200 Gbps of nominal network bandwidth, an expected bus bandwidth of about 400+ GB/s, and a typical algorithm bandwidth of about 230+ GB/s at 2 GB messages.
Older measurements show a gap between theory and practice. On an IBM POWER9 system, two GPUs reached 70 GB/s against a theoretical 75 GB/s, but four GPUs fell to about 30 GB/s because the CPU interconnect became the bottleneck. On AWS P3, the 100 Gbit/s inter-node limit was reachable only with multi-flow connections, and a single flow got 10 Gbit/s. Measured inter-node Ethernet between two P3 instances came out at about 1 GB/s.
On H100 clusters, the NIXT study traced Nemotron-4 pretraining on up to 2,048 H100 GPUs. ReduceScatter and AllGather carried 58.5% and 41.2% of communication volume, and 70.4% of bytes travelled over NIC-only paths. Moving from the 15B to the 340B model at a fixed GPU count pushed more traffic onto the inter-node network, and bandwidth dropped. Mean bus bandwidth in nccl-tests agreed with production training to within 4-8%, but variation was 2-5 times larger in production. Runs with stragglers showed a 3-4 times drop in bandwidth.
Cutting communication volume is one lever. ZeRO++ reduced ZeRO’s communication volume by 4 times and improved throughput by up to 2.16 times at 384 GPUs.
Rules like “keep tensor parallelism inside the node” or “fewer bytes per message means a smaller fabric” are sound in context, and that context comes from profiling your own run, not from a paper’s headline.
How should you size checkpoint storage for a large training run?
Size checkpoint storage as two separate numbers: the write per checkpoint and the read at restart. AWS (2025) gives the base formula as single-replica checkpoint size = P × Wp + P × Os, with typically 2 bytes per parameter for BF16/FP16 weights and 8 bytes for optimizer state, so about 10 bytes per parameter. The total write equals one replica’s checkpoint. The total read equals that checkpoint multiplied by the number of model replicas, because each replica loads the full checkpoint.
AWS works through a 100B-parameter model on 4,000 accelerators across 500 eight-accelerator nodes, with 32 accelerators per replica and therefore 125 replicas. Weights take 200 GB and optimizer state 800 GB, so each checkpoint writes 1 TB, while restarting reads 125 TB in total across replicas. Loading that in two minutes would need about 8.3 Tbps from storage with concurrent loading, against about 66.7 Gbps with hierarchical distribution.
Pin down the accounting before you trust the result. The distributed-training survey cites 980 GB for a 70B-parameter model. That is 14 bytes per parameter, not the AWS 10-byte convention. Write bursts are their own sizing problem: Meta’s Llama 3 storage sustained 2 TB/s with a 7 TB/s peak, and its bursty checkpoint writes saturated the fabric for short periods, with per-GPU state from 1 MB to 4 GB. Failure rate sets the cadence. Llama 3 pretraining saw 466 job interruptions in a 54-day snapshot.
Does training in ternary instead of FP16 change how many nodes and how much interconnect you need?
Not by the bit-width ratio. TechnoLynx’s first step is to fix the target model size, the node-count range and the metrics to size.
LynxBenchAI measures the accelerators themselves and compares them on one scale; its published results inform which GPUs to consider, not how your training job runs.
Where the ternary saving is real
The saving shows in stored weights. In the Spectra suite at 3.9B parameters, model size is 63.83 billion bits for the FP16 model, 20.59 billion for 4-bit and 10.76 billion for ternary. That is about 5.9 times smaller, not the roughly 10 times 16 ÷ 1.58 suggests, and embeddings and the language-model head stay in higher precision. Spectra’s authors calculate that an FP16 model fills a single H100 at 34B parameters, while a ternary model of over 300B parameters can fit with appropriate packing, and that calculation excludes KV cache, activations and compilation overhead. TernaryLM reports 2.4 times less memory and 4.0 times less storage than FP32, measured at inference on a 132M-parameter model on a single T4. These are inference and deployment figures, not cluster pretraining figures.
Why pretraining keeps much of the FP16 footprint
During training, ternary models keep latent master weights in higher precision, ternarise them on the fly in the forward pass, and use a straight-through estimator to compute gradients for those latent weights. The distributed-training survey describes 1-bit weight training in the same way: gradients, optimizer states and a high-precision latent copy are all kept. Model-state memory therefore still carries the high-precision latent weights, gradients and optimizer state. How much activation memory the format saves depends on the recipe.
Ternary also brings communication patterns of its own. Spectra found that synchronising the per-matrix scale across model-parallel devices caused significant communication overhead, so each device computes its own scale over its shard instead. Its largest models reached 3.9B parameters, trained with Megatron-style 2D parallelism and ZeRO stage 2 on V100 GPUs in FP16.
TernGrad, which ternarises gradients rather than weights, gives communication evidence. TernGrad cuts worker-to-server traffic by a theoretical 20.18 times, but its authors found that the speedup depends on the model’s communication-to-computation ratio, the number of GPUs and the bandwidth. TernGrad is more efficient on slower links, reaching 3.04 times for AlexNet on 8 GPUs over 1 Gbps Ethernet. On a fast NVLink and InfiniBand fabric, smaller messages are likely to buy less.
NVIDIA’s NVFP4 results show gains and limits for another low-precision format. NVIDIA reports up to 1.59 times higher throughput than BF16 on GB200 NVL72 and a micro-batch that doubled from 2 to 4, but fully NVFP4 models diverged, and stable training needed the final four transformer layers kept in BF16.
Three caveats belong in any ternary sizing estimate:
- Different training patterns. Ternary training brings its own communication patterns, such as per-shard scale computation.
- Compute as the limit. Smaller checkpoints do not mean proportionally fewer nodes if compute, not memory, is the binding constraint.
- Non-linear scaling. In one study, a larger model at a fixed GPU count pushed more traffic onto the inter-node network.
Provision for the FP16 worst case, or commit to ternary from day one?
Measure first, then decide on what the measurement shows:
| What the measured runs show | Provisioning decision |
|---|---|
| Ternary memory and all-reduce savings are large and stable across the node counts you will run | Provision for ternary from day one, sized from the measured footprint |
| Savings are marginal, uncertain or the ternary run is unstable | Provision against the FP16 footprint and keep the ternary saving as a hypothesis to test |
Two questions sit next to this decision but are separate pieces of work: a stable ternary QAT recipe at this scale (optimizer, learning-rate schedule, gradient scaling, straight-through estimator settings), and a side-by-side pilot of BitNet b1.58 against an existing bf16 pipeline, including checkpoint and tracking compatibility. BitNet b1.58’s authors claim it matches FP16/BF16 at the same model size and token count. That is their claim, and it is exactly the kind of claim the pilot exists to test on your scale.
A measured sizing sequence for a Megatron-LM or DeepSpeed cluster
Run these steps in order, because each one feeds the next:
- Fix the model and the accounting. Set the parameter count, sequence length, global batch size and the bytes-per-parameter convention (16, 18 or 20) for each candidate format.
- Compute the memory floor. Use model states ÷ total GPUs plus the largest layer’s memory, then add an activation estimate.
- Measure the real per-GPU footprint. For each format, read both an allocator snapshot and raw device usage, since NCCL memory is invisible to PyTorch’s profiler.
- Measure the fabric. Run nccl-tests on your nodes at the message sizes your layout produces, then profile collectives in a real training step, since production variation is wider than in isolated tests. The networking and scheduling validation covered in sizing and proving out a multi-GPU cluster before you buy fits here.
- Measure throughput at two or three node counts. Fit the data-parallel degree to the time budget from achieved TFLOP/s, not peak.
- Size checkpoint write, read and cadence. Use the measured per-replica checkpoint size, the replica count and the failure rate you expect at that scale.
- Choose FP16 worst case or ternary-first from the decision table, and record which measurements would reverse the choice.
Who can measure FP16 and ternary footprints on our H100 cluster before we commit?
TechnoLynx measures the memory, all-reduce and checkpoint cost of each candidate weight format on your own H100 nodes and turns those measurements into a node, interconnect and storage recommendation. The work is profiling and distributed-systems engineering on your Megatron-LM or DeepSpeed stack, delivered as a GPU Performance Engineering engagement. When choosing any partner, ask to see measured bus bandwidth and per-GPU memory from your topology, not extrapolations from spec sheets. Where the open question is which accelerator to buy in the first place, LynxBenchAI is the tool for measuring the hardware itself. Its published results compare accelerators on one scale. They do not run your training job.
NCCL Inspector profiling cost under 2% end-to-end in the NIXT measurements. AWS’s NCCL Tests guide also covers validating cluster network performance before training.
Sources trained at up to 3.9B parameters on V100s do not show whether ternary pretraining keeps its storage advantage once latent weights, optimizer state and per-shard scales are counted. Until someone measures that on a real topology, the FP16 footprint is the number to provision against, and the ternary saving is a hypothesis with a test attached.
