To know whether a multi-GPU cluster will meet your nightly deadline, your p95 latency target and your scaling plan, run your own workload, or a faithful proxy of it, on the candidate configuration at the scale you plan to buy. Vendor specifications and generic benchmarks describe someone else’s operating point. The Introduction to Machine Learning Systems textbook (mlsysbook.ai) puts it directly: a benchmark earns its value only when its operating point matches the deployment’s, not the datasheet’s.

A GPU cluster is a set of GPU servers connected at two levels: fast links between the GPUs inside each node (NVLink, NVSwitch or PCIe), and a scale-out fabric between nodes (InfiniBand or RoCE Ethernet), with a scheduler such as Slurm or Kubernetes handing that capacity out to jobs. Three things weigh on whether it meets a target: where the scheduler places each job on the fabric, how much of each step goes to collective communication, and how that overhead grows as you add GPUs. Measure your workload at target scale before capital is committed, because every other number is a proxy whose error you have not measured.

For teams building AI infrastructure and production SaaS, getting this wrong means a fleet bought around the wrong bottleneck, so the method below runs before the purchase order is signed. Sizing a cluster on paper and proving it on hardware are two different multi-GPU questions, planning and validation. What follows concentrates on the validation evidence you need before any money moves.

Generic benchmarks measure someone else’s operating point

The same textbook says a headline throughput or latency figure becomes useful only once you can map it to your model, batch shape, data movement, sustained operating point and power envelope. MLPerf HPC’s benchmarks, CosmoFlow and DeepCAM, are both scientific machine learning training applications.

Do not start from a rule-of-thumb minimum size for a dedicated surrogate-model cluster. Size it from the programme’s own job profile, and cover intermittent peaks with cloud burst capacity rather than extra permanent nodes.

Because MLPerf HPC tests training only, evidence that one partition can run MPI wave propagation alongside PyTorch surrogate training has to come from a concurrent proxy of both, run on each candidate fabric at the target partition size.

How do we load-test a cluster at small scale before buying the whole fleet?

Get a slice of the candidate hardware, ideally a few nodes spread over more than one rack, and run a fixed sequence of tests on it, starting with component checks and ending with your real distributed job. Each step isolates one layer. The order matters, because a failure at one step makes the later steps meaningless.

1. Component baselines. NVIDIA’s nvbandwidth measures host-to-device, device-to-host and GPU-to-GPU bandwidth across NVLink and PCIe. Keep its numbers as the baseline that later regression checks compare against.
2. Collective tests at three scales. NVIDIA’s Run:ai tutorial runs nccl-tests all_reduce on one node to validate NVLink, on two nodes in one rack for the intra-rack fabric, and on three nodes in different racks for the cross-rack fabric. A short run also confirms the driver, NCCL and MPI stack, GPU visibility, and that pods land where the fabric expects them.
3. Every node, not a sample. The ClusterBench authors (arXiv, 2026) argue that validation must cover every node, because the slowest process sets the runtime of a multi-node job. They measured up to 5% variation across components that are identical by specification.
4. A real distributed workload. NVIDIA notes that a cluster can pass standard diagnostics and still fail under a real distributed job, and that some failures only appear at a particular scale.
5. A scaling sweep of your own job. HPC Wiki recommends job sizes in powers of two, several runs per size, and a configuration that matches production, with no simplified models.
6. Sustained duration. MLPerf Inference requires every run to last at least 60 seconds so that power management and frequency scaling reach equilibrium. ClusterBench runs each benchmark for at least 10 minutes.

A slice tests per-node and per-rack behaviour. It does not prove behaviour at hundreds of nodes, because in strong scaling the communication overhead grows with the number of processes. Treat the small-scale result as an input you still have to test at target size, on rented or loaned capacity. Where that capacity is out of reach, trace replay is a partial substitute: CCL-Bench (arXiv, 2026) replays execution traces through a network simulator to estimate how much step time would improve if a resource such as scale-out bandwidth were doubled.

Can a pilot batch of real genomes on cloud GPUs prove we beat our on-prem DRAGEN SLA?

Yes, if the pilot uses real or representative genomes on the actual pipeline. Push a batch of real or representative genomes through the actual variant-calling pipeline on the candidate cloud GPU instances, then compare end-to-end turnaround and sustained throughput with the on-prem cluster’s current SLA.

LynxBenchAI publishes results showing how each candidate GPU runs the public PyTorch stack, so the first step is to check those results when shortlisting cloud GPU instances for the pipeline’s PyTorch-based stages.

TechnoLynx runs that pilot as an engineering engagement. It records the on-prem pipeline’s throughput, turnaround and peak volume as the baseline, picks representative genome batches and candidate instance types, ports and configures the pipeline including data staging, then measures the results against the SLA, adjusting the instance choice or the pipeline if the first run falls short. For shortlisting GPUs, LynxBenchAI publishes results showing how each device runs the public PyTorch stack. Those results are broadly indicative for PyTorch-based stages, but they do not run or measure your variant-calling pipeline.

On AWS HealthOmics, the candidates are the GPU instance types it supports. In March 2025 it added NVIDIA L4 and L40S GPUs to its existing A10G and T4 support, and its documentation lists accelerated instances such as omics.g6e with 48 GiB of memory per GPU. When a task accelerator covers several instance types, HealthOmics chooses by availability and prefers the cheaper one. That makes it worth pinning the instance type if you want pilot runs you can compare.

AWS has published a precedent. AWS’s benchmark of PacBio’s whole-genome variant pipeline on HealthOmics found that the best configuration, an A10G instance with dynamic storage, finished in 8.67 hours for $21.26 of compute, and that GPU-based DeepVariant cut cost by 21.3% and run time by 8.5% compared with CPU. The same write-up warns that memory requirements change with sequencing depth and with the genetic diversity of the population. For that reason, run the pilot on your own genomes, not someone else’s.

Include data transfer and staging in the turnaround, not compute alone. MLPerf HPC added data staging to its measured time to capture how moving massive scientific datasets affects results. Keep the pilot and the SLA separate in your thinking. The pilot shows what the tested genomes achieve, while guaranteed throughput at peak sequencing volume is a contractual commitment you negotiate with the provider, not something to infer from a pilot. If the pilot meets or beats the on-prem SLA across representative volumes and sample types, move on to negotiating peak-capacity terms. If it falls short or varies from run to run, hold the commitment until instance choice or pipeline tuning closes the gap.

How do we prove a configuration meets its p95 latency target under peak load?

Use a load generator that sends requests at peak rate on a realistic arrival pattern, runs long enough to reach equilibrium, and collects enough queries for the percentile you are claiming. An average latency proves nothing about p95.

MLPerf Inference’s LoadGen sends server-scenario queries on a Poisson distribution to mimic real arrival rates. Its server metric is the query rate a system sustains while meeting the latency bound, which the authors call latency-bounded throughput, rather than raw throughput. The number of queries comes from statistics: it depends on the tail-latency percentile, the required margin and the required confidence, and a 99th-percentile guarantee needs 262,742 queries, rounded up to 270K. A looser p95 target needs fewer queries, but the rule is the same: work out the count before the test, not after.

Unconstrained throughput tells you little about a latency target. The MLPerf authors (2019) found that every system they compared delivered less throughput in the server scenario than offline, because the latency constraint forces less efficient batching. They concluded that a comparison without a latency constraint has little bearing on one with it. Fan-out makes the tail worse at cluster scale. Dean and Barroso (2013) show that if each server’s 99th-percentile latency is one second, a request that collects responses from 100 such servers is slow 63% of the time.

Low utilisation is not a safety margin either. The inference-fleet-sim study (arXiv, 2026) found that a fleet at about 30% utilisation could still miss its P99 time-to-first-token SLO, because long-tail requests caused back-pressure that cascaded through the queue. The same study found that its analytical queueing model approved broken fleets that only discrete-event simulation caught.

How do we combine latency, throughput and quality into one score the board can defend?

Build the score from parts you disclose: a throughput or latency figure measured under a fixed SLA, an accuracy metric suited to the task, and stated weights, with the sub-metrics published next to the composite. MLPerf Training handles speed and quality together by measuring time to train to a defined quality target, which turns both into one end-to-end number. Its authors declined to produce a suite-wide summary score, because a summary implies a weighting and no weighting scheme fits every user.

If you combine normalised results, use the geometric mean. Fleming and Wallace (1986) showed that the arithmetic mean of normalised benchmark results leads to mistaken conclusions. CCL-Bench gives an example of a disclosed composite: it scores configurations as w × T0/T + (1 − w) × N0/N, with w = 0.5, to balance step time against the number of accelerators.

For video analytics, the defensible score is a documented composite of throughput or latency under a fixed SLA plus a task accuracy metric, never frames per second standing in for quality. If the board can accept two numbers, throughput at fixed accuracy and accuracy at fixed throughput are easier to defend than any blend. A survey of edge video analytics (arXiv, 2022) lists the task-level accuracy metrics to choose from: top-1 or top-5 accuracy for classification, mAP for object detection and MOTA for multi-object tracking.

How many GPUs, and what interconnect, does a nightly risk run need to finish with margin?

Size it from your own scaling curve, not from vendor reference numbers. Benchmark the real risk kernel at several GPU counts, find the smallest configuration whose measured wall-clock time finishes inside the window with margin to spare, and let the shape of the curve show whether the interconnect is the constraint.

Benchmarking our own Monte Carlo VaR kernel at 8, 16 and 32 GPUs

STAC-A2 makes a good template. It is a vendor-independent market-risk benchmark specified by trading firms, it computes option Greeks with Monte Carlo methods, and it supports two kinds of scaling: workload scaling (portfolio size and path count) and system scaling (several defined system sizes). Its capacity test asks how many assets a system can process within a fixed 10-minute window. That maps directly onto a nightly deadline: swap the 10 minutes for your batch window and the asset count for your book.

Running your own kernel at 8, 16 and 32 GPUs follows the power-of-two job sizes HPC Wiki recommends. The same guidance says to compare wall-clock times rather than FLOP rates and to report only measured results, never extrapolations. Strong-scaling efficiency is the speedup divided by the increase in GPU count. Efficiency across 8, 16 and 32 GPUs indicates the best match between job size and GPU count.

The margin has to cover the spread between nodes, since the slowest process sets the runtime. If efficiency holds at 32 GPUs, the fabric is not the constraint at that size. If it drops, the diagnosis below shows whether tuning or new hardware closes the gap.

Is communication or compute limiting our job at the target GPU count?

Whether the interconnect is the bottleneck depends on how much of each step goes to collective communication rather than compute. That share moves with model size, GPU count, parallelism strategy and message sizes, so the only reliable answer is a measured ratio of communication to compute for your workload at your target GPU count.

TechnoLynx measures that ratio through its GPU Performance Engineering work. It establishes your model sizes, current and planned GPU counts and parallelism plan, runs NCCL and end-to-end scaling tests on target or representative hardware, and delivers strong- and weak-scaling curves that separate compute, communication and halo-exchange time. The result is a recommendation to re-tune, upgrade or change scheduling that you can take to a board. Before any hardware is on loan, LynxBenchAI published results show how each candidate device runs the public PyTorch stack. They are broadly indicative for custom PyTorch models, but they do not measure your workload or your fabric.

Published measurements show how large the communication share can get. CCL-Bench recorded GPU runs spending between 35.2% and 93.1% of traced time in communication across five workloads. The MLPerf HPC analysis found 10 to 40% of application time going to collectives as runs scaled across GPUs. Whether an upgrade pays off depends on the workload. In CCL-Bench’s simulations, doubling scale-out bandwidth sped up communication-heavy multi-node training by up to 28.7% but did not improve single-node inference, while 128- and 256-GPU training workloads gained only 3.9% and 3.4% because they were compute-bound.

For a fine-tuning job that fits on one multi-GPU machine, the relevant test is intra-node communication and gradient-synchronisation efficiency across GPUs linked by NVLink or PCIe. If communication remains a small share of step time there, PCIe is not what limits the job today. That changes if the roadmap moves to a larger model or more nodes, so test the planned scale as well.

At what node count does a Monte Carlo VaR run stop scaling, and how do we find it first?

Scaling stops when the extra inter-node traffic from another node outweighs the compute it saves. You find that point with a strong-scaling sweep, keeping the book and path count fixed while increasing the node count. Amdahl’s law assumes an infinitely fast network, but real fabrics have latency and finite bandwidth, so actual speedup falls further behind ideal speedup as nodes are added. In HPC Wiki’s worked example, speedup peaked at four nodes and then fell, because growing inter-node traffic made each extra node a net cost.

Record communication fraction and compute-communication overlap next to wall-clock time, the same metrics CCL-Bench computes from traces. CCL-Bench’s traces let you attribute measured performance to specific infrastructure components. The MLPerf HPC authors note that model-parallel scalability depends heavily on overlapping halo exchanges with local computation.

Re-tune NCCL first, or buy new switches and NICs?

Re-tune first when the diagnosis points to inefficient configuration. Buy hardware only when measured bus bandwidth is already close to the link’s line rate and communication still dominates the step. NVIDIA says users generally should not need to set NCCL environment variables to reach peak performance, because NCCL detects the topology and tunes itself. It also notes that the defaults can become suboptimal because of the network switch vendor, virtualisation, the CPU or the PCI configuration.

Read nccl-tests output as bus bandwidth (busbw), not algorithm bandwidth (algbw). Bus bandwidth applies a correction for each collective, 2(n−1)/n for AllReduce, so you can compare it with hardware peak whatever the rank count. At 8 ranks that factor is 2 × 7/8 = 1.75. Plot busbw against message size. NVIDIA’s tuning article says a dip as message size grows, or a flat stretch across a doubling, strongly suggests NCCL is picking the wrong algorithm and protocol.

NCCL_NET_GDR_LEVEL controls when GPUDirect RDMA is used between a NIC and a GPU, NCCL_NETDEVS_POLICY sets how network devices are assigned to GPUs, NCCL_BUFFSIZE sets the buffer size for GPU-pair transfers, and the CTA count trades communication throughput against compute resources. NVIDIA recommends tuner plugins as the way to override specific choices without changing application code.

What the measurement shows	Where the next pound goes
busbw well below line rate; S-curve dips or flat stretches	NCCL re-tuning or tuner-plugin overrides
busbw near line rate; communication fraction high and rising with node count	Faster scale-out fabric or a larger scale-up domain
Communication fraction low, yet efficiency falling	Kernel or single-node work; check halo exchange and rank imbalance
Scales well in isolation, poorly under the production scheduler	Topology-aware placement

For payback in cycle time, CCL-Bench’s utility metric has the right form: the fractional improvement in step time from doubling one resource while holding everything else fixed. Multiply that by the number of runs a year and set it against the cost of the upgrade.

Build, buy or optimise: the backtester and the scheduling layer

Two build-versus-buy decisions come up alongside cluster sizing here, and both should be made the same way: from a profiled bottleneck and a head-to-head measurement, never from the assumption that a rewrite or a custom layer must be faster. TechnoLynx weighs both decisions against the team’s runway, profiles the existing stack before recommending any rewrite, and builds a test harness that runs NCCL all-reduce scheduling against a custom MPI topology on the actual Monte Carlo VaR workload at production scale, as part of its GPU Performance Engineering service.

Rewrite the backtester in C++/CUDA, or optimise vectorised GPU Python?

For a two-person team with limited runway, start by optimising the vectorised GPU Python backtester. Fund a C++/CUDA rewrite only if a profiled bottleneck survives that optimisation and the speed target is still out of reach.

Buying a GPU backtesting platform is a third option. QuantConnect reported in its GPU beta announcement that a client strategy, stopped after more than 24 hours on a CPU-only machine, ran in 17 minutes on its GPU machine. Treat that as a vendor claim about one strategy. The same announcement says GPUs incur overhead loading data and that best results come from batch analysis at weekly or monthly intervals. Buy when a platform covers your data sources, order types and performance needs at a licence cost lower than the build time. Otherwise, optimise what you have.

Who can benchmark our scheduling layer and tell us whether to build it in-house?

TechnoLynx, through its GPU Performance Engineering service, runs an independent review of your NCCL/MPI scheduling and interconnect design against the overnight requirement, and settles the topology question with a production-scale benchmark rather than an opinion. On in-house depth, the review checks whether your team can tune collectives at the target scale and says plainly where it cannot. A specialist mainly adds method: bus-bandwidth S-curves, communication fractions per rank, and a like-for-like comparison of NCCL against a custom MPI topology on your Monte Carlo VaR workload. If NCCL tuning meets the overnight window, keep the work in-house. If gaps remain, the engagement implements and validates whichever topology won the benchmark.

This engagement does not suit a team that wants a generic cluster health check, which nccl-tests and NVIDIA’s open-source readiness tooling already cover. It also does not suit a team buying engineering hours rather than a decision. Realistic alternatives are the hardware vendor’s solution architects or an in-house spike if someone on the team has done collective tuning at this scale before. Whoever you choose, ask for the harness, the traces and the configuration as well as the recommendation, so you can rerun the result yourself.

The evidence explains why measurement beats assumption here. In a study of Allgatherv on multi-GPU systems, NCCL ran on average 1.2x faster than CUDA-aware MPI across the tensor-factorisation data sets on a conventional cluster. In the same study, on one irregular data set, MVAPICH2-GDR’s communication runtime differed 3.1x between 1MB and 4MB settings of one tuning variable, whose best value was 512MB at 2 GPUs and 16B at 8 GPUs.

Can one GPU server batch four camera stations without dropping frames or accuracy?

It can, provided the batching across stations preserves accuracy and per-item latency stays inside the reject window under sustained peak load. That has to be validated on the target server, not assumed.

LynxBenchAI publishes results showing how each candidate GPU runs the public PyTorch stack, so the first step is to check those results when shortlisting the server, then validate the four-station batching on it.

Research on packing frames from several cameras together shows the approach can hold accuracy. MOSAIC (arXiv, 2023) packed frames from six cameras into shared canvases on a Jetson TX2 and reached 4.75x the throughput of first-come-first-served processing, with at most 1% accuracy loss at batch size 4. That result is for pedestrian detection on an edge device, not a defect classifier on a data-centre GPU, so it demonstrates the mechanism but not your numbers. It also shows the trade-off is real: shrinking all six streams uniformly onto one canvas cost about 8% accuracy.

The test is the distribution of per-tablet latency, not its mean, because intermittent misses under bursty load can hide behind a healthy average. The worry that results slow down when the line is busiest can be measured. On an air-cooled A100 partition, ClusterBench found that higher GPU temperature increased SM frequency throttling and reduced DGEMM performance. ClusterBench records power, clock frequency and temperature over the same interval as each benchmark. Do the same here, at sustained peak, for long enough to reach thermal equilibrium.

If shared batching holds latency and per-class accuracy at sustained peak, one server can handle all four stations. If it does not, re-tune the batching first, and add GPU capacity, up to one GPU per station, only if tuning cannot close the gap. Any change to the model or camera resolution means validating again.

Choosing a GPU cluster topology and scheduler that still holds when the GPU count doubles

A topology survives doubling when three conditions hold: its scaling has been measured at the planned size, the fabric has spare switch radix to add spine capacity without replacing hardware, and the scheduler keeps tightly coupled jobs inside one locality domain. TechnoLynx designs and validates multi-GPU cluster scheduling and networking within GPU Performance Engineering. It measures your training job’s scaling on the candidate fabric, sets the NCCL and placement configuration, and repeats the sweep at the planned GPU count before you commit.

A folded-Clos (fat-tree) fabric grows by adding spine switches rather than replacing existing hardware. Overlaid’s guide, Fabric Topologies for GPU Clusters (2026), recommends rail-optimised Clos as the default anywhere from tens of GPUs to several thousand in one pod, and reserves Dragonfly variants for clusters of tens of thousands. One reference design is NVIDIA’s DGX H100 SuperPOD: scalable units of 32 DGX H100 systems, a rail-optimised, balanced full fat tree of NDR switches, and 400 Gbps NDR InfiniBand. A reference architecture gives you a tested starting point, but it does not tell you how your job scales on it.

Link speeds alone cannot tell you whether HDR InfiniBand is enough for a given risk book or whether you need NDR. Run the strong-scaling sweep on each and compare communication fractions. An InfiniBand design guide from AscentOptics suggests first confirming that the workload justifies InfiniBand at all, typically above 256 GPUs for training.

Placement matters as well as the fabric. NVIDIA describes how poor placement fragments topology domains and pushes traffic across shared links, cutting throughput. Its open-source Topograph discovers topology from cloud APIs, or from InfiniBand and NetQ on-premises, and exports it as Slurm configuration or Kubernetes labels. KAI Scheduler and Kueue can use those labels for topology-aware gang placement. For rented A100s spread across availability zones, ask the provider for a placement group or interconnect tier that keeps the job inside one locality domain, then measure the scaling difference before paying the premium. How much it buys depends on your job’s communication fraction.

Parallelism strategy and framework also change the result. CCL-Bench found that TorchTitan and Megatron-LM reached their best Llama-3.1-8B step times at different TP/DP/PP settings on 16 GPUs, with a 3.4x gap between the two frameworks’ best results. Large data-parallel minibatches also change convergence: MLPerf Training reports that ResNet-50 needed more than 80 epochs at a 16K minibatch, against about 64 at 4K. Moving from NCCL to a custom topology later costs an amount that is hard to estimate in advance. Keeping communication behind the framework’s collective interface limits that cost, and a head-to-head benchmark at the planned scale tells you whether you will ever need to pay it.

Turning genomics batches and an LLM tutoring pilot into a spec vendors can quote against

Size each workload on its own terms (GPU memory, interconnect bandwidth and storage throughput), then combine them at peak concurrent demand across departments rather than average utilisation, because batch jobs and interactive serving collide.

LynxBenchAI publishes results showing how each candidate GPU runs the public PyTorch stack, so the first step for the tutoring serving line is to check those results for the devices a vendor proposes. Those results give broadly indicative per-device figures for the serving line of the spec, though they do not measure your genomics pipelines or your campus traffic.

1. Tutoring: GPU memory per session (model weights, context length, batch) multiplied by expected peak concurrent sessions.
2. Genomics: memory, storage and interconnect footprint per batch job multiplied by expected concurrent jobs. For scale, AWS’s PacBio benchmark lists DeepVariant calling at 64 CPUs and 239 GB of RAM for HiFi reads, with GPU acceleration available.
3. Interconnect: sized for the worst case where both workloads peak at once, not the average.
4. Storage: sized separately for large sequential genomics I/O and for tutoring’s lighter, latency-sensitive access.

Overlaid’s guide, Fabric Topologies for GPU Clusters (2026), makes the same point about fabrics: non-blocking provisioning is based on assumed traffic patterns, so size and monitor against realistic worst-case concurrent bursts, especially in multi-tenant fabrics. Sizing for peak costs more. A campus may prefer a cost-versus-SLA trade-off, such as queuing in a degraded mode while the two workloads overlap, enforced by scheduling policy rather than extra hardware. Department forecasts are uncertain, so revisit the spec after a pilot period with production monitoring of real utilisation data.

How do we keep the cluster benchmark reproducible and auditable for contract renewal?

Keep the evidence, not only the results: the execution trace, a card describing the workload and infrastructure, and the scripts that launched each run. CCL-Bench uses that structure so that performance can be attributed without rerunning anything. MLPerf Training does the same in public, publishing system descriptions, logs and all the code needed to reproduce each submission. It also requires several runs per benchmark, drops the fastest and slowest, and averages the rest.

ClusterBench reuses the same pre-compiled binaries across repeated runs to remove build-to-build variance, and stores each result with its node, timestamp and software environment. Keep your acceptance-day component baselines as well, since nvbandwidth figures taken then are what later runs are compared against to catch regressions.

At renewal, rerun the same harness on the same workload cards on the hardware you are paying for. The negotiation then turns on whether the cluster still delivers what you measured at acceptance, settled with your own numbers rather than the vendor’s account. One thing stays uncertain however carefully you record: whether the workload itself has changed since acceptance. So version the workload cards alongside the results, and treat a renewal benchmark on an outdated job profile as no better than a vendor benchmark.

FAQ

Does NCCL use RDMA?

Yes. NCCL uses GPUDirect RDMA, NVLink and other hardware acceleration to get low-latency, high-bandwidth communication between GPUs.

What is NVIDIA NCCL used for?

NCCL is NVIDIA’s library of collective communication operations, such as all-reduce, for multi-GPU and multi-node systems. It integrates with frameworks including PyTorch and TensorFlow, and is a key component in scaling deep learning workloads.