A team copies an accelerate launch command from a tutorial, points it at eight GPUs, and watches training speed up by less than 2x. The kernels are fine. The model is fine. The launch configuration decided, before a single forward pass ran, how the work would be split across devices — and it split it badly. This is the part of multi-GPU training that gets read as boilerplate. You install Hugging Face Accelerate, run accelerate launch train.py, and treat the command as a thin wrapper around python. It is not. accelerate launch is where you commit to a decomposition strategy: how many processes run, whether each device holds a full model replica or a shard of one, and what numerical precision the compute runs in. Those are algorithmic decisions about the shape of the computation, not micro-optimizations you tune afterward. Change them and you change the arithmetic the GPUs actually perform. What does accelerate launch do when you run it? At its core, accelerate launch reads a configuration — either from the interactive accelerate config file or from flags on the command line — and then spawns and coordinates a set of worker processes, typically one per GPU. Each process runs your training script, but each one knows its own rank, the total world size, and the communication backend (NCCL on NVIDIA hardware) that stitches the processes into a single logical training job. That coordination is the whole point. A single python train.py invocation sees one device and one process. accelerate launch --num_processes 8 sees eight processes that must agree on gradients, share optimizer state where required, and synchronize at defined points. How they agree — all-reducing full gradients versus exchanging only shards — is set at launch. The kernels inside each process are identical; what differs is the collective-communication pattern wrapped around them. The reason this matters is structural. When you scale from one GPU to many, the bottleneck moves. On a single device you are usually compute-bound or memory-bound within that device. Across eight devices connected over NVLink or PCIe, you become bound by how much data crosses the interconnect and how often the devices have to wait for each other. A launch configuration that ignores this leaves devices idle — the classic symptom of a multi-GPU run that scales far worse than the device count suggests. In configurations we have worked through, moving multi-GPU utilization out of the common 40–60% range toward 85%+ is almost always a launch-and-decomposition problem, not a kernel one (observed pattern across GPU engagements; not a benchmarked figure). Which accelerate launch flags actually change the GPUs’ work? Three flags do most of the structural work. The rest are plumbing. --num_processes sets how many worker processes join the job — usually one per GPU. This is the coarsest lever: it defines the world size and therefore how the batch is split and how gradients are reduced. Under plain data parallelism, doubling processes roughly doubles the aggregate batch you can push per step, but it also doubles the volume of gradient all-reduce traffic across the interconnect. --mixed_precision (fp16 or bf16) changes the numerical format of the compute. This is not just a memory trick. Lower-precision matrix multiplies map onto Tensor Cores that process them at higher throughput than fp32, and the smaller activation and gradient tensors reduce interconnect traffic per step. On modern NVIDIA parts bf16 is usually the safer default because its wider exponent range avoids the loss-scaling gymnastics fp16 sometimes needs. The sharding strategy — expressed through --use_fsdp and the Fully Sharded Data Parallel config, or through a DeepSpeed ZeRO plugin — is the deepest lever of all. It decides whether each device holds a full copy of the model, optimizer, and gradients, or whether those are partitioned across devices and gathered on demand. That single choice determines whether a model even fits, and it changes the communication pattern from “all-reduce gradients once per step” to “all-gather parameters layer by layer during the forward and backward pass.” Quick reference: what each launch lever changes Launch lever What it changes on the GPUs When it is the right move --num_processes World size; batch split; gradient all-reduce volume Scaling out when the model fits on one device --mixed_precision bf16 Compute format; Tensor Core throughput; tensor sizes on the wire Almost always, on hardware with bf16 support FSDP / ZeRO sharding Full replica vs partitioned params, grads, optimizer state When the model no longer fits, or optimizer memory dominates --gradient_accumulation_steps Effective batch without more devices or memory When you want a larger effective batch than memory allows The point of the table is that each row restructures the computation differently. Reading it as one undifferentiated “make it multi-GPU” setting is exactly the mistake that leaves throughput on the floor. When is a launch configuration an algorithmic decision, not kernel tuning? Here is the distinction that separates the two mindsets. Kernel tuning asks: given this exact sequence of operations, how do I make each operation run faster? Fusing attention into a FlashAttention kernel, letting torch.compile build a fused graph, picking a better TensorRT engine — these all take the computation as fixed and shrink the time each step takes. Launch configuration asks a prior question: what sequence of operations should run on each device in the first place? Data parallel, sharded data parallel, and pipeline parallel are three different answers, and they produce genuinely different compute-and-communication schedules. Switching from replication to sharding does not make a kernel faster; it changes which tensors exist on which device and when they move. That is why we treat it as algorithmic. It belongs to the same class of decision as choosing an attention variant or a batching scheme — the structure of the work, not the polish on it. This is the boundary a GPU performance audit is built to draw. When the audit produces an optimization roadmap, launch-configuration changes — process count, sharding strategy, precision — land in the algorithmic bucket (a C2-class decision), separate from the kernel-level bucket. accelerate launch happens to be the concrete place where several of those algorithmic decisions get expressed in a single command line, which is why it deserves more attention than its tutorial-boilerplate reputation gives it. The practical consequence: if you tune kernels before you fix the decomposition, you are polishing operations that a better launch config would restructure or remove. Fix the structure first. It is the same order-of-operations principle that runs through our work on multi-GPU training and inference with Hugging Face Accelerate — the framework abstracts the boilerplate so you can reason about the decomposition, not so you can ignore it. How does the parallelism strategy interact with batch size and occupancy? The levers are coupled, and this is where many multi-GPU plans quietly underperform. Occupancy — how fully the GPU’s streaming multiprocessors are kept busy — depends heavily on having enough work in flight per device. If you shard a model across eight GPUs but keep the per-device batch tiny, each device may be underutilized even though the aggregate looks impressive. Sharded data parallelism changes the arithmetic in a useful way. Because FSDP partitions parameters, gradients, and optimizer state, each device frees up memory it would otherwise spend holding full replicas. That freed memory can go toward a larger per-device batch, which raises occupancy and improves memory-bandwidth utilization — you are moving more useful data through HBM per unit time rather than leaving bandwidth idle. So the decision to shard is not only about fitting a model; it can activate batch sizes that improve the efficiency of every kernel that runs. Our deeper treatment of this lives in how Fully Sharded Data Parallel works in practice, which walks through the parameter-gathering pattern that makes the trade-off work. The counterweight is communication. Sharding gathers parameters layer by layer, which adds all-gather traffic the forward and backward passes must wait on. On a well-connected system — NVLink between devices, high HBM bandwidth — that overlap is cheap and the batch-size win dominates. On a PCIe-only topology it can eat the gains. This is why the same launch config can be excellent on one machine and mediocre on another, and why there is no universally correct answer to publish. The right configuration is the one that keeps devices busy on your interconnect. How do I tell a launch-config problem from a kernel ceiling? The diagnostic is mostly about where the time goes. A launch-config problem shows a specific signature; a kernel ceiling shows a different one. Device utilization sits at 40–60% and stalls line up with synchronization points. When nvidia-smi or a profiler shows GPUs idling in step with all-reduce or all-gather operations, the decomposition is starving the devices. That is a launch-config problem. Scaling is far below linear. If eight GPUs deliver closer to 2–3x the throughput of one rather than something near-linear, the launch strategy — process count, sharding, or precision — is the first suspect, well before any individual kernel. Devices are near-saturated and the hot kernels dominate the timeline. If utilization is already high and the profiler shows most time inside a handful of well-defined kernels, you are closer to a genuine kernel ceiling, and tools like torch.compile, FlashAttention, or a TensorRT engine become the right next move. Memory, not compute, is the wall. If you are out of memory before you can raise the batch, that is a sharding decision waiting to be made, not a kernel to optimize. Run this check before reaching for kernel-level tools. The failure mode we see most often is a team that spends weeks on kernel optimization while a one-line launch-config change would have unlocked most of the speedup — a misread of the bottleneck that a short profiling pass would have caught. FAQ What’s worth understanding about accelerate launch first? accelerate launch reads a configuration and spawns a set of coordinated worker processes — typically one per GPU — each running your training script with knowledge of its rank, the total world size, and the communication backend (NCCL on NVIDIA). In practice it is the point where you commit to a decomposition strategy: how the batch is split, how gradients are reduced, and how devices synchronize. It is not a thin wrapper around python; it defines the collective-communication pattern the whole job runs under. What do the key accelerate launch flags (num_processes, mixed_precision, sharding strategy) actually change on the GPUs? --num_processes sets the world size, which governs the batch split and gradient all-reduce volume. --mixed_precision (bf16 or fp16) changes the compute format, mapping matrix multiplies onto Tensor Cores for higher throughput and reducing tensor sizes on the interconnect. The sharding strategy (FSDP or ZeRO) decides whether each device holds a full replica or a partition of parameters, gradients, and optimizer state — which changes the communication pattern from a single per-step all-reduce to layer-by-layer all-gather. When is choosing a launch configuration an algorithmic decision rather than a kernel-tuning one? Kernel tuning takes the computation as fixed and makes each operation faster; launch configuration decides what work runs on each device in the first place. Data parallel, sharded data parallel, and pipeline parallel are different answers that produce genuinely different compute-and-communication schedules, so switching between them restructures the arithmetic rather than polishing it. That is why we classify process count, sharding, and precision as algorithmic (C2-class) decisions. How does the launch-time parallelism strategy interact with batch size, GPU occupancy, and memory bandwidth? Occupancy depends on having enough work in flight per device, so a tiny per-device batch underutilizes the GPUs even across many of them. Sharding frees the memory otherwise spent on full replicas, which can go toward a larger per-device batch that raises occupancy and moves more useful data through HBM. The counterweight is added all-gather traffic, so the win holds on NVLink-connected systems and can erode on PCIe-only topologies. How do I tell whether slow multi-GPU training is a launch-config problem or a kernel-level ceiling? Profile where the time goes. Utilization stuck at 40–60% with stalls aligned to synchronization points, or far-below-linear scaling, points to the launch configuration. Near-saturated devices with time dominated by a handful of hot kernels — or hitting a memory wall before you can raise the batch — points instead to a kernel ceiling or a sharding decision. Run this check before reaching for kernel-level tools. What launch configuration should I pick when a model no longer fits on a single GPU? Move from plain replication to sharded data parallelism (FSDP or a ZeRO plugin), which partitions parameters, gradients, and optimizer state so the model fits and frees memory for a larger per-device batch. Pair it with bf16 mixed precision on supported hardware to cut tensor sizes and raise Tensor Core throughput. Validate on your actual interconnect: the added all-gather traffic is cheap on NVLink and can be expensive on PCIe-only systems. How does accelerate launch relate to the optimization roadmap in a GPU performance audit? A GPU performance audit’s roadmap separates algorithmic changes from kernel-level ones. Launch-configuration changes — process count, sharding strategy, mixed precision — fall in the algorithmic bucket, and accelerate launch is the concrete command line where several of those decisions are expressed. Reading it that way keeps you fixing the decomposition before polishing kernels that a better launch config would have restructured. The honest close is that there is no launch configuration to memorize. The right --num_processes, precision, and sharding strategy depend on your model’s memory footprint and the interconnect between your devices — and the only way to know whether a slow run is starved by its decomposition or capped by its kernels is to profile the schedule and watch where the GPUs wait. Get the structure right at launch time first; the kernel work you do afterward will then actually move the number.
