Copy a working flag set from a tutorial, paste it into your build script, and forget about it. That habit survives right up until the day a GPU-ported RF propagation simulation runs fast but produces coverage maps that no longer match the CPU reference. The kernel compiled. It launched. It returned numbers. The numbers are just quietly wrong — off by enough to move a predicted cell-edge boundary, not enough to crash anything. Compile flags are not boilerplate. In a CUDA port of a physics-based simulation, the flag set is a deliberate part of the port: it decides which GPU architecture the code actually runs on, whether the compiler is allowed to reorder floating-point arithmetic, and how aggressively kernels get optimised. Get them right and the multi-day-to-hours speedup your algorithmic redesign earned is trustworthy. Get them wrong and you either leave performance on the table or trade correctness for speed without noticing. What does working with a compile flag involve in practice? A compile flag is an instruction to the compiler — nvcc, in CUDA’s case — that changes how source code is translated into machine instructions. It does not change what your code says. It changes what the compiler is permitted to assume while turning that code into something the GPU executes. That distinction matters more than it sounds. Some flags are purely about output selection: which target architecture, whether to embed debug symbols. Others hand the compiler latitude to rewrite your arithmetic — reassociate additions, replace a division with a reciprocal-multiply, fuse a multiply and add into a single rounding step. When you pass a flag like that, you are telling the compiler the result “close enough” is acceptable. For a lot of code it is. For a simulation whose whole value is agreement with a reference model, “close enough” is a decision you should make on purpose, not inherit from a copied build line. The nvcc toolchain sits on top of a host compiler (typically GCC or Clang) and adds its own device-side flags. So a CUDA build actually carries two flag sets: host flags that shape the CPU-side glue and device flags that shape the kernels. Confusing the two is a common early mistake. Our companion piece on tuning nvcc, Clang, and SYCL builds walks the toolchain split in detail; this article stays on what the device-side choices do to a simulation. Which GPU compile flags actually affect performance versus just correctness? Not every flag pulls the same lever. It helps to separate them by what they change — because the ones that change numerical results are the ones that bite silently, and the ones that change performance are the ones people reach for first. Here is the split that matters most in a CUDA port: Flag family Example Changes performance? Changes numerical results? When it bites Architecture target -arch=sm_90, -gencode Yes — big No Wrong target → JIT recompile at launch, or no run at all Optimisation level -O3 (host), -O2 device default Yes Rarely Under-optimised host glue starves the GPU Fast-math --use_fast_math, -ffast-math Yes — moderate Yes Results drift from CPU reference; no error raised FMA control --fmad=true/false Small Yes (rounding) Subtle last-bit differences accumulate over iterations Precision selection -default-stream, dtype in code Yes — large Yes FP32 vs FP64 changes both speed and fidelity Debug -g -G Yes — slows device code No Left in accidentally → 10× slower kernels The pattern worth internalising: architecture and optimisation flags are almost entirely about speed and almost never about correctness. Fast-math and FMA flags are the reverse — modest speed gains bought with real changes to what the arithmetic produces. Debug flags are the sneaky one; -G disables device-side optimisation entirely, and a -g -G left in a release build is one of the most common reasons a “GPU port” looks disappointing. We see this pattern regularly when auditing a port that “should be faster.” What does targeting a specific GPU architecture change, and what happens if it’s wrong? Every NVIDIA GPU has a compute capability — a version number describing which hardware features it exposes (sm_70 for Volta, sm_80 for Ampere A100, sm_90 for Hopper H100, and so on). The -arch and -gencode flags tell nvcc which of these to compile device code for. When you compile for the exact architecture the code will run on, nvcc emits SASS — the actual machine code for that GPU. It runs immediately. When you compile only to a virtual PTX target and run on a newer card, the CUDA driver has to JIT-compile PTX to SASS at kernel launch. That JIT step is a one-time cost per process, but in a simulation that spins up short-lived kernels or launches from a fresh process repeatedly, the recompilation overhead is real and recurring. Matching the target architecture to the deployment hardware is what avoids it — per NVIDIA’s published CUDA documentation, embedded SASS for the running device skips JIT entirely. Get the target wrong in the other direction — compile for sm_90 and try to run on an sm_80 card — and the kernel simply will not launch. That is the good failure mode: it’s loud and immediate. The quiet failure is compiling for an older architecture than you have, which works but leaves newer instructions (better tensor paths, more efficient memory ops) unused. The code runs correctly and slower than it should, and nothing tells you. A robust build uses -gencode to embed SASS for every architecture in your fleet plus a PTX fallback for forward compatibility. That is the practical answer to “which architecture do I target”: all the ones you actually deploy on, explicitly. When do fast-math flags matter for a physics-based simulation’s numerical fidelity? This is where a CUDA simulation port diverges hardest from a machine-learning inference port. In inference, small numerical perturbations usually wash out — a classification is a classification whether the logit was 4.71 or 4.69. In a physics-based simulation, the whole point is that the output is the number, and downstream decisions depend on it. --use_fast_math bundles several relaxations: it flushes denormals to zero, enables less-precise but faster intrinsics for transcendental functions (sin, exp, sqrt, division), and permits FMA contraction. For an RF propagation model — path loss over distance, diffraction terms, interference summation across many contributions — those relaxations accumulate. A single approximated exp() in a path-loss term is tiny. Summed across thousands of propagation paths and iterated over a coverage grid, the drift can move a predicted signal boundary by enough to change a network-planning decision. That is the exact scenario where a validated multi-day-to-hours speedup quietly becomes an unvalidated one. The correct framing is not “fast-math is dangerous, never use it.” It is: fast-math is a numerical-tolerance decision, and only the person who owns the simulation’s accuracy requirements can make it. Sometimes the tolerance is generous and fast-math is free money. Sometimes a single --fmad=false to disable multiply-add contraction is what brings your GPU output back into agreement with the CPU reference. The failure is treating the flag as a performance knob when it is actually a fidelity knob. If you are also weighing precision formats alongside math mode, the trade-offs in what -O3, -march, and fast-math actually change generalise the same tension across CPU and GPU targets. Whether fast-math is acceptable is an observed-pattern judgment tied to each simulation’s tolerance — not a benchmarked rate you can quote once and reuse. We validate it per workload, against the reference, every time. How do I tell whether slow kernel performance is a flag problem or an algorithm problem? A GPU port that runs but underperforms sends people down the wrong debugging path constantly. They start rewriting kernels when the problem is a build flag, or they tune flags for a week when the real issue is that the algorithm was never memory-bandwidth-friendly. The two failure classes look identical from the outside — “the GPU version is slower than I expected” — so you need a way to tell them apart before spending effort. Diagnostic checklist: flag problem or algorithm problem? Work through these in order. The first few catch flag problems, which are cheap to fix; the later ones point at algorithmic issues, which are not. Are -g -G in the release build? Device debug disables kernel optimisation. If present, remove and re-measure before anything else. This alone explains many “disappointing port” reports. Does -arch match the running GPU’s compute capability? Check with nvidia-smi for the card, compare to your -gencode lines. A mismatch means JIT overhead or unused hardware features. Is optimisation actually on? Confirm -O3 on host glue and that device code is not being built in a debug configuration by the surrounding build system (CMake Debug vs Release catches people). Does the kernel’s compute-to-memory ratio explain the ceiling? Run a profiler (Nsight Compute). If the kernel is memory-bandwidth-bound at near-peak bandwidth, no flag will help — that is an algorithm and data-layout problem. Is occupancy the limiter? Low occupancy from register pressure or block sizing is structural, not a flag you flip. If items 1–3 are clean and the profiler still shows you pinned against a hardware limit, stop tuning flags. The port needs algorithmic work — memory coalescing, kernel fusion, better data layout — not a compiler switch. Recognising that boundary early is most of the skill. The broader question of what it even means to move a workload correctly is covered in our explainer on what it means to port a workload to new hardware. What compile-flag set is a sensible starting point for a CUDA simulation port? There is no universal correct set, but there is a sensible default posture to start from and then validate. Treat this as a baseline you adjust against your measurements, not a copy-paste answer. A reasonable starting configuration for a physics-based CUDA port: -arch/-gencode for every deployment GPU plus a PTX fallback. Explicit, not guessed. -O3 on the host compiler, standard device optimisation (the nvcc default), and no -g -G in release. Fast-math OFF by default. Start numerically conservative. Turn individual relaxations on only after validating against the CPU reference, and only where the tolerance allows. --fmad=true is the default and usually fine, but keep --fmad=false in your pocket as the first thing to try when GPU and CPU results diverge in the last few bits. Validate before you optimise. Get a correct, reference-matching GPU result first, measure its speed, then relax flags for performance while re-checking fidelity at each step. That last point is the whole method. The naive order is optimise-then-hope-it’s-correct; the reliable order is match-the-reference-then-earn-the-speed. Choosing GPU flags for a browser or mobile inference target follows different logic — memory footprint and instruction-set portability dominate there rather than reference fidelity — which is why the multi-platform edge inference flag choices frame the same knobs for a different deployment context. Any of this can be built on and profiled against your actual simulation on our GPU acceleration engineering practice, where flag configuration is treated as part of the port rather than an afterthought. FAQ What should you know about a compile flag in practice? A compile flag instructs the compiler — nvcc for CUDA — on how to translate source into machine code without changing what the code says. Some flags select output (target architecture, debug symbols); others grant the compiler latitude to rewrite arithmetic for speed. A CUDA build carries two flag sets: host flags for CPU-side glue and device flags for the kernels themselves. Which GPU compile flags actually affect performance versus just correctness? Architecture-target and optimisation-level flags almost entirely affect speed and rarely change results. Fast-math and FMA-control flags do the reverse: modest speed gains bought with real changes to the arithmetic output. Debug flags like -g -G disable device optimisation and are a common hidden cause of a slow port left in a release build. What does targeting a specific GPU architecture change, and what happens if it’s wrong? The -arch/-gencode flags select which compute capability nvcc compiles device code for. Match the running hardware and you get native SASS that runs immediately; target only PTX on a newer card and the driver JIT-compiles at launch, adding recurring overhead. Compiling for a newer architecture than you have prevents launch (a loud failure); compiling for an older one runs correctly but leaves newer instructions unused and slower. When do fast-math or floating-point flags matter for a physics-based simulation’s numerical fidelity? They matter whenever the simulation’s output value drives a downstream decision, as in RF propagation planning. --use_fast_math relaxes transcendental precision, flushes denormals, and permits FMA contraction; summed across thousands of paths and iterated over a grid, that drift can move a predicted boundary. Fast-math is a numerical-tolerance decision owned by whoever owns the accuracy requirement — not a free performance knob. How do I tell whether slow kernel performance is a flag problem or an algorithm problem? Work cheap-to-expensive: check for stray -g -G, confirm -arch matches the running GPU, confirm optimisation is actually enabled. If those are clean and a profiler (Nsight Compute) shows the kernel pinned against memory bandwidth or occupancy limits, stop tuning flags — the port needs algorithmic work like coalescing or fusion, not a compiler switch. What compile-flag set is a sensible starting point for a CUDA simulation port, and how do I validate it? Start with -gencode for every deployment GPU plus a PTX fallback, -O3 on host code, no -g -G in release, and fast-math off. Get a correct, reference-matching GPU result first, measure its speed, then relax flags for performance while re-checking fidelity at each step. Validate against the CPU reference before optimising, not after. When a build looks fast but its numbers drift, the failure class is a mismatch between the build configuration and the simulation’s numerical tolerance — exactly what an A1 GPU audit checks: compile flags, target architecture, and math mode against the hardware and the accuracy the workload actually requires.
