A team ports an inference path from Python to C++, benchmarks a stock build, sees a modest speedup, and files the port as “not worth it.” The build was compiled with defaults. That single decision quietly decided the verdict — and it decided it wrong. This is the failure we see most often when a port comes back looking marginal: the flag set was treated as an afterthought. Someone compiled with whatever cmake handed them, measured, and attributed the whole result to the fact that the code is now C++ instead of Python. But a default build can leave a large fraction of the achievable native-code gain on the table. When that happens, the number you measured isn’t the port’s ceiling — it’s the port’s floor, wearing the port’s name. Flags are a measurement variable, not a switch you flip and forget. If you want to know whether the port paid off, you have to separate two things that get collapsed into one: the gain from moving to a compiled language, and the gain from telling the compiler to actually optimise. They are different levers, and they attribute to different decisions. How does a compiler flag work, and why does it decide a port verdict? A compiler flag is an instruction to the toolchain — GCC, Clang, or emcc for WebAssembly — about how to translate your source into machine (or WASM) code, not what the code does. The output is functionally the same program; the flag changes how aggressively the compiler reorders, inlines, vectorises, and specialises the instructions it emits. The reason this matters for a port decision is that the “before” and “after” of your benchmark are two different builds, and if they were compiled differently, the comparison is rigged. A tuned Python path — say, one already leaning on NumPy’s vectorised BLAS backend — against a default -O0 or -O2 C++ build is not a like-for-like test. You are comparing someone’s optimised interpreter path against your unoptimised native path and calling the difference “the port.” We treat flag configuration as part of the port’s measured gain for exactly this reason. The same profiling baseline used to justify the port — the one from profiling the Python inference path before a C++ or WASM port — is the baseline against which each flag group must earn its place. The point is not to reach some abstract “fastest build.” It’s to attribute the latency delta correctly. What do -O2 vs -O3, LTO, and -march=native each actually change? These four levers change different things, and conflating them is where teams lose the plot. Here is a working decomposition. Flag / group What it changes Typical effect on inference Risk it introduces -O2 Standard optimisation: inlining, dead-code elimination, basic loop transforms The sane default; most of the “compiled-language” gain lands here Very low; safe for production -O3 Adds aggressive loop unrolling, vectorisation heuristics, more inlining Can help tight numeric loops; can also bloat code and hurt I-cache Occasionally slower than -O2; must be measured, not assumed -flto (LTO) Link-time optimisation: inlining and dead-code elimination across translation units Helps when hot code crosses file/library boundaries Longer build/link times; heavier debugging -march=native Emits instructions for the exact host CPU (AVX2, AVX-512, FMA) Often the largest single win on CPU inference — SIMD width on math kernels Binary won’t run on older/different CPUs; a portability trap Two of these deserve emphasis. -O3 is not a strictly better -O2; it trades instruction-level aggressiveness for code size, and on a memory-bound inference kernel the extra unrolling can regress. Treat -O3 as a hypothesis to test against -O2, not an upgrade. -march=native is usually where the real native-code gain hides, because it unlocks the CPU’s wide vector units on your matmul and activation loops — the same SIMD-width story we cover in what SIMD width buys a ported inference path. But it bakes the host’s instruction set into the binary. Compile with -march=native on an AVX-512 build server, deploy to an AVX2 target, and the process dies on an illegal instruction. If your deployment fleet is heterogeneous, you want -march=x86-64-v3 or an explicit baseline, not native. For the toolchain-specific mechanics of these switches on GPU host code and beyond, GPU compilation flags explained: tuning nvcc, Clang, and SYCL builds goes deeper on the compiler side; this article stays on the attribution question. How much of a port’s gain is the language versus the compiler? This is the question the whole exercise exists to answer, and the honest reply is: you don’t know until you measure it in groups. Attribute by layering flags against a single fixed baseline and recording the delta at each step. Here is a worked attribution structure — the numbers are illustrative placeholders to show the shape of the measurement, not results from any specific run: Assume a profiled Python baseline latency of L_py per inference, and each build measured on the same input distribution, same warm cache, same core-pinning. -O0 C++ build vs L_py → this is the “language only, no optimisation” delta. Often disappointing, sometimes slower than tuned Python. -O2 build vs step 1 → the baseline-optimisation contribution. Usually the largest jump. + -flto vs step 2 → the cross-module inlining contribution. Small unless hot paths span libraries. + -march=native vs step 3 → the vectorisation contribution. On dense-math kernels, frequently the second-largest jump after -O2. Run this ladder and the verdict changes character. A port that looked like a 1.3× win at -O0 might be a 3–4× win at -O2 -flto -march=native (observed pattern across ported CPU inference paths we’ve worked on; not a benchmarked rate, and the multiplier depends heavily on how vectorisable the model’s hot loops are). Crucially, most of that additional gain is compiler-driven, not language-driven. If you report it all as “the port,” you mislead the next decision. If you skip it entirely by shipping the -O0 number, you kill a port that would have paid off. When does -ffast-math help, and what does it break? -ffast-math (or its component flags like -funsafe-math-optimizations and -ffinite-math-only) tells the compiler it may assume IEEE-754 corner cases don’t happen: no NaNs, no infinities, associativity of floating-point addition, reciprocal approximations. That assumption lets it reorder and fuse floating-point operations it otherwise couldn’t, which can meaningfully speed up reduction-heavy code. The catch is that it can silently change your numbers. Floating-point addition is not associative; fast-math lets the compiler pretend it is. For a softmax, a layer-norm, or an accumulation over a long sequence, the reordered arithmetic can shift the output by more than you’d assume — enough to flip a classification near a decision boundary or drift a detection score. This is a real risk for inference, not a theoretical one. So the rule we apply: never enable -ffast-math as part of the “make it fast” reflex. Enable it only after you’ve measured the numerical divergence explicitly — run the fast-math build and the strict build against the same inputs and compare outputs, not just latency. If the divergence is within your model’s tolerance (and you’ve defined that tolerance), keep it. If you can’t state the tolerance, you can’t use the flag. The same discipline applies on WASM builds via emcc, covered in compiler flags for WASM inference: what they do to Pyodide performance. Which flags carry over to a WASM/Emscripten build? Not all of the native flag set survives the trip to WebAssembly. The optimisation levels do — emcc -O2 and -O3 map onto the same LLVM optimisation pipeline. LTO carries over (-flto) and often matters more in WASM because the module is a single deliverable. Fast-math carries over too, with the same numerical caveats. What does not carry over is -march=native. There is no “host CPU” at compile time for a WASM module — it targets a virtual instruction set. What you get instead is the SIMD-128 proposal: -msimd128 enables WebAssembly’s fixed 128-bit vector instructions, which is the WASM analogue of unlocking vector units, but at a fixed width rather than the host’s actual (possibly 256- or 512-bit) width. That single difference is why a WASM port of the same kernel typically lands below the -march=native C++ number: the vector width is capped by the standard, not the silicon. GCC compiler flags that matter for inference and WASM builds works through the native/WASM split flag by flag. What flag defaults should a fair port-or-not benchmark use? Use the same optimisation posture on both sides of the comparison, and document it. A fair port benchmark isn’t “Python as-is vs C++ as-is” — it’s “best reasonable Python path vs best reasonable native path,” where “best reasonable” is defined once and applied symmetrically. In practice that means the native build is measured at -O2 -march=<deployment-target> minimum, with -flto and -march=native reported as separate attribution rows, and fast-math excluded unless numerically cleared. This is the discipline the [inference cost cut pack](Inference Cost-Cut Pack) folds into its port-decision step: flag configuration becomes part of the measured baseline so the profiling comparison is like-for-like optimised builds, not a tuned interpreter path against a default native one. The comparison is only decision-grade when both sides were built with equal care. FAQ How does a compiler flag work, and what does it mean in practice for a ported inference path? A compiler flag instructs the toolchain how to translate source into machine or WASM code — how aggressively to inline, vectorise, and reorder — without changing what the program computes. For a ported inference path it means the “before” and “after” builds can differ purely in flags, so a default build compared against a tuned interpreter path is not a like-for-like test and produces a misleading verdict. What do -O2 vs -O3, LTO, and -march=native each actually change in the generated code? -O2 is the safe standard optimisation level and usually captures most of the compiled-language gain. -O3 adds aggressive unrolling and vectorisation heuristics that can help tight numeric loops but sometimes regress on memory-bound kernels. -flto enables optimisation across translation units, and -march=native emits instructions for the exact host CPU (unlocking AVX2/AVX-512), often the single largest CPU-inference win — at the cost of binary portability. How much of a port’s measured latency gain is the language versus the compiler configuration? You cannot know without layering flags against one fixed profiling baseline and recording the delta at each step: -O0 (language only), then -O2, then +LTO, then +-march=native. Much of the total gain is typically compiler-driven rather than language-driven, so reporting the whole speedup as “the port” misattributes it, while shipping the -O0 number understates the port and can kill a worthwhile one. When does -ffast-math help, and what numerical risk does it introduce for inference outputs? -ffast-math lets the compiler assume no NaNs/infinities and treat floating-point addition as associative, which speeds up reduction-heavy code. The risk is that reordered arithmetic can silently shift outputs — enough to flip a classification near a decision boundary — so it should only be enabled after explicitly measuring output divergence against a strict build and confirming it stays within a defined tolerance. Which flags carry over to a WASM/Emscripten build, and which are native-only? Optimisation levels (-O2/-O3), LTO, and fast-math all carry over to emcc/WASM builds with the same behaviour and caveats. -march=native is native-only; WASM has no host CPU at compile time, so you use -msimd128 for fixed 128-bit vectors — a capped vector width that usually explains why a WASM port lands below the -march=native C++ number. How do we attribute a flag’s contribution against the port-assessment profiling baseline? Measure each build against the same profiled baseline under identical conditions — same inputs, warm cache, core-pinning — and record the latency delta as each flag group is added. Each delta attributes to a specific decision (language, base optimisation, cross-module inlining, vectorisation) rather than being lumped into a single “port” number. What flag defaults should a fair port-or-not benchmark use so the comparison isn’t rigged? Apply the same optimisation posture symmetrically: measure the native build at -O2 -march=<deployment-target> minimum, report -flto and -march=native as separate attribution rows, and exclude fast-math unless numerical divergence has been cleared. The comparison is only decision-grade when both the interpreter path and the native path were built with equal care. The deeper lesson is that the flag set isn’t tuning you do after the port decision — it’s part of the decision. Before you conclude a port did or didn’t pay off, ask which build produced the number, and whether the other side of the comparison was held to the same standard. A verdict from an unoptimised build is a verdict about your cmake defaults, not about the port.
