You profiled the Python inference path, decided Cython was worth it, and cythonized the hot function. The .pyx compiles, the .c file gets emitted, and then a build toolchain you never configured compiles that C with whatever flags it defaults to. That last step is where a large share of the speedup you were counting on either lands or quietly disappears. Cython does not make your code fast. It emits C, and the C compiler makes that code fast — or not, depending on the flags. Treating compilation as a black box that “just works” leaves measurable performance on the table, and in the case of one flag in particular, it can silently change what your model outputs. The flags that matter are few, and each one is a tool with a cost, not a free switch you toggle to on. How GCC compiler flags work in a Cython build When you build a Cython extension, setuptools (or meson, or whatever build backend you use) invokes GCC on the generated .c file to produce a shared object — a .so on Linux. The flags passed to that GCC invocation control how aggressively the compiler transforms your code, which CPU instructions it is allowed to emit, and what numerical shortcuts it may take. The default flags are set by the Python distribution that built your interpreter, not by you. On many CPython builds the C extensions inherit -O2 and a conservative baseline architecture, but this is not guaranteed and varies across distributions, conda channels, and manylinux wheels. The practical consequence: unless you override extra_compile_args, you are shipping whatever the toolchain packager chose, which was tuned for portability across an unknown fleet of machines — not for your deployment target. The distinction that matters here is the one that separates a Cython decision from a compilation decision. Deciding whether to use Cython is a profiling question, covered when we discuss profiling the Python inference path before a C++ or WASM port. Deciding how to compile the C that Cython emits is a separate question, and it is the one that determines whether the profiled speedup materialises. Get the first right and the second wrong, and you paid the porting cost for a fraction of the return. Which GCC flags matter for a Cython C-extension? Most of GCC’s hundreds of options are irrelevant to a numeric inference extension. Four decisions carry almost all the weight, and each answers a different question. Flag What it controls Typical effect on a hot numeric loop Cost / risk -O2 Optimisation level (safe default) Large improvement over -O0/-O1; inlining, loop optimisation, register allocation None material — this is the sane baseline -O3 Aggressive optimisation Adds vectorisation and more aggressive inlining; sometimes helps, sometimes neutral Larger binaries; occasional regressions; must be measured, not assumed -march=native Instruction-set target Lets GCC emit AVX2/AVX-512 for the build host’s exact CPU Binary tied to build host’s CPU family — may crash with illegal-instruction on older deployment CPUs -ffast-math IEEE 754 floating-point strictness Reorders and fuses FP ops, enables faster math; can help vectorisation Changes numerical results — can alter inference outputs That is the whole surface for most extensions. The interesting engineering is not in knowing the flags exist — it is in knowing which of them is safe against your deployment target and your correctness budget. We treat those two constraints, not a copied build recipe, as the inputs that decide the flag set. What is the difference between -O2 and -O3, and when is the extra worth it? -O2 is the workhorse. It turns on the optimisations that reliably pay off — function inlining, loop-invariant code motion, register allocation, dead-code elimination — without the compiler making risky size/speed trade-offs. For most Cython extensions, -O2 recovers the bulk of the C-level speedup you were expecting. -O3 adds more aggressive inlining and, importantly, auto-vectorisation of loops GCC judges profitable. On a dense numeric kernel — the kind of tight loop you cythonized in the first place — -O3 can help. But it is not a guaranteed win. The extra inlining can bloat the instruction cache footprint and, in cases we have measured, produce a build that is slightly slower than -O2 on the same workload (observed pattern across porting engagements; not a benchmarked rate you should port blindly). GCC’s own documentation is explicit that -O3 does not always outperform -O2. The rule that holds up: start at -O2, measure, then try -O3 on the specific hot path and keep it only if the measurement improves. -O3 is a hypothesis to test, not a default to assume. When does -march=native help, and what does it cost? -march=native tells GCC to compile for the exact microarchitecture of the machine doing the build. On a modern CPU that means it can emit AVX2 or AVX-512 instructions, wider SIMD, and architecture-specific scheduling. For a vectorisable numeric loop this is often the single largest lever after -O2 — the difference between scalar and 256-bit or 512-bit vector math on the hot path. The cost is portability, and it is a hard one. A binary built with -march=native on an AVX-512 build server will execute illegal instructions — and crash — on a deployment machine that only supports AVX2, or on an older cloud instance type. This is the classic “works on my machine” failure with a compiler behind it. The same trade-off shows up in what -O3, -march and fast-math actually change for ported inference, where the deployment fleet is heterogeneous by design. If you control the deployment CPU exactly — a fixed on-premise box, a pinned cloud instance type, an edge device you specify — -march=native (or the more explicit -march=<specific-arch>) is safe and worth it. If you do not know the deployment CPU, or the fleet is mixed, do not use -march=native. Target a conservative baseline like -march=x86-64-v2 (roughly SSE4.2-era) or -march=x86-64-v3 (AVX2-era) that you know every target supports, and accept the smaller gain in exchange for a binary that runs everywhere it lands. What does -ffast-math change, and when is it acceptable? -ffast-math is the flag that requires the most judgment, because it does not just make things faster — it makes them different. It relaxes IEEE 754 semantics: the compiler is allowed to reorder floating-point operations, assume no NaNs or infinities appear, fuse multiply-add operations, and treat a + b + c as freely re-associable. Those relaxations unlock optimisations, especially vectorisation, that strict IEEE compliance forbids. The problem is that floating-point arithmetic is not associative. Reordering a reduction changes the rounding, and the result drifts. For a lot of code the drift is in the last bit and nobody cares. For an inference path, “the last bit” can propagate: a softmax that tips a classification the other way, a threshold that flips, a numerically sensitive normalisation that diverges from the reference output your validation suite was built against. Our position is simple. -ffast-math is acceptable when you have a reference output and a tolerance, and you have verified that the flag’s drift stays inside that tolerance for your model. It is not acceptable as a speculative “make it faster” toggle on a path whose outputs feed a decision. If you cannot afford to validate the numerical change, you cannot afford the flag. The relaxed semantics are a trade you make deliberately against a stated correctness budget — the same discipline the porting and performance-assessment methodology uses to decide what is safe before any code is touched. How do I verify a flag set improved latency without changing outputs? A flag change has two things to prove: it made the hot path faster, and it did not change what the model produces. Both need a measurement, and the order matters — check correctness first, because a faster wrong answer is not an improvement. Use this checklist before you keep any non-default flag set: Freeze a reference output. Run the model on a fixed input set with the default (or strict-IEEE) build and save the outputs. This is your correctness baseline. Rebuild with the candidate flags and re-run the same inputs. Compare against the reference with an explicit tolerance — bit-exact for anything without -ffast-math, and a documented numerical tolerance if -ffast-math is in play. Measure latency on the deployment target, not the build host — especially if -march is involved. A gain on the build CPU is meaningless if it does not appear on the machine that runs in production. Confirm the binary actually runs on every target CPU. If you used -march=native, verify against the oldest CPU in the fleet, or you will find the illegal-instruction crash in production instead of in CI. Record the delta. The additional latency reduction from -O3 and -march tuning over the default flags is the number that justifies keeping the more aggressive build; if it is negligible, ship the safer flags. This is where the compile step connects back to the cost case that put you on this path. The whole point of a positive Cython-vs-rewrite decision was a latency and cost improvement on the hot path, and correct flag selection is what realises the speedup the assessment predicted — the concern our Inference Cost-Cut Pack scopes at the build step. Sensible defaults when you don’t know the deployment CPU The common real-world case is that you are building a wheel or a container that will run on machines you have not enumerated. In that situation the safe default set for a Cython numeric extension is: -O2 as the baseline — reliable, no surprises. A conservative -march baseline you know every target supports (for example -march=x86-64-v2 or v3), not -march=native. No -ffast-math, unless you have a validated tolerance for the specific model. This gives you most of the C-level speedup with a binary that runs anywhere in the target class. When you later pin the deployment CPU — a fixed edge device, a specific instance type — you can rebuild with -march tuned to that exact target and re-measure. The point is to earn the aggressive flags with a measurement against a known target, not to inherit them from a copied recipe. If your builds are heading toward WebAssembly rather than native, the same reasoning takes a different shape in GCC arguments for WASM and native inference builds. FAQ What does working with gcc compiler flags involve in practice? When you build a Cython extension, the build backend invokes GCC on the C code Cython emits, and the flags passed to that invocation control how aggressively the code is optimised, which CPU instructions are allowed, and what numerical shortcuts are taken. In practice the defaults come from whoever packaged your Python interpreter and were tuned for portability, not for your deployment target — so unless you override extra_compile_args, you are shipping someone else’s compromise. Which GCC flags matter most when compiling a Cython C-extension for inference, and what does each one do? Four decisions carry almost all the weight: -O2 sets a safe optimisation baseline, -O3 adds aggressive inlining and auto-vectorisation, -march targets a specific instruction set, and -ffast-math relaxes IEEE floating-point semantics. -O2 and a conservative -march are broadly safe; -O3 must be measured because it is not always a win; -ffast-math changes numerical results and must be validated. What is the difference between -O2 and -O3, and when is the extra optimisation worth it? -O2 turns on the optimisations that reliably pay off and recovers the bulk of the C-level speedup. -O3 adds more aggressive inlining and loop vectorisation that can help a dense numeric kernel but sometimes produces a slower binary due to instruction-cache pressure. Start at -O2, measure, then keep -O3 only if it improves the specific hot path. When does -march=native help, and what portability cost does it introduce for a deployed binary? -march=native lets GCC emit instructions for the build host’s exact CPU — often the largest lever after -O2 on a vectorisable loop. The cost is that the binary is tied to that CPU family and will crash with illegal instructions on an older deployment machine. Use it only when you control the deployment CPU exactly; otherwise target a conservative baseline that every machine in the fleet supports. What does -ffast-math change, and when is its relaxed numerical behaviour acceptable for inference? -ffast-math relaxes IEEE 754 semantics — reordering, fusing, and re-associating floating-point operations — which unlocks faster math but changes results, because floating-point arithmetic is not associative. For an inference path that drift can flip a classification or diverge from your reference output. It is acceptable only when you have a reference output and a tolerance and have verified the drift stays inside it. How do I verify that a chosen flag set actually improved latency without altering model outputs? Freeze a reference output with the strict build, rebuild with the candidate flags, and compare against that reference with an explicit tolerance — check correctness before latency. Then measure latency on the actual deployment target, not the build host, and confirm the binary runs on every target CPU. Keep the aggressive flags only if the recorded latency delta justifies them. What are sensible default flags for a Cython extension when I don’t yet know the exact deployment CPU? Use -O2 as the baseline, a conservative -march level you know every target supports (such as -march=x86-64-v2 or v3) instead of -march=native, and no -ffast-math unless you have a validated tolerance. This gives most of the C-level speedup with a binary that runs anywhere in the target class, and you can rebuild with a tuned -march once the deployment CPU is pinned. The flag is not the decision — the target is The trap with compiler flags is treating them as a menu of speed switches, when they are really a set of trade-offs against two constraints you already own: the deployment target and the correctness budget. -march=native is not “faster,” it is “faster here and broken elsewhere.” -ffast-math is not “faster,” it is “faster and numerically different.” The engineering is in knowing which cost you can afford. So the question to carry out of a positive Cython decision is not “which flags make it fast” but “which CPUs will this binary land on, and how much numerical drift can this inference path tolerate before someone downstream gets a wrong answer?” Answer those two, and the flag set writes itself — as the failure class of a build tuned to the wrong target, the inference-cost-cut-pack scopes the compile step precisely so the speedup the assessment promised is the speedup you actually deploy.
