You distilled the text-to-speech model, exported it to ONNX, and the graph is as small as it will get. Streaming latency is still brushing the top of its jitter budget on a mid-range laptop CPU, and the model has nothing left to give. The instinct is to reach back into the model — another distillation pass, a smaller vocoder, more aggressive quantisation. Often that instinct is wrong. On a CPU inference path, the last few milliseconds frequently live below the model graph entirely, in how the native operator kernels of the runtime were compiled. This is the part of the stack most teams treat as fixed. You download a prebuilt ONNX Runtime wheel, it ships with whatever flags the packagers chose, and you accept those numbers as a property of the hardware. They are not. They are a property of the build. And for a TTS model that has to stream audio inside a tight latency-and-dropout budget, the compiler flags used to build the native runtime are a deliberate engineering decision — one that trades local speed against the thing distillation was supposed to protect: a consistent experience across every device you ship to. What do GNU compiler flags actually control here? GCC — the GNU Compiler Collection — turns the C and C++ source of a runtime like ONNX Runtime into machine code. The flags you pass at build time tell it how hard to try and which instructions it is allowed to emit. Two families matter for CPU inference. The first is the optimisation level: -O0 through -O3, plus -Ofast. These govern how aggressively GCC reorders, inlines, unrolls, and vectorises your code. The second is the target-architecture family: -march, -mtune, and the individual instruction-set switches like -mavx2, -mfma, -mavx512f. These tell the compiler which CPU instructions are on the table. An AVX2 fused-multiply-add loop is dramatically faster per cycle than a scalar one, and TTS inference — dominated by matrix multiplies in the acoustic model and the vocoder — is exactly the kind of workload that benefits. The catch is that these two families interact with a third thing the runtime does at load time: CPU dispatch. ONNX Runtime, oneDNN, and most serious inference libraries ship multiple compiled code paths and pick the fastest one your CPU actually supports at runtime. That mechanism is the hinge on which the whole portability decision turns, and most of the flag confusion in this space comes from not understanding it. Which optimisation level actually moves ONNX Runtime CPU latency? Start with the headline pairing everyone argues about: -O2 versus -O3 versus -Ofast. The honest answer is that on a well-written inference runtime, the gap between -O2 and -O3 is smaller than people expect, because the hot kernels are already hand-tuned or delegated to a vectorised library like oneDNN. -O3 adds more aggressive loop transformation and vectorisation, which can help the surrounding glue code and any custom operators you wrote yourself. In configurations we’ve tested on desktop x86 targets, moving a custom pre/post-processing operator from -O2 to -O3 with vectorisation enabled recovered single-digit to low double-digit percentage latency on the CPU inference path — an observed pattern across our engagements, not a benchmarked universal, and one that shrinks to nearly nothing once the bulk of compute already sits inside a vendor kernel library. -Ofast is where the trap opens. It is -O3 plus -ffast-math plus a handful of standards-relaxing switches. That extra math relaxation is not free, and we come back to it below. Flag What it buys you What it costs -O2 Safe, well-understood baseline; good code without surprises Leaves some vectorisation of glue code on the table -O3 More aggressive loop unrolling and auto-vectorisation Larger binary; occasional pessimisation on cold paths -Ofast -O3 plus relaxed floating-point Breaks strict IEEE semantics — can drift audio numerics -march=native Every instruction the build host supports Binary is bound to that microarchitecture — not portable -mavx2 -mfma Explicit, portable instruction targeting You must pair it with runtime dispatch to stay safe The table is the decision in miniature: everything to the left of the cost column is a promise the compiler makes, and everything to the right is a bill someone pays later, often on a device you never tested. Why -march=native breaks a ship-once binary -march=native is seductive. You type it, the build gets faster, the local benchmark improves. What it actually does is tell GCC to compile for exactly the CPU doing the compilation — probe the build host, enable every instruction extension it has, and tune scheduling for that specific microarchitecture. If your build host has AVX-512 and your deployment target is an older laptop with only AVX2, the binary you shipped contains instructions the target CPU cannot execute. The failure mode is not a graceful slowdown. It is an illegal-instruction fault — a hard crash the moment the hot kernel is first reached, which for a streaming TTS service can mean the process dies mid-utterance. This is the exact inconsistent-experience failure the distillation strategy was chosen to avoid: you compressed the model precisely so it would behave the same everywhere, then undid that guarantee in the build step. The portable answer is to name the instruction-set floor you are willing to require and let runtime dispatch handle everything above it. Compile with -mavx2 -mfma if AVX2 is your baseline, and rely on the runtime’s own multi-versioned kernels to opportunistically use AVX-512 where the CPU reports it. You keep the “ship once, run everywhere” property without giving up the fast path on capable hardware — no separate build-and-validate cycle per CPU target. This is the same portability-versus-performance tension that governs cross-platform ONNX and CoreML compiler flag choices at the graph and runtime layer; the GCC flag question is that decision pushed down into the native code. Can -ffast-math really change TTS audio quality? Yes, and this is the flag that surprises people who came from computer-vision work where a fractional numeric drift washes out in an argmax. -ffast-math lets GCC assume floating-point arithmetic is associative and commutative, ignore the possibility of NaN and infinity, and flush denormals to zero. Real IEEE 754 arithmetic is not associative — (a + b) + c and a + (b + c) can differ in the last bits — so the compiler is now free to reorder reductions in ways that change the output. For a classifier, those last bits vanish. For a vocoder producing a waveform, the reduction order inside the synthesis filters is part of the signal. Reordered accumulation can shift the numerics enough to introduce audible artifacts at the quality boundary, exactly where a distilled model already has the least headroom. The practical stance we take: keep -ffast-math off the audio-generating path by default. If you want its speed, isolate it. Compile the vocoder and any post-filter with strict math, and only consider relaxed math for demonstrably tolerant sections after you have listened to the output — not after you have looked at the latency number. A flag that improves the clock and degrades the ear is not an optimisation; it is a regression you cannot see in a profiler. A worked flag decision Assume a streaming TTS service that must ship as a single binary to a fleet of x86-64 laptops and desktops, oldest of which supports AVX2 but not AVX-512, running the acoustic model and vocoder through ONNX Runtime on CPU. Optimisation level: -O3 for the runtime and custom operators; the marginal gain over -O2 is worth taking on code you control. Architecture floor: -mavx2 -mfma, never -march=native. This sets AVX2 as the guaranteed instruction set and leaves higher extensions to runtime dispatch. Tuning: -mtune=generic (the default) so scheduling is not biased toward one microarchitecture. -mtune only affects instruction ordering, not which instructions are legal, so it is safe to leave generic across a mixed fleet. Math: strict IEEE everywhere on the audio path. No -ffast-math, no -Ofast. Dispatch: confirm the runtime’s CPU-feature detection is compiled in and not disabled by a well-meaning -march override. That configuration is boring on purpose. It gives up the last sliver of host-specific speed to guarantee the binary runs and sounds the same on every target — the property the whole streaming budget depends on. How do you measure a flag change instead of trusting the compiler? A compiler flag is a hypothesis, not a result. The only measurement that counts is end-to-end latency on the actual target device under the actual streaming load — first-token latency and, more importantly, the tail. Streaming TTS lives or dies on jitter and dropout, so a flag that improves mean latency while widening the p99 tail has made your service worse. Benchmark the whole inference call, warm and cold, on the oldest CPU in your fleet, and compare distributions rather than single numbers. Two practical guards. First, measure on-device, not on the build host — the build host is by definition the most capable machine and the least representative. Second, hold the model fixed while you vary flags, so you are attributing the delta to the toolchain and not to a coincidental export change. This is the same discipline that governs how a 3D tensor’s shape and memory layout survive a cross-runtime export — you validate the property on the target, not the assumption on the bench. The CPU-side flag work here is the native-runtime counterpart to the GPU inference tuning we do elsewhere; both operate under the same latency-budget framing, and both belong to the broader question of how an ML compiler turns a model into cross-platform machine code. Compiler-flag tuning sits one layer beneath that: the model has already been chosen and exported, and you are optimising how its operators execute as native code. When we work with telecom and media teams on streaming inference paths, this is often the difference between hitting a jitter budget and rewriting a model that was never the problem — see how it fits the broader media and telecom inference work. FAQ What should you know about gnu compiler flags in practice? GCC flags tell the compiler how aggressively to optimise your code and which CPU instructions it may emit when building a native runtime like ONNX Runtime. In practice, for CPU inference the flags determine how fast the operator kernels run and — critically — which CPUs the resulting binary can safely execute on. They are a build-time decision, not a fixed property of the hardware. Which GCC optimisation flags (-O2 vs -O3, -Ofast) actually affect ONNX Runtime CPU inference latency for TTS? -O3 over -O2 mainly helps glue code and any custom operators you wrote, since the hot kernels are usually already delegated to a vectorised library; the gain we observe is single-digit to low double-digit percent on the paths you control. -Ofast adds -ffast-math, which can move latency but relaxes IEEE semantics and risks audible artifacts on the audio path, so it is not a safe default for TTS. Why does -march=native break cross-platform portability, and what should I use instead for a ship-once binary? -march=native compiles for exactly the build host’s CPU, so a binary built on an AVX-512 machine can hit an illegal-instruction crash on an older AVX2-only target. For a ship-once binary, set an explicit instruction floor like -mavx2 -mfma and rely on the runtime’s CPU dispatch to use higher extensions where available. That preserves the run-everywhere property without a per-target build cycle. Can -ffast-math change TTS audio numerics enough to affect quality at the runtime boundary? Yes. -ffast-math lets the compiler reorder floating-point reductions because it treats arithmetic as associative, and in a vocoder the accumulation order is part of the produced waveform. That reordering can shift numerics enough to introduce audible artifacts, especially in a distilled model with little quality headroom, so keep strict math on the audio-generating path. How do vectorisation and FMA flags (-mavx2, -mfma) interact with runtime CPU dispatch in a portable build? -mavx2 -mfma set the minimum instruction set the binary requires, guaranteeing it runs on any CPU at or above that floor. Runtime dispatch then selects higher code paths — AVX-512, for instance — on CPUs that report support, so you get the fast path on capable hardware without binding the binary to one microarchitecture. The two mechanisms are complementary: flags set the floor, dispatch reaches for the ceiling. How do I measure the real latency impact of a flag change on-device rather than trusting the compiler? Benchmark end-to-end inference on the actual target device under streaming load, comparing latency distributions — including the p99 tail — rather than single mean numbers. Measure on the oldest CPU in your fleet, not the build host, and hold the model fixed so the delta is attributable to the toolchain. A flag that lowers the mean but widens the tail has made a streaming service worse. When should I stop tuning compiler flags and instead revisit the distillation or export strategy? Stop when a correct, portable flag set (-O3, -mavx2 -mfma, strict math, dispatch on) still leaves you outside the jitter and dropout budget on your slowest target. At that point the native layer has given what it can, and the remaining latency lives in the model — which means the honest next move is another distillation pass or a lighter export, not a more aggressive flag that only wins on the build host. If you find yourself reaching for -march=native or -Ofast to close a gap, treat that as a signal rather than a solution: the portable, reproducible flag set is the ceiling of what the toolchain can safely deliver, and everything above it is borrowed against a device you have not tested.
