A distilled text-to-speech model passes QA on desktop, ships to iOS and Android, and one target starts producing a faint buzz on sustained vowels. The graph is identical. The weights are identical. What changed is the flag set the runtime was built with — and nobody chose it, they inherited it. That is the failure mode this article is about. When a model has to run inside a real-time latency budget across iOS, Android, and desktop, the compiler flags used to build the inference runtime are not a detail you defer to defaults. They decide whether the same graph produces the same output on every target. Treat the flag set — optimization level, execution-provider selection, fast-math and precision toggles, target architecture — as part of the deployment contract, not as packaging trivia the toolchain settles for you. What does a “flag compiler” actually do in cross-platform inference? There is no single tool called a flag compiler. The phrase describes a real thing, though: the layer where build-time flags feed a compiler that turns your model graph into executable kernels for a specific target. In ONNX Runtime that compiler is the graph-optimization and execution-provider pipeline; in Apple’s stack it is the CoreML compiler that ingests an .mlmodel or .mlpackage and lowers it onto the CPU, GPU, or Neural Engine. In both cases a set of flags controls how aggressively the graph is rewritten, which hardware backend runs each operator, and how loosely the arithmetic may be reordered. The naive assumption is that these flags are performance dials with no correctness consequence — that turning them up makes inference faster and turning them down makes it slower, and the output stays the same either way. That is where it breaks. Some flags are purely mechanical: they change how kernels are scheduled without touching the numbers. Others license the compiler to reassociate floating-point operations, fuse a matmul-plus-activation into one kernel that accumulates differently, or drop to a lower-precision path. Those flags change the bits that come out. On a bounding-box detector a one-ULP difference vanishes into the confidence threshold. On a TTS decoder feeding a vocoder, the same difference can ring through as an audible artifact. The practical consequence: the same distilled model can build to a predictable latency across ONNX Runtime and CoreML if the flag set is chosen deliberately, and can diverge silently across targets if it is not. Which build flags move latency, and which are cosmetic? The honest answer is that most of the latency lives in a handful of flags, and a lot of what teams tweak is noise. Below is how we tend to sort them when configuring a cross-platform build. The evidence class is an observed pattern across deployment engagements, not a published benchmark — treat the direction as reliable and the exact magnitudes as things to measure on your own target. Flag / setting Runtime Latency impact Numeric impact Verdict Graph optimization level (ORT_ENABLE_ALL vs BASIC) ONNX Runtime Large — enables fusion, layout opt Fusion can reorder accumulation Set deliberately, validate output Execution provider selection (CPU / CUDA / CoreML EP / NNAPI) ONNX Runtime Large — different kernel libraries Different EPs use different math paths Pin per target; do not auto-fallback silently CoreML compute units (.all / .cpuAndGPU / .cpuOnly) CoreML Large — Neural Engine vs CPU ANE uses reduced-precision paths Choose per quality budget Fast-math / FP reassociation Both Moderate High — reorders float ops The main drift risk; off by default for TTS FP16 / mixed-precision toggle Both Moderate–large High — precision loss Validate against MOS/jitter boundary Target architecture (-mcpu, arm64 vs x86-64, AVX/NEON) Both (native build) Moderate Low if precision preserved Match the deployment SoC Thread-count / intra-op parallelism Both Situational None Tune late; cosmetic vs the above Debug symbols / assertion flags Both Small runtime, large binary None Cosmetic for latency The two rows that catch teams out are graph optimization level and execution-provider selection. Between a debug build with basic optimization and a fully optimized build with the right execution provider, we routinely see on-device latency gaps in the 20–40% range (observed across engagements; not a benchmarked rate). If that gap is discovered late in QA — because the debug build was what everyone tested against — the schedule is already committed. That is the ROI argument for choosing flags up front: you remove a per-platform validation loop instead of paying for it after the fact. How does a fast-math flag introduce numeric drift that degrades TTS on one target? This is the mechanism worth understanding in detail, because it is the one that produces the “identical graph, different output” bug. IEEE-754 floating-point addition is not associative. (a + b) + c and a + (b + c) can produce different results in the last bits. Standard compilation preserves the order your graph specifies. A fast-math or FP-reassociation flag tells the compiler that order does not matter — it may reorder sums, fold reciprocals, fuse a multiply-add into an FMA that rounds once instead of twice, or contract a whole reduction into a vectorized path with a different accumulation tree. Each of these is individually tiny. In a deep decoder they accumulate. For a classifier or detector, that accumulated drift disappears below the decision threshold; the argmax is unchanged. For a TTS model the output is a continuous waveform-adjacent signal — mel-spectrogram frames, or the input to a neural vocoder. Small systematic errors in that signal are not rounded away by a threshold; they are rendered. Depending on where they land they surface as a faint buzz on sustained phonemes, a metallic edge on fricatives, or jitter that a mean-opinion-score (MOS) evaluation flags as reduced quality. The same distilled model shipped across runtimes can sound clean where fast-math is off and degraded where the packaged runtime enabled it by default. The tensor shape and layout are preserved perfectly; the numeric path is not. The reason it shows up on one target and not another is that fast-math is a per-build, per-backend decision. The CoreML Neural Engine path may quantize to a reduced-precision internal format regardless of your ONNX-side settings. An x86 desktop build with AVX-512 may fuse a reduction differently from an arm64 mobile build with NEON. Nobody wrote code to make the targets diverge — the flag defaults did. A safe default flag set for a latency-constrained distilled TTS model Here is the starting point we reach for when a distilled model must hit a real-time budget on every target and quality is judged by MOS and jitter thresholds. It is a starting point, not a universal answer — the boundary conditions below matter. Optimization level: enable full graph optimization (ORT_ENABLE_ALL in ONNX Runtime) but treat fusion output as something to validate, not assume. Fusion is where correctness and speed collide. Fast-math / FP reassociation: off. This is the single most important default for TTS. The latency it buys is moderate; the quality risk is high. Turn it on only per-operator, per-target, after you have measured the output. Precision: keep the mel-spectrogram decoder in FP32 or a validated FP16 path; do not let a backend silently drop to a lower-precision route. If you are considering aggressive quantization, understand what reduced floating-point precision does to model output before you commit. Execution provider: pin one per target and disable silent fallback. An EP that falls back from CoreML to CPU mid-graph changes both latency and math without telling you. Target architecture: build against the actual deployment SoC’s instruction set (arm64 + NEON for mobile, the correct AVX level for desktop). This is a latency win with low numeric risk when precision is otherwise preserved. Thread count: tune last. It moves latency situationally and has no numeric consequence, so it is safe to defer. The boundary condition: this set optimizes for output equivalence first, latency second. If your model is a detector or classifier where a one-ULP difference is invisible below the threshold, you can afford fast-math and reap the latency — the safe default for TTS is deliberately conservative because the output is continuous and audible. Setting flags consistently from one build pipeline The point of treating flags as a deployment contract is that they live in one place, versioned, applied to every target. The failure pattern is per-platform teams each accepting their toolchain’s defaults, so the iOS build has CoreML’s fast-math path on, the Android build routes through NNAPI with a different precision policy, and the desktop build ran in debug the whole time. Three targets, three flag sets, none written down. A single build definition that declares optimization level, execution provider, precision policy, fast-math state, and target architecture per platform — and fails the build if a platform can’t honor the precision contract — is the mechanism that closes the divergence. This is the compiler-level layer of the same latency-versus-quality decision the distillation strategy already makes at the model level; it belongs in the same inference-optimization discipline, not in each platform’s packaging step. For teams shipping real-time inference across mobile networks, the on-device build contract is part of the broader media and telecom deployment picture, where a per-platform tuning cycle is exactly the kind of hidden cost worth engineering out. The equivalence question — how you verify two differently-built runtimes still produce output within your quality boundary — is where this connects to the GPU side of the same problem: execution-provider flags and target-architecture toggles are the same class of decision whether the target is a phone or a datacenter GPU. FAQ How does a flag compiler work, and what does it mean in practice for cross-platform inference? There is no single “flag compiler” tool; the term describes the build-time layer where flags feed the compiler that lowers your model graph into executable kernels for a target — the graph-optimization and execution-provider pipeline in ONNX Runtime, or Apple’s CoreML compiler. In practice it means the same model can produce different latency and different numeric output on each platform depending on which flags the build used, so the flag set has to be chosen deliberately rather than inherited from defaults. Which ONNX Runtime and CoreML build flags actually affect on-device latency versus which are cosmetic? Graph optimization level and execution-provider selection carry most of the latency; between a basic debug build and a fully optimized build with the right execution provider we routinely see on-device gaps in the 20–40% range (observed across engagements, not a benchmarked rate). Target architecture and precision toggles are also material. Thread count is situational and debug/assertion flags are effectively cosmetic for latency, so they should be tuned last. How can a fast-math or fusion flag introduce numeric drift that degrades TTS audio quality on one target but not another? Floating-point addition is not associative, so a fast-math or fusion flag that lets the compiler reorder or fuse operations changes the last bits of the result. On a classifier that drift vanishes below the decision threshold, but a TTS model’s output is a continuous signal that gets rendered, so the same error surfaces as a buzz, a metallic edge, or jitter that MOS evaluation flags. It appears on one target and not another because fast-math is a per-build, per-backend default — one runtime enabled it and another did not. How do I set optimization level and execution-provider flags consistently across iOS, Android, and desktop from one build pipeline? Declare optimization level, execution provider, precision policy, fast-math state, and target architecture in a single versioned build definition applied to every platform, and fail the build if a platform cannot honor the precision contract. The failure pattern is each platform team accepting its own toolchain defaults, producing three undocumented flag sets; centralizing the contract is what closes the divergence. What is the safe default flag set for a distilled TTS model that must hit a real-time latency budget on every target? Enable full graph optimization but validate fusion output, keep fast-math off, hold the decoder in FP32 or a validated FP16 path, pin one execution provider per target with no silent fallback, build against the actual deployment SoC, and tune thread count last. This set prioritizes output equivalence first and latency second, which is the conservative posture a continuous, audible output demands; a detector or classifier can afford to relax it. How do I verify that two runtimes built with different flags still produce equivalent output within my quality boundary? Compare the runtimes on the same inputs against the specific quality boundary your product measures — for TTS that means MOS and jitter thresholds on the rendered audio, not just a numeric tolerance on intermediate tensors. Because a below-threshold numeric difference can still be audible, verification has to happen at the output the user actually experiences, and any flag that fails that check is the one to disable regardless of its latency benefit. The buzz on the sustained vowel is not a mystery once you know where to look — it is a flag someone never chose. The open question for any cross-platform build is simpler than it seems: do you know, for each target, which flags were licensed to change your numbers, and did you decide that on purpose?
