A flag that halves latency on one runtime is not a free switch you flip once and forget. The same nominal optimisation flag maps to a different graph transform on ONNX Runtime, CoreML, and TensorRT — and some of those transforms change the numbers your model outputs. Treat compilation flags as a universal “make it fast” setting and you will eventually ship a build that runs quickly on one edge target and produces subtly wrong predictions on another, with nothing in your CI logs to tell you why. That is the trap this article is about. Compilation flags — the optimisation switches you pass to a runtime compiler or model converter, such as ONNX Runtime’s graph optimisation levels, CoreML’s compute-unit selection, or TensorRT’s precision and workspace settings — look like a per-target performance knob. They are. But they are a within-target optimisation, which means they do not reduce the work of validating a model across a deployment matrix. They multiply it. What do compilation flags actually do? When you hand a trained model to an edge runtime, the runtime does not execute your graph as-authored. It rewrites it. Constant folding, operator fusion, layout transforms, precision downcasting, and kernel selection all happen inside the compiler or converter before a single inference runs. Compilation flags are how you influence that rewrite. The naive mental model is that a flag is a dial on a shared machine: turn it up, everything gets faster, the output stays the same. That model is wrong in two ways. First, the “machine” is different on every runtime — ONNX Runtime’s graph optimiser, Apple’s CoreML converter, and NVIDIA’s TensorRT builder are entirely separate pieces of software with separate transform catalogues. Second, some of the transforms a flag enables are not numerically neutral. They trade a small, usually acceptable change in output for a speed gain, and “usually acceptable” is doing a lot of quiet work in that sentence. Concretely, three flag families show up in almost every multi-platform edge build: ONNX Runtime graph optimisation levels (ORT_DISABLE_ALL, ORT_ENABLE_BASIC, ORT_ENABLE_EXTENDED, ORT_ENABLE_ALL). Basic covers constant folding and redundant-node elimination — safe, structure-only. Extended and all bring in fusions (Conv+BatchNorm, attention fusion) and layout transforms that can reorder floating-point accumulation. CoreML compute-unit selection (cpuOnly, cpuAndGPU, all including the Neural Engine). This is not just a scheduling hint. Routing an op to the Apple Neural Engine can force a precision or an operator substitution the CPU path would never use. TensorRT precision and workspace flags (FP16, INT8, BF16 enablement, plus the tactic/workspace budget). Enabling FP16 tells the builder it may pick half-precision kernels wherever it judges the accuracy loss tolerable — and the builder decides where, not you. Each of these exists to deliver a real latency gain. None of them is portable in the sense that the same flag produces the same graph, or the same numbers, on a different backend. If you want the full compiler-side picture of how these switches interact with nvcc, Clang, and SYCL builds, our walkthrough of GPU compilation flags for nvcc, Clang, and SYCL builds covers the lower-level toolchain layer this article sits above. Which compilation flags are portable across runtimes? The useful distinction is not “safe vs unsafe” — it is portable vs runtime-specific. A portable flag has a semantic equivalent on every target and does not change numerical output. A runtime-specific flag either has no equivalent elsewhere, or it changes the numbers, or both. Scoping a flag correctly to its category is what keeps per-target validation bounded. Flag portability decision table Flag / setting Portable across runtimes? Changes numerical output? Validation obligation Constant folding (ORT BASIC) Yes — every runtime does it No Validate once; reuse across targets Redundant-node elimination Yes No Validate once Operator fusion (Conv+BN, attention) Partially — fusion catalogues differ Usually no, but accumulation order can shift Re-validate per runtime Layout transforms (NCHW↔NHWC) No — backend-driven Rarely, via accumulation reorder Re-validate per runtime FP16 / BF16 precision (TensorRT, CoreML) No Yes Re-validate per target, per input distribution INT8 quantisation No — calibration is target-specific Yes Full accuracy re-validation CoreML Neural Engine routing No — Apple-only Yes, via op substitution Re-validate on-device Workspace / tactic budget (TensorRT) No Indirectly — changes kernel selection Re-validate if kernels change The pattern in that table is the whole point. Structure-only transforms — the top rows — are portable and cheap: validate the numerical output once and trust it everywhere. Precision and hardware-routing flags — the bottom rows — are runtime-specific and expensive, because they change what your model computes, not just how fast it computes it. This is the same portability boundary that governs where inference work should live at all, which we lay out in our piece on heterogeneous architecture for CPU, GPU, and WASM targets. How can aggressive flags silently change numerical output? The dangerous word in “silently” is that nothing errors. The build succeeds. The model loads. Latency drops. Only the predictions are slightly different — and on an edge deployment matrix, “slightly different on one of three targets” is exactly the kind of divergence that survives a smoke test and fails in production. The mechanism is straightforward once you see it. When TensorRT’s builder is allowed to use FP16, it evaluates candidate kernels and picks half-precision implementations wherever it estimates the accuracy loss is acceptable. That estimate is a heuristic, not a guarantee about your input distribution. A softmax over logits with a wide dynamic range, a normalisation layer near a saturation point, an accumulation loop over a long sequence — these are the places where FP16 rounding compounds into a visible output shift. The builder does not know your data; it knows the operator. CoreML’s Neural Engine routing is a second, distinct failure path. When you request all compute units, the converter is free to run an operator on the Neural Engine, which may substitute a different implementation — sometimes a different activation approximation, sometimes a forced precision — than the CPU or GPU path would use. Your CPU-target validation passes. Your Neural Engine target quietly disagrees on the tail of the output distribution. Detecting this is not glamorous, but it is reliable. In configurations we have worked through, the practice that catches it is a per-target numerical parity check: run the same fixed input set through each compiled target and compare outputs against a reference (typically the unoptimised FP32 graph), not just against each other. Compare on a metric that matches your task — per-class logit deltas for classification, IoU or box-coordinate drift for detection, per-frame audio deltas for TTS — rather than a single top-1 accuracy number, which hides distribution-tail divergence (observed pattern across edge-porting engagements; not a benchmarked threshold). If you are building the metric set for this, our guide to machine learning model metrics for multi-platform edge covers which signals actually distinguish a real regression from noise. How flags interact with a compression choice Compilation flags do not exist in isolation. If you have already made a model-compression decision — distillation versus quantisation — the flags layer on top of it, and the interaction is where validation cycles quietly multiply. Quantisation and aggressive precision flags overlap. If you distilled a smaller FP32 model and then let TensorRT apply FP16 at build time, you now have two independent sources of numerical change stacked on one target — the distillation error and the FP16 rounding — and they interact non-linearly. If instead you post-training-quantised to INT8, the compilation flags that select INT8 kernels are part of your compression decision, not separate from it, and the calibration set you used is target-specific. The flag-portability question and the compression question are not two decisions; on the precision-flag rows of the table above, they are the same decision viewed from two angles. The practical consequence: a compression choice made once at the model level still has to be re-validated per target if it is realised through runtime-specific precision flags. That is the multiplicative cost the parent decision warns about. Distillation, done at training time, produces one model whose numerical behaviour you validate once and carry across targets. Quantisation realised through per-runtime precision flags produces a different numerical model on each backend. The flag portability table is really a map of which compression decisions stay bounded and which ones fan out across your deployment matrix. A per-target flag-testing strategy that avoids the whole-matrix blow-up The failure mode people are trying to avoid is obvious once named: re-testing every flag combination across every target. On a three-target build, if you treat flags as an unbounded space to sweep, you get a combinatorial explosion multiplied by three deployment platforms. The escape is not to test less — it is to test the right thing at each layer. Bounded validation checklist Classify every flag as portable or runtime-specific using the table above before you write any test. Portable flags get validated once; runtime-specific flags get a per-target plan. This step alone removes most of the redundant work. Validate structure-only transforms once, on any target. Constant folding and redundant-node elimination do not change numbers. Confirm that once with a parity check and reuse the result — do not re-run it per platform. Establish an FP32 reference output per model version. This is your ground truth. Every compiled target is compared against it, never against another compiled target (comparing two optimised targets to each other hides the case where both drifted the same way). Run a numerical parity gate per runtime-specific flag, per target. FP16, INT8, and Neural Engine routing each get their own gate. If the parity metric stays inside a task-defined tolerance, the flag is cleared for that target — and only that target. Pin the flag set per target in your build config and version it alongside the model. A flag combination that passed is a validated artifact; treat it like one. When the model version advances, the portable-flag results carry forward and only the runtime-specific gates re-run. Measure latency separately from correctness. The flags exist for speed, so record the per-target latency gain each cleared flag delivers. A runtime-specific flag that passes parity but delivers a negligible speedup is not worth the validation cost it imposes. Scoped this way, a three-target build validates portable flags once and runs a bounded per-target gate only on the handful of flags that actually change numerical output — capturing the latency the flags exist to deliver without paying the 3× re-validation cost that treating every flag as universal would incur. This is a within-target optimisation done deliberately, which is what keeps it from compounding the cross-target burden. If you want the toolchain-level view of what the individual switches change in a ported build, our reference on what -O3, -march, and fast-math actually change in ported inference sits directly underneath this decision layer, and the broader GPU optimisation practice is where the per-target flag audit lives as a repeatable step. Which flags matter most for edge latency? If you are optimising for latency and want the shortest list of high-leverage switches: on ONNX Runtime, ORT_ENABLE_EXTENDED captures most of the safe fusion gains without the layout-transform surprises that ORT_ENABLE_ALL can introduce — a reasonable default before you go further. On TensorRT, FP16 enablement is almost always the single largest latency lever on supported hardware, which is exactly why it demands the most careful parity gate; the workspace budget matters mainly because a larger budget lets the builder consider faster tactics. On CoreML, compute-unit selection is the lever, and the honest answer is that all is fastest and riskiest — the Neural Engine path is where you get the biggest win and the biggest chance of a silent numerical shift. None of these is a recommendation to flip blindly. Each is a place where the latency gain is real and the runtime-specific validation obligation is real, and the table earlier in this article is the tool for keeping the two in balance. FAQ How does compilation flags work in practice? Compilation flags are the optimisation switches you pass to a runtime compiler or model converter — ONNX Runtime graph-optimisation levels, CoreML compute-unit selection, TensorRT precision and workspace settings. They influence how the runtime rewrites your model graph before inference: fusion, constant folding, layout transforms, precision downcasting, and kernel selection. In practice they are a per-target performance knob, not a universal one, because the same nominal flag maps to a different graph transform on each backend. Which compilation flags are portable across runtimes and which are specific to CoreML, ONNX Runtime, or TensorRT? Structure-only transforms — constant folding, redundant-node elimination — are portable and numerically neutral: validate once, reuse everywhere. Precision and hardware-routing flags are runtime-specific: FP16/INT8 in TensorRT, INT8 calibration, and CoreML Neural Engine routing all change numerical output and have no equivalent that behaves identically on another backend. The decision table in this article maps each flag family to its portability and validation obligation. How can aggressive optimisation flags silently change numerical output, and how do I detect it? Aggressive flags such as FP16 or Neural Engine routing let the builder substitute lower-precision kernels or different operator implementations wherever it estimates the accuracy loss is tolerable — but that estimate does not know your input distribution, so rounding can compound at softmax, normalisation, or long accumulation loops. Nothing errors; only the predictions shift. Detect it with a per-target numerical parity check against the unoptimised FP32 reference, using a task-matched metric (per-class logit deltas, IoU drift) rather than a single top-1 number that hides tail divergence. How do compilation flags interact with a distillation-vs-quantisation compression choice — do they add validation cycles? They do, because precision flags and quantisation are the same decision viewed from two angles. Distillation done at training time produces one model whose numerical behaviour you validate once and carry across targets. Quantisation realised through per-runtime precision flags produces a different numerical model on each backend, so it re-validates per target — that is the multiplicative cost the parent decision warns about. What is a sensible per-target flag-testing strategy that avoids re-validating the whole deployment matrix? Classify every flag as portable or runtime-specific first, validate structure-only transforms once, and establish a single FP32 reference output per model version. Then run a bounded numerical parity gate only on the runtime-specific flags (FP16, INT8, Neural Engine routing), per target, pinning and versioning the cleared flag set alongside the model. This captures the per-target latency gains without sweeping every flag combination across every platform. Which ONNX Runtime graph-optimisation levels and TensorRT precision flags matter most for edge latency? On ONNX Runtime, ORT_ENABLE_EXTENDED captures most of the safe fusion gains without the layout-transform surprises that ORT_ENABLE_ALL can introduce. On TensorRT, FP16 enablement is usually the single largest latency lever on supported hardware — which is exactly why it needs the most careful parity gate — with the workspace budget mattering because it lets the builder consider faster tactics. Each is a genuine speed gain paired with a genuine runtime-specific validation obligation. The question worth carrying out of this is not “which flags are fast” but “does my model compute the same thing across every target after the flags are applied” — the cross-platform validation obligation is the constraint, and flags are one of the things that quietly expands it. An A1 GPU/Inference Optimization review looks precisely at whether aggressive optimisation is silently altering numerical output across your deployment matrix, which is the failure class this whole exercise exists to keep bounded.
