Ask most teams what an ML compiler does and you get a one-line answer: you export the model, it makes it faster, you ship. That mental model is convenient, and it is wrong in the way that matters most — it hides the fact that a compiler makes target-specific decisions. The same source checkpoint compiled to ONNX Runtime on Android and to CoreML on iOS produces different operator support, different fusion choices, and different numerical behaviour. Treat the compiler as a black box and you find those differences the hard way: on-device, after the build has shipped. An ML compiler is better understood as a lowering pipeline. It takes a framework-level computation graph — a PyTorch or TensorFlow model, say — and progressively rewrites it into operations a specific target runtime can actually execute. Along the way it fuses kernels, folds constants, picks hardware backends, and drops in fallbacks for anything the target cannot run natively. Understanding what happens at each of those stages is what lets you predict cross-platform differences instead of debugging them. How does an ML compiler work in practice? Start with the artifact you hand it: a graph. Every framework represents a model as a directed graph of operators — convolutions, matrix multiplies, activations, normalisations — connected by tensors. The compiler’s job is to transform that graph, step by step, into something a runtime backend can schedule onto real hardware. Each transformation is a rewrite that preserves the mathematical intent of the model while changing how it is expressed. The word “compiler” is borrowed deliberately. A C compiler lowers source through intermediate representations down to machine instructions, applying optimisation passes at each level. An ML compiler does the same thing with tensor programs. Stacks like Apache TVM, XLA (behind JAX and TensorFlow), torch.compile with its TorchInductor backend, and the graph optimisers inside ONNX Runtime and CoreML all follow this shape. They differ in which intermediate representations they use and which hardware they target, but the pattern — high-level graph, progressively lowered, backend-specific code emitted — is shared. The practical consequence is that “compiled” is not a single state. A model compiled for one runtime is a different artifact from the same model compiled for another, even when the source checkpoint is identical. That is the divergence point this whole article turns on. What are the stages of a model compilation pipeline? The pipeline is easiest to reason about as a sequence of rewrites, each narrowing the graph toward a specific target. The stages below are a composite view — exact names vary by toolchain, but the responsibilities are consistent. Stage What the compiler does Where it can diverge across targets Import / graph capture Parses the framework model into a target-neutral graph IR Unsupported source ops may be rejected or approximated at import Graph-level optimisation Constant folding, dead-node elimination, layout canonicalisation Layout preferences (NCHW vs NHWC) differ by backend Operator fusion Merges adjacent ops (e.g. conv + bias + activation) into one kernel A fusion that fires on one runtime may not exist on another Backend selection Assigns each subgraph to a hardware executor (CPU, GPU, NPU) iOS routes to the Apple Neural Engine; Android to NNAPI or GPU delegates Kernel lowering / codegen Emits the actual executable kernels for the chosen backend Numerical precision and rounding can differ per kernel implementation Fallback insertion Wraps unsupported ops in a runtime that can run them A CPU fallback in a mostly-accelerated graph is a latency cliff Read that table top to bottom and the naive “compile once, correct everywhere” assumption falls apart at every row. The graph you shipped is the same; the decisions each compiler made about it are not. How do the compilers behind ONNX Runtime and CoreML differ? This is where the abstraction gets concrete. Take one distilled model and compile it for two targets. ONNX Runtime, running on Android through execution providers, decides fusion and backend assignment based on the providers available on that device — NNAPI, a GPU delegate, or plain CPU. CoreML, compiling the same graph for iOS, makes its own fusion decisions and prefers to route work onto the Apple Neural Engine when the operator set allows it. Two things follow. First, operator support diverges: an operator that CoreML fuses into a single Neural Engine kernel might, under ONNX Runtime on a given Android device, fall back to a separate CPU op because no accelerated implementation is registered. Second, numerical behaviour diverges: different kernel implementations round differently, and a model that was quantised for one backend can produce subtly different outputs on another. For a text-to-speech or vision model, that shows up as audible or visible quality drift, not a crash — which is exactly why it survives to production undetected. The controllable levers here are the compiler flags and export options you set per runtime. We cover those knobs in detail in our breakdown of what compiler flags for ONNX and CoreML inference actually do; the point for now is that these flags change fusion and backend decisions, so they are part of the artifact, not cosmetic tuning. Tensor layout choices matter here too — the way a runtime expects shapes and memory order affects which kernels are even eligible to fuse, a topic we unpack in how 3D tensor shape and layout affect cross-runtime portability. Why can the same compiled model behave differently on iOS versus Android? Because “the same compiled model” is a category error. You compiled one source, and two compilers produced two different executables from it. The differences accumulate across the pipeline stages above: Backend routing sends the bulk of the graph to different silicon — the Apple Neural Engine on iOS, NNAPI or a GPU delegate on Android — and each has its own supported operator set. Fusion decisions collapse different groups of operators into single kernels, changing both the intermediate precision and the memory-access pattern. Kernel implementations for the same logical operation round and accumulate differently, so a distilled or quantised model drifts numerically between targets. Fallbacks land on different operators, so the latency profile — where the time actually goes — is not the same shape on the two platforms. None of this is a bug. It is the compiler doing its job for each target. The failure is assuming the outputs are identical because the input was. What does operator fallback mean, and how do you spot it? Operator fallback is what happens when the compiler encounters an operation the preferred backend cannot execute. Rather than fail, it inserts a bridge: the tensor is moved to a runtime that can run the op — usually the CPU — and then moved back. Correctness is preserved. Performance is not. A single fallback in an otherwise accelerated graph is a latency cliff. The data transfer between accelerator and CPU, plus the slower CPU kernel, can cost more than the rest of the graph combined. In practice, the tell-tale signs are: on-device latency far worse than your benchmark predicted, a profiler showing large gaps between accelerated regions, and a compiler log listing operators as “unsupported” or “assigned to CPU EP.” Reading those logs is the single highest-leverage debugging habit for cross-platform inference — the compiler already told you what it did; you just have to look. A quick diagnostic checklist for fallback and fusion Run through this before you decide a cross-platform quality or latency problem is a model problem: Did you read the compilation report? ONNX Runtime and CoreML both emit one listing per-operator backend assignment. Start there. Is any operator assigned to CPU on an otherwise accelerated device? That is your latency cliff candidate. Did an expected fusion fire? Compare the operator count before and after compilation; a graph that barely shrank did not fuse. Does the numerical output match your reference within tolerance? Run the same inputs through the source model and each compiled artifact and diff the outputs. Is the precision the same on both targets? A backend that silently ran a quantised kernel where you expected float is a source of drift. If steps 1–3 come back clean and the outputs still differ, the divergence is numerical, not structural — and that points at precision, not fusion. How does compilation relate to distillation and quantisation? Compilation is one of three inference-optimisation strategies that operate at different levels, and confusing them wastes effort. Distillation changes the model — a smaller student network learns to mimic a larger teacher. Quantisation changes the numerical representation — weights and activations move from float32 to int8 or lower. Compilation changes how the existing graph is executed — fusion, backend selection, codegen. They compose: a distilled checkpoint can be quantised, then compiled, and each stage buys something the others cannot. The payoff of understanding all three together is a single export path. In one text-to-speech engagement we worked through, a single distilled checkpoint compiled to both ONNX and CoreML met the same latency budget without maintaining divergent export pipelines (observed in that engagement; not a published benchmark). The measurable difference is one export path validated once, versus a separate compile-and-QA cycle for every runtime. That is the ROI of predicting compiler behaviour rather than debugging it after the fact. For teams weighing where numerical precision sits in this stack, our explainer on when 4-bit floating-point precision is the right trade-off covers the quantisation side of the same decision. The kernel-fusion and backend-selection concepts here are not unique to mobile targets — the same mechanisms drive GPU-side inference optimisation, where fusion decisions determine how much of a transformer actually stays resident on the accelerator. If you are shipping client-side text-to-speech or vision inference across mobile networks, the media and telecom deployment context is where these portability constraints bite hardest, because the device fleet is heterogeneous by definition. How do you inspect what a compiler actually produced? The tools for this are less exotic than the problem sounds. ONNX Runtime exposes the graph optimisation level and can dump the optimised model, showing you exactly which fusions fired and which execution provider owns each node. CoreML’s compilation produces a model you can inspect with Apple’s tooling to see Neural Engine versus CPU assignment. For deeper analysis, Netron renders any of these graphs visually so you can compare pre- and post-compilation topology side by side. The discipline is simple to state and easy to skip: compile for every target you ship to, dump the compiled graph for each, and diff them against each other and against your reference outputs before you cut a release. That is what converts cross-platform inference from a source of on-device surprises into something you can predict at build time. We go deeper on the face-recognition case in how ML compilers optimise face-recognition inference, where the same inspection habits apply. FAQ How does an ML compiler actually work? An ML compiler is a lowering pipeline that takes a framework computation graph and progressively rewrites it into operations a target runtime can execute — fusing kernels, folding constants, selecting hardware backends, and inserting fallbacks. In practice it means “compiled” is not one state: the same source checkpoint becomes a different executable for each target, so you cannot assume identical behaviour across runtimes. What are the stages of a model compilation pipeline, from framework graph to runtime executable? The stages run from graph import and capture, through graph-level optimisation (constant folding, layout canonicalisation), operator fusion, backend selection (which silicon runs each subgraph), kernel lowering and code generation, and finally fallback insertion for unsupported ops. Each stage can make target-specific decisions, which is why the same graph diverges across runtimes. How do ML compilers behind ONNX Runtime and CoreML differ in operator support and kernel fusion? ONNX Runtime chooses fusion and backend assignment based on the execution providers available on the device (NNAPI, GPU delegate, or CPU), while CoreML makes its own fusion decisions and prefers the Apple Neural Engine when the operator set allows. The result is that an operator fused into one accelerated kernel on iOS may fall back to a separate CPU op under ONNX Runtime on Android, and kernel implementations round differently. Why can the same compiled model behave differently on iOS versus Android at the numerical or latency level? Because you compiled one source into two different executables. Backend routing sends work to different silicon, fusion collapses different operator groups, kernel implementations round differently, and fallbacks land on different operators — so both numerical output and latency profile differ even though the input model was identical. What does operator fallback mean, and how do I tell when a compiler failed to fuse or offload an operation? Operator fallback is when the compiler moves an unsupported operation to a runtime that can execute it — usually the CPU — preserving correctness but costing latency. You spot it by reading the compilation report for per-operator backend assignment, watching for on-device latency worse than benchmarked, and comparing operator counts before and after compilation to confirm expected fusions fired. How does compilation relate to distillation and quantisation as inference optimisation strategies? They operate at different levels and compose: distillation shrinks the model, quantisation changes the numerical representation, and compilation changes how the existing graph is executed through fusion and backend selection. A distilled checkpoint can be quantised then compiled, and understanding all three lets a team validate one export path once rather than running separate compile-and-QA cycles per runtime. How do I inspect what a compiler actually produced so I can predict cross-platform behaviour before deploying? Dump the optimised graph from each toolchain — ONNX Runtime can export its optimised model with execution-provider assignments, CoreML exposes Neural Engine versus CPU routing, and Netron renders any graph visually for side-by-side comparison. Compile for every target, diff the compiled graphs and reference outputs before release, and cross-platform differences become predictable at build time instead of surprises on-device. The gap this leaves for the next evaluation The reframe worth keeping is small and load-bearing: a compiler is not a switch that makes a model fast, it is a decision-maker that produces a different artifact for every target you point it at. Once you accept that, the work shifts from debugging on-device surprises to reading what the compiler already told you it did. The single-checkpoint export path — one model validated once across CoreML and ONNX — is not a lucky outcome; it is what happens when you can predict fusion and fallback before you ship, which is the mechanism behind the inference optimisation engineering that keeps cross-platform quality under control.
