Machine Learning Compilers: What They Do in the Inference Serving Path

When a model runs slower than the hardware should allow, the reflex is to swap the model or add GPUs. Both are expensive ways to skip a cheaper question: is the compiled execution path leaving latency on the table? A machine learning compiler is the layer that lowers a trained graph — the sequence of operations your model performs — into optimised, target-specific kernels. When it works, it recovers latency the framework’s default eager execution never claimed. When it is adopted on faith, it optimises a layer that may never have been the constraint.

That distinction is the whole point of this article. “Machine learning compiler” sounds like a switch you flip, and vendors are happy to let it read that way. In practice it is a tuning lever — one among several — and like every lever it only earns its place after you have measured what the bottleneck actually is. Choosing a compiler before profiling the serving path is the same mistake as adding GPUs before profiling: you spend engineering time on a layer you have not shown to be slow.

What does a machine learning compiler actually do?

Start with what runs by default. In eager mode, a framework like PyTorch executes your model op by op — each matrix multiply, each activation, each normalisation dispatched to a kernel as the Python interpreter reaches it. That is flexible and easy to debug, but it means the runtime never sees the whole graph at once. It cannot reason about what comes next, so it cannot fuse work, pick better memory layouts, or specialise kernels to your exact shapes.

A compiler changes the unit of work. It captures the graph, applies a series of transformation passes, and emits code specialised to a target device. The names differ — torch.compile with its Inductor backend, XLA behind JAX and TensorFlow, TensorRT for NVIDIA inference, Apache TVM as a standalone stack — but the mechanics rhyme. Three optimisations do most of the work, and it is worth understanding each on its own terms because they fail and succeed for different reasons.

Operator fusion is the one people reach for first. Instead of launching a separate kernel for a matmul, then another for the bias add, then another for a GeLU activation, the compiler fuses them into a single kernel. The win is not arithmetic — the same floating-point work happens — it is memory traffic. Each unfused kernel reads its inputs from GPU memory and writes its outputs back; fusion keeps intermediate values in registers or shared memory and skips the round trip. On workloads where the model is memory-bandwidth-bound rather than compute-bound, this is where most of the latency lives. If you have not established which side of that line your model sits on, our note on when memory bandwidth is the actual bottleneck in AI inference is the prerequisite reading.

Layout selection is quieter but often decisive. Tensors can be stored in different memory orderings — NCHW versus NHWC for convolutions, row-major versus tiled layouts for the tensor cores that GEMM kernels want to feed. A compiler that sees the whole graph can choose a layout that keeps the fast path fed and insert the minimum number of transpose operations, rather than paying for a layout mismatch at every layer boundary. Get this wrong and you can spend more time reshuffling data than computing on it.

Kernel autotuning is the part that costs you wall-clock time up front. For a given operation and a given input shape, there are many valid kernel implementations — different tile sizes, different degrees of loop unrolling, different ways to split work across thread blocks. Autotuning benchmarks candidates on your target hardware and picks the fastest for your shapes. This is why the first compiled run is slow: it is measuring. It is also why compiled models are shape-sensitive — a kernel tuned for a batch of 8 and a sequence length of 512 may be a poor fit when production traffic sends you a batch of 1 and 2,000 tokens.

Compiler or runtime — and when do you need both?

This is the distinction that trips up most teams, and the confusion is understandable because the categories overlap in practice. An inference runtime — think an SGLang or vLLM server, or ONNX Runtime — manages the serving lifecycle: batching requests, scheduling them onto devices, managing the KV cache, handling concurrency. A compiler transforms the model graph into faster kernels. They operate at different layers, and modern runtimes frequently embed a compiler: TensorRT-LLM compiles the model and serves it; vLLM leans on custom fused kernels and increasingly on torch.compile under the hood.

The clean way to think about it: the runtime decides when and how requests hit the model; the compiler decides how fast the model itself runs per request. A well-batched runtime with an unoptimised graph leaves per-request latency on the table. A beautifully compiled graph served one request at a time leaves throughput on the table. Most production serving paths need both to hit a cost-per-request target — which is exactly why we treat neither as a first move but as ranked levers against a profiled bottleneck.

Where does the compiler sit among the other levers?

Batching, caching, quantisation, and compilation all reduce cost-per-request, but they attack different constraints and they compound differently. Adopting them in the wrong order wastes effort — quantising a model that is bottlenecked on request scheduling changes nothing the user can feel.

Lever	What it attacks	Typical latency win	Effort / risk	When it ranks first
Dynamic batching	Under-utilised GPU at low concurrency	High for throughput; can raise p95 tail	Low; a runtime config	Traffic is bursty and GPU utilisation is low
KV / response caching	Repeated or overlapping requests	High when hit rate is high	Low–medium	Workload has cache-friendly structure
Quantisation (INT8 / FP8)	Compute and memory bandwidth	Moderate–high	Medium; accuracy validation required	Model is compute- or bandwidth-bound and accuracy budget allows
Compilation (fusion + autotune)	Kernel-level memory traffic and dispatch overhead	Moderate; workload-dependent	Medium; shape stability and build time	Eager overhead or unfused memory traffic dominates the profile

The evidence class here matters: these rankings are an observed pattern across the inference serving paths we profile, not a benchmarked rate you can port to your model unchanged. The point of the table is not the numbers — it is the ordering discipline. Profile first, rank by latency-per-effort, then apply the lever the profile points at. To see how those levers hang off the actual request flow, our walkthrough on mapping the serving path for performance lays out where each one attaches.

How do you know whether compiling will help before you invest in it?

You do not decide by reputation; you decide by profile. The honest pre-check is short and it front-loads the cheap questions.

Is the model actually the slow part? If a request spends most of its time in tokenisation, network hops, or waiting in a scheduler queue, no amount of kernel fusion helps. Profile end to end before touching the graph.
Is per-op dispatch overhead visible? In a profiler like Nsight Systems or the PyTorch profiler, many tiny kernels with gaps between them signal launch overhead that fusion can collapse. A few large, back-to-back GEMM kernels signal you are already compute-bound and fusion has little to grab.
Are your input shapes stable? Compilers reward shape stability. If production traffic sends wildly varying batch sizes and sequence lengths, you will either recompile constantly or accept kernels tuned for the wrong shape. Dynamic-shape support exists but narrows the win.
Can you tolerate the build cost? Autotuning and ahead-of-time compilation add minutes to hours to your build and deploy loop. For a stable production model that is amortised instantly; for a model you retrain daily it is a real tax.

If three of those four point the right way, a compiler pass is worth a measured trial. If they do not, the profile is telling you the constraint is somewhere else — and the same profile usually names where.

Measuring whether the compiler actually helped

A compiler pass is a change to the serving path, and it earns its keep on the same metrics every other tuning lever is judged against: p95 latency before and after, throughput at a fixed cost, and GPU utilisation from better kernel occupancy. The failure mode we see most is a team that reports a win from a synthetic single-request benchmark, ships it, and finds production p95 unchanged because the compiled kernels were tuned for a shape production never sends.

Measure under production-representative load, not a microbenchmark. Watch the tail, not the mean — fusion that improves median latency but destabilises p99 under batching has moved your problem, not solved it. And confirm the kernel-level story at the source: a GPU profiler shows whether occupancy and memory traffic actually improved, which is the difference between a real gain and a lucky measurement. Our companion piece on reading processor throughput numbers in AI inference covers why a throughput headline and a real latency win are not the same thing.

There is also a numerics dimension worth naming. Compiling for a specific target — especially with reduced precision or fused activation-plus-normalisation paths — can shift outputs slightly. Usually it is negligible; occasionally a fused path changes the order of floating-point operations enough to move a decision boundary. If your model’s output feeds a threshold, validate accuracy on a held-out set after compilation, the same way you would after changing the numerics of a serving-path port. This is the same discipline that governs classic compiler optimisation flags, where -O2 versus -O3 and -march trade speed against reproducibility; an ML compiler is the same bargain at a higher level of abstraction.

FAQ

How does a machine learning compiler work?

It captures your trained model’s computation graph and runs transformation passes that lower it into optimised, target-specific kernels — rather than dispatching operations one at a time as eager execution does. In practice it means the runtime can see the whole graph and fuse operations, choose better memory layouts, and autotune kernels for your hardware and shapes, recovering latency the default execution path left unclaimed.

What does an ML compiler actually optimise — operator fusion, memory layout, kernel autotuning — and how does each affect inference latency?

Operator fusion merges adjacent operations into one kernel to cut memory round-trips, which is the biggest win on memory-bandwidth-bound models. Layout selection chooses tensor orderings that keep tensor-core GEMM kernels fed and minimise transposes. Kernel autotuning benchmarks candidate implementations for your exact shapes and picks the fastest — the reason the first compiled run is slow and why compiled models are shape-sensitive.

How does an ML compiler differ from choosing an optimised inference runtime, and when do I need both?

A runtime (SGLang, vLLM, ONNX Runtime) manages the serving lifecycle — batching, scheduling, KV cache, concurrency — deciding when and how requests hit the model. A compiler transforms the graph so the model runs faster per request. Most production serving paths need both to hit a cost-per-request target, and modern runtimes often embed a compiler internally.

How do I know whether compiling my model will help before I invest engineering time in it?

Profile end to end first: confirm the model is the slow part, look for many tiny kernels with gaps (dispatch overhead fusion can collapse), check that input shapes are stable, and confirm you can tolerate the build-time cost. If most of those signals point the right way, a measured compiler trial is worth it; if they do not, the profile is pointing you at a different constraint.

How do I measure whether a compiler pass actually improved p95 latency or GPU utilisation in production?

Measure under production-representative load — not a single-request microbenchmark — and watch the tail (p95/p99), not just the mean. Confirm the kernel-level story with a GPU profiler to see whether occupancy and memory traffic actually improved, which separates a real gain from a lucky measurement on a shape production never sends.

What accuracy or compatibility trade-offs should I expect when compiling a model for a specific target?

Compiling for a target — especially with reduced precision or fused activation-plus-normalisation paths — can change the order of floating-point operations and shift outputs slightly. Usually negligible, but if your model’s output feeds a threshold or decision boundary, validate accuracy on a held-out set after compilation before trusting the compiled path in production.

Where does the compiler sit relative to other tuning levers like batching, caching, and quantisation?

It is one ranked lever, not a first move. Batching attacks under-utilised GPUs, caching attacks repeated requests, quantisation attacks compute and bandwidth, and compilation attacks kernel-level memory traffic and dispatch overhead. Profile the serving path, rank by latency-per-effort, and apply the lever your profile points at — compilation earns its place only when eager overhead or unfused memory traffic dominates the measurement.

What the next review should check

The compiler question is really the same question that governs every other serving-path decision: what does the profile say is slow, and which lever returns the most latency per unit of engineering effort? A machine learning compiler is a strong lever when kernel-level memory traffic or dispatch overhead dominates, a wasted one when the bottleneck is scheduling, caching, or the network. The discipline that turns it from a black box into a knob is measuring the compiled path against the same baseline as batching, caching, and quantisation.

That ranking is exactly what a serving-path profile produces. If you want the levers ranked against your own p95 and cost-per-request rather than a generic table, the [inference cost-cut audit](Inference Cost-Cut Pack) profiles the serving path and tells you where a compiler pass sits among the alternatives — and our broader engineering services pick up from there. The failure class to guard against is optimising a layer you never proved was the constraint; the audit exists to make sure the compiler is the answer to a question your profile actually asked.