“Just optimize the model” is the sentence that hides the whole problem. When a model that scored well on a data-centre GPU has to run inside a phone-class memory ceiling with a tail-latency budget measured in hundreds of milliseconds, “optimization” stops being a single knob you turn until the model is smaller. It becomes a choice among several distinct techniques — distillation, quantisation, pruning, operator fusion, runtime-specific compilation — each of which trades accuracy, latency, and memory in a different direction. Picking the wrong one produces a model that looks great in a benchmark table and still misses the production boundary. The reason this matters at the edge, and rarely in the data centre, is that the constraint changes shape. On a rented A100 you have headroom: you optimize because you want lower cost per token, but a suboptimal choice usually still runs. On a phone, an embedded NPU, or a browser WebGL context, the constraint is a hard wall. The model either fits the memory ceiling and hits the latency budget, or it does not run acceptably at all. Optimization is what stands between “fits the device” and “forces a rewrite at the production boundary.” What does model optimization actually mean in practice? The word “optimization” gets used as if it names one operation. In practice it names a family of transformations applied to a trained model to make it cheaper to run, and they are not interchangeable. Distillation trains a smaller student model to imitate a larger teacher, changing the model’s architecture and parameter count. Quantisation reduces the numerical precision of the weights and activations — for example from 32-bit float to 8-bit integer — changing the representation without changing the shape. Pruning removes weights or whole structures deemed low-contribution, changing the density. Operator fusion and runtime-specific compilation leave the mathematics of the model alone and instead change how the computation is scheduled on the target hardware. Those four verbs touch different parts of the system, so they fail in different ways. A distilled student can drift on the long tail of inputs the teacher handled well. Aggressive quantisation can collapse accuracy on numerically sensitive layers — attention softmax and layer norm are common offenders — even when average-case accuracy looks fine. Unstructured pruning often produces sparsity that a runtime cannot actually exploit, so you lose accuracy without gaining speed. This is the same lesson that shows up across inference work: the reduction that helps depends on which axis your target device is actually constrained on. We treat “which technique” as a measurement question, not a default. If you want the deeper decision between the two most common levers, the distillation-versus-quantisation trade-off for multi-platform edge targets is worth reading alongside this piece; dynamic quantisation in particular behaves differently across runtimes in ways that surprise teams shipping to more than one device class. The techniques, and how they diverge Here is the comparison surface most teams actually need before they commit engineering time. The evidence class for the accuracy and footprint notes is observed-pattern — drawn from edge engagements across mixed hardware targets, not a single published benchmark. Technique What it changes Typical footprint effect Main accuracy risk Runtime dependency Distillation Architecture / parameter count Large — a smaller student can be a fraction of the teacher Tail behaviour drift; loses rare-case competence Low — student is a normal model Quantisation (post-training) Weight/activation precision Roughly 2–4× smaller at INT8 vs FP32 Sensitive layers (attention, norm) can degrade High — kernel support varies by runtime Quantisation (quant-aware training) Precision, learned during training Similar footprint, better accuracy retention Requires retraining budget High Pruning (structured) Density of layers/channels Moderate; exploitable by runtimes Capacity loss if over-pruned Medium — needs structured-sparsity support Operator fusion Computation scheduling Small size change; latency and cold-start gains Low — math is preserved High — compiler/runtime specific Two things fall out of this table. First, footprint reduction and latency reduction are not the same goal — distillation attacks parameter count while fusion attacks scheduling, and a device that is memory-bound needs the former while a device that is latency-bound in orchestration overhead may need the latter. Second, the right column dominates: several of these techniques only pay off if the target runtime can execute the transformed graph efficiently. That is why the runtime is not an afterthought. How do runtime-specific compilers change which optimization is worth applying? A quantised model is only as fast as the kernels available to run it. Post-training INT8 quantisation looks like a free win until you deploy to a runtime with no optimized INT8 path for your operators, at which point the runtime silently falls back to float and you get the accuracy loss with none of the speed. This is the divergence point that catches teams: the optimization passed a desktop benchmark on one stack and then collapsed on the target stack. The three runtimes edge teams meet most often illustrate the spread. Apple’s CoreML compiles to the Neural Engine and is aggressive about fusion and 8-bit / palettized weights, but it constrains which operators it will accelerate — an unsupported op drops the whole subgraph to CPU. ONNX Runtime is the portable middle ground: broad operator coverage across execution providers (CPU, CUDA, and mobile NNAPI/Core ML delegates), but the effective optimization depends entirely on which execution provider actually claims your nodes. WebGL and the browser WebGPU path trade the tightest memory ceiling of all for reach — you are optimizing not just for size but for shader-compatible operators and a cold-start budget that includes downloading and compiling the model in the tab. Cross-platform runtime choice is a topic in its own right, and if your edge target is heterogeneous, OpenCL’s role in cross-platform edge inference covers the portability layer beneath these higher-level runtimes. The point for optimization is narrower: the technique you choose is only real once you have confirmed the target runtime can execute the transformed graph on its accelerated path — otherwise you have optimized on paper. Why does a model that passes a data-centre benchmark still fail at the edge? Because the data-centre benchmark measured the wrong budget. Three gaps recur, and they are worth naming because each has a different fix. The first is the memory ceiling gap. A data-centre benchmark reports peak throughput on a GPU with tens of gigabytes of HBM. It never touches the constraint that matters on a phone: the app has a few hundred megabytes before the OS kills it. An optimization measured as “halved memory” is meaningless unless the absolute residency fits under the device ceiling, and that includes activation memory during the forward pass, not just the stored weights. The second is the tail-latency gap. Data-centre benchmarks love median throughput. Edge products live and die on the tail — the 95th and 99th percentile response, where garbage collection, thermal throttling, and orchestration overhead pile up. We have repeatedly seen a technique that improves median latency do nothing for, or even worsen, the tail once the model runs inside a real application loop rather than a tight benchmark harness. This is an observed-pattern from edge engagements, not a published figure, but it is consistent enough to plan around. The third is the runtime footprint and cold-start gap. The benchmark keeps the model warm. The product cold-starts it — sometimes on every invocation in a serverless or tab context. Cold-start time, driver initialization, and the size of the runtime itself are part of the real budget and never appear in a throughput number. Reducing them is often where operator fusion and a leaner runtime earn their place, even when they do nothing for steady-state speed. The underlying discipline here is the same one we apply to edge-constrained agent inference more broadly: measure against the device budget you will actually ship on, not the one the benchmark was written for. For a grounded reference on what constrained-hardware benchmarking looks like when done honestly, MLPerf Tiny’s approach to benchmarking AI on constrained edge hardware is a useful anchor. A decision rubric: choose the technique by the constraint Before picking a technique, name the binding constraint. The rubric below is deliberately conditional — there is no universal ordering. If the device is memory-bound (weights + activations exceed the ceiling): start with distillation to cut parameter count, then quantise the student. Pruning helps only if the runtime exploits the resulting sparsity. If the device is steady-state-latency-bound but fits in memory: quant-aware training to INT8 plus operator fusion on the target runtime usually gives the best latency-for-accuracy return. If the device is cold-start-bound (serverless, browser tab, intermittent invocation): prioritise runtime footprint and fusion; a smaller runtime and a fused graph cut initialization time more than weight compression does. If accuracy is the binding constraint and footprint has slack: prefer quant-aware training over post-training quantisation, and validate on the long tail, not just the average case. In every case: measure the transformed model on the target runtime’s accelerated path before trusting any headline number. A technique that falls back to CPU or float has not been applied — it has been proposed. The accuracy-versus-footprint trade-off in step 4 deserves its own measurement. Before shipping, evaluate the optimized model against a held-out set that over-represents the rare and hard cases, and compare it to the unoptimized baseline on the same set. If you only look at aggregate accuracy you will not see the tail degradation that quantisation and distillation both tend to introduce — and the tail is exactly where a shipped model earns complaints. How do optimization decisions feed back into framework selection? This is the part that surprises teams building on-device agents. The optimization you can afford constrains the agent framework you can run. If your model has to be distilled to a small student to fit memory, the framework’s orchestration overhead — its planner, its tool-call loop, its memory management — now competes with the model for the same tail-latency budget. A heavyweight agent framework that is invisible on a data-centre GPU can be the dominant cost on a phone once the model itself is small. So model optimization is not a downstream cleanup step performed after the architecture is fixed. It is an input to the architecture. The compression decisions here feed directly into the edge-aware framework assessment: how much latency the model leaves for orchestration, how much memory the runtime and framework can consume, and whether the whole system — model plus runtime plus agent loop — fits under the device wall. Getting the model optimization right early is what keeps the framework choice open; getting it wrong forces the rewrite the ROI depended on avoiding. Optimization on edge devices is GPU-adjacent engineering, and the GPU acceleration work that underpins TechnoLynx inference engagements is where these techniques get measured against real device budgets rather than headline compression ratios. If you want the applied form of the distillation-versus-quantisation decision across multiple platforms, that is where it lives. FAQ How does model optimization work in practice? Model optimization is a family of transformations applied to a trained model to make it cheaper to run — not a single knob. Distillation changes the architecture, quantisation changes numerical precision, pruning changes density, and operator fusion changes how the computation is scheduled. Each touches a different part of the system and trades accuracy, latency, and memory differently. What are the main model optimization techniques and how do they differ? Distillation trains a smaller student model to imitate a larger teacher, cutting parameter count. Quantisation reduces weight and activation precision (e.g. FP32 to INT8), roughly 2–4× smaller at INT8. Pruning removes low-contribution weights or structures. Operator fusion and runtime compilation leave the maths alone and improve scheduling. They diverge because footprint reduction and latency reduction are not the same goal. How do I choose an optimization technique against an edge device’s memory ceiling and tail-latency budget? Name the binding constraint first. If the device is memory-bound, start with distillation then quantise; if steady-state-latency-bound, use quant-aware INT8 plus fusion; if cold-start-bound, prioritise runtime footprint and fusion. In every case, measure the transformed model on the target runtime’s accelerated path before trusting a headline number. Which optimizations trade accuracy for footprint, and how do I measure that trade-off before shipping? Quantisation and distillation both tend to introduce tail degradation even when average accuracy looks fine. Prefer quant-aware training over post-training quantisation when accuracy is binding. Measure by evaluating the optimized model against a held-out set that over-represents rare and hard cases, comparing to the unoptimized baseline on the same set — aggregate accuracy hides tail loss. How does runtime-specific compilation change which optimizations are worth applying? A quantised model is only as fast as the kernels the runtime can execute. CoreML is aggressive about fusion and 8-bit weights but drops unsupported operators to CPU; ONNX Runtime is portable but depends on which execution provider claims your nodes; WebGL/WebGPU add a cold-start compile budget. An optimization that forces a float or CPU fallback delivers the accuracy loss with none of the speed. Why does an optimization that passes a data-centre benchmark still fail at the edge boundary? Because the benchmark measured the wrong budget. Data-centre benchmarks report median throughput on GPUs with tens of gigabytes of HBM, while edge products face a hard memory ceiling, a tail-latency budget, and cold-start costs the benchmark never touches. A “halved memory” result is meaningless unless the absolute residency — weights plus activations — fits under the device ceiling. How do model optimization decisions feed back into agent framework selection for on-device inference? The optimization you can afford constrains the framework you can run. If the model must be distilled small to fit memory, the framework’s orchestration overhead competes with the model for the same tail-latency budget. So optimization is an input to the architecture, not a downstream cleanup step: getting it right early keeps the framework choice open.
