Dynamic quantisation looks like a free win: flip a flag at load time, quantise the weights, and expect the model to run smaller and faster. On an edge-constrained agent, that assumption is where the production rewrite begins. DynaQ — the shorthand for the dynamic-quantisation path in most modern runtimes — does reliably shrink a model’s memory footprint, but the way it earns that footprint reduction is exactly the thing that can blow through a tail-latency budget on phone-class hardware.

The core confusion is easy to state. People treat DynaQ as a portable default, a knob you turn once and carry across every deployment target. In practice it is a per-backend measured decision, because the same quantisation profile that wins on one runtime can regress on another. If your agent has to run inside a few hundred milliseconds on-device, that distinction is not academic — it decides whether the feature ships.

What DynaQ actually does at load time and per call

Dynamic quantisation splits the work across two moments. Weights are quantised ahead of time — typically to INT8 — when the model is loaded or exported. Activations, on the other hand, are quantised on the fly, per inference: the runtime observes the actual value ranges flowing through each layer at call time, computes scale factors, casts to a lower-precision integer representation, runs the integer kernel, and dequantises back. Nothing about the activation path is precomputed.

That design is what makes DynaQ attractive and what makes it dangerous. Because weights are stored quantised, the model on disk and in memory is smaller. Because activations are quantised at runtime rather than calibrated in advance, you skip the calibration dataset and the accuracy-sensitive tuning that static quantisation requires. The trade is that every single inference now pays a per-call cost to quantise and dequantise activations — a cost that does not exist in a full-precision FP32 or FP16 run.

On a desktop with headroom, that per-call overhead disappears into the noise. On an edge-constrained agent, it lands directly on the tail-latency budget, and that budget is not measured in seconds. It is measured in the hundreds of milliseconds a user will tolerate before an on-device assistant feels broken. The overhead that was invisible on the benchmark box becomes the thing that decides your p95.

How is dynamic quantisation different from static quantisation and full precision?

The three approaches sit on a spectrum of when precision is decided and what you pay for it. Full precision decides nothing — it runs FP32 or FP16 end to end, carries the largest footprint, and has no quantisation overhead. Static quantisation decides everything ahead of time: it uses a calibration pass over representative data to fix activation scales offline, so at inference time there is no per-call quantisation work, but you owe an accuracy-validation step and a calibration dataset. DynaQ sits in the middle — weights fixed offline, activations decided per call.

Approach	Weights	Activations	Footprint	Per-call overhead	Setup cost
Full precision (FP32/FP16)	Full precision	Full precision	Largest	None	None
DynaQ (dynamic)	Quantised offline (INT8)	Quantised per inference	~2–4x smaller (INT8 weights)	Activation quant/dequant every call	Low — no calibration set
Static quantisation	Quantised offline	Quantised offline via calibration	~2–4x smaller	Minimal	Higher — calibration + accuracy validation

The footprint figures are approximate — roughly 2–4x reduction for INT8 weight quantisation is the range we typically see, and it is enough to bring many models under a phone-class memory ceiling (observed-pattern; the exact factor depends on the share of parameters that are quantisable versus kept in full precision). The reason DynaQ gets reached for first is that its setup cost is low: no calibration data, no offline accuracy sweep. The reason it fails in production is that its per-call cost is the highest of the three, and that cost is where the edge budget lives. For the fuller decision between compression levers, our write-up on model optimization for edge inference, covering distillation, quantisation, and runtime fit sets DynaQ in context alongside the alternatives it competes with.

Why the same DynaQ profile wins on one backend and regresses on another

Here is the part that breaks the “portable default” mental model. DynaQ is not a single algorithm with one performance signature — it is a contract that each runtime implements differently. CoreML on Apple silicon, ONNX Runtime on a mid-range Android SoC, and a browser backend running through WebAssembly or WebGPU each make different choices about which layers get INT8 kernels, how activation scales are computed, whether the quant/dequant boundary fuses into the surrounding operators, and what the fallback path looks like when a layer has no quantised kernel.

Those choices are not cosmetic. When a runtime has a fast fused INT8 path for the operators your model actually uses, DynaQ is close to free and you keep the footprint win with negligible latency cost. When it does not — when the runtime has to insert explicit dequantise-requantise casts around every unsupported layer — the per-call overhead can dominate, and a “quantised” model runs slower than the FP16 version it was supposed to improve on. This is not a bug in DynaQ; it is the consequence of quantisation being a runtime property, not a model property.

We see this pattern regularly when an agent’s inference is validated once on a workstation and then ported. The workstation runtime had a mature quantised kernel library; the target device’s runtime did not, or supported a different operator subset. The profile that passed on the desktop was never a property of the model — it was a property of the desktop’s software stack. The cross-platform kernel story matters here too, which is why understanding OpenCL and cross-platform edge inference is worth doing before you commit to a single backend’s quantisation behaviour.

A worked example: reading the footprint-versus-latency pairing

Assume an agent’s on-device language model is 1.2 GB in FP16, and the phone-class target has a practical per-process memory ceiling of around 700 MB after the OS, the app, and other resident buffers take their share. In FP16 the model does not fit — it is evicted or fails to allocate. The only reason to reach for DynaQ is that INT8 weight quantisation brings it under the ceiling.

• FP16 baseline: ~1.2 GB footprint — over ceiling, does not run. Latency is irrelevant because it never loads.
• DynaQ INT8, Runtime A (fused kernels): footprint drops to roughly 400 MB — under ceiling — and measured p95 stays near the FP16 desktop figure because the activation quant/dequant fuses into the matmul path. This is the outcome you want.
• DynaQ INT8, Runtime B (no fused path): same ~400 MB footprint, under ceiling, but measured p95 rises well past the FP16 figure because every layer pays an unfused quant/dequant tax. The model fits and misses the latency budget — a shipped-but-broken agent.

The point of the example is the pairing, not the numbers (which are illustrative). A footprint number alone tells you nothing about whether the agent ships. What you need is a latency/footprint pairing measured per target runtime, before framework commitment — not a single footprint figure quoted from a desktop export. That is the measurable question DynaQ actually poses: does the per-inference activation overhead keep the agent inside its hundreds-of-milliseconds budget on each backend you intend to ship to?

When DynaQ is the right lever — and when it is not

DynaQ earns its place when the binding constraint is memory and the target runtime has good quantised-kernel coverage for your model’s operators. If the model does not fit in FP16 and the runtime fuses the quant path, DynaQ is close to a free footprint reduction. It is the wrong lever when latency is already tight and the runtime lacks fused INT8 kernels, or when the accuracy hit from unclamped activation ranges matters more than the footprint saved — in which case static quantisation with a calibration pass, or a distilled smaller model, is the better tool. This is the same trade-off space our data-centric approach to edge-constrained agent inference works through from the data side rather than the compression side.

If you are sizing edge hardware before you commit, benchmarks like MLPerf Tiny for constrained edge devices are useful for reasoning about the envelope, but they measure the device, not your quantisation profile on your operator set. The gap between a public benchmark and your agent’s measured p95 is exactly the gap DynaQ hides until you test it on the real backend. TechnoLynx’s broader work on GPU and inference optimisation lives on our GPU engineering practice page and our generative AI practice page; the edge-inference side of that work is where DynaQ, distillation, and runtime fit get decided together.

FAQ

How should you think about DynaQ in practice?

DynaQ quantises a model’s weights ahead of time — usually to INT8 — while quantising activations on the fly during each inference. That gives a smaller memory footprint without a calibration dataset, but it means every call pays a per-inference cost to quantise and dequantise activations. In practice it means you get the footprint reduction cheaply and pay for it on latency.

How is dynamic quantisation different from static quantisation and full precision for on-device inference?

Full precision runs FP32/FP16 end to end with the largest footprint and no quantisation overhead. Static quantisation fixes both weight and activation scales offline via a calibration pass, so it carries no per-call overhead but requires calibration data and accuracy validation. DynaQ sits between them — weights fixed offline, activations decided per call — giving the lowest setup cost and the highest per-inference overhead of the three.

What does DynaQ trade, and how does that hit an edge tail-latency budget?

It trades a smaller memory footprint (roughly 2–4x for INT8 weight quantisation) against a per-inference activation-quantisation cost. On a desktop that cost is negligible, but on phone-class hardware it lands directly on a tail-latency budget measured in hundreds of milliseconds rather than seconds — so the overhead that was invisible on a benchmark box can decide your p95 on-device.

Does DynaQ behave the same across CoreML, ONNX Runtime, and browser runtimes?

No — it needs per-backend measurement. Each runtime chooses which layers get INT8 kernels, how activation scales are computed, and whether the quant/dequant boundary fuses into surrounding operators. A profile that fuses cleanly and stays fast on one backend can fall back to unfused casts and regress on another, because quantisation performance is a property of the runtime, not the model.

When does DynaQ bring a model under a phone-class memory ceiling, and when does it fall short of the latency budget?

DynaQ brings a model under the ceiling when INT8 weight quantisation shrinks the footprint enough to fit the device’s practical per-process memory limit. It falls short when the target runtime lacks fused INT8 kernels: the model fits but pays an unfused quant/dequant tax on every layer, pushing p95 past the agent’s budget — a model that loads but feels broken.

How does a DynaQ decision feed back into agent framework selection?

Because DynaQ’s latency signature depends on the runtime, the measured latency/footprint pairing on each target backend becomes an input to framework choice rather than an afterthought. If a framework’s default runtime lacks fused quantised kernels for your operators, that is a reason to reconsider the framework — not to accept a slower agent. The decision is made before commitment, from per-backend measurements, not after the port fails.

DynaQ is not a flag you set and forget; it is a per-backend latency/footprint contract you have to measure on each runtime you intend to ship to. The question worth carrying forward is which compression lever — dynamic quantisation, static quantisation, or distillation — actually fits your agent’s memory profile and latency budget on your target hardware, and that sits inside the broader distillation-versus-quantisation decision for multi-platform edge targets that governs whether an on-device agent ever meets its tail-latency budget.