Match intermediate features and the fused embedding rather than logits alone, build the student from layers the accelerator’s toolchain will schedule (depthwise-separable convolutions, ReLU, 8-bit tensors), balance the audio, video and fusion loss terms so neither modality is starved, and quantise after distillation. The final check runs on the device, not the training box, and asks whether the student still uses both modalities.
TechnoLynx handles this work through its AI Porting & Deployment Pack: it gets the workload running on your edge board or embedded target, from a feasibility study to a reproducible runbook.
The reason this deserves its own engineering effort is simple. A fusion model that scores well on a training GPU is a different artefact from one that runs on an appliance accelerator, and closing that gap is engineering work, not a hand-off. Each of the four choices interacts with the others: the student you can build limits where you can attach feature losses, the loss weights decide whether a narrow student keeps both senses, and quantisation can undo the balance you trained for.
Should knowledge distillation match the logits, the fused embedding or the branch features?
Put the main distillation signal on intermediate branch features and the fused embedding, and keep the soft-target logit loss as a secondary term. Studies of cross-modal transfer report problems with logit-only distillation when knowledge crosses from one modality to another.
The C2KD paper (CVPR 2024) found that the modality gap, meaning modality imbalance and misaligned soft labels, makes traditional knowledge distillation ineffective when knowledge crosses modalities. The frequency-decoupled distillation work (2025) argues that intermediate features carry richer modal detail and semantic information than logits, and so play the more important role in cross-modal transfer. In the UMT study (ICML 2023), naive self-distillation on soft labels or on features brought only limited improvement, while distilling pre-trained uni-modal features into the matching branches improved overall performance. UMT applies that distillation before fusion, which is why it suits late-fusion architectures.
Several studies combine both kinds of term. G2D (ICCV 2025) adds an L2 feature-alignment loss specifically to stop the student discarding information from the weaker modality, alongside a KL logit loss that carries inter-class relationships. A 2025 audio-visual sound event localisation study found that adding hidden-layer feature distillation on top of response distillation improved the student further. The authors of the SAS on-device speech detector (2026) propose the same idea for their next step: distil the teacher’s intermediate representations, not only its final probabilities.
One caution on how far this evidence reaches. C2KD, UMT and G2D study cross-modal transfer or balanced multimodal training rather than compressing a fusion model for an NPU.
Two practical details decide whether feature matching helps or hurts. Audio and visual features often sit on different numerical ranges, and the frequency-decoupled work standardises them (mean subtraction, then L2 normalisation) before computing the matching loss. A student branch is also narrower than its teacher, so each matched layer needs a small learned projection, such as a 1×1 convolution or a linear layer, into the teacher’s width. Where feature maps differ in size, the localisation study’s HCL loss used spatial pyramid pooling to produce fixed-length representations for comparison.
Choosing the distillation target
| Distillation target | What it transfers | Where it fits | Main risk |
|---|---|---|---|
| Logits only (soft targets) | Inter-class “dark knowledge” from the teacher’s probability spread | Small classification students with a simple task | The modality gap can make it ineffective across modalities |
| Per-branch intermediate features | Uni-modal representations each branch should learn | Late-fusion students where one branch tends to lag | Feature scale and distribution mismatch between modalities |
| Fused embedding | The joint audio-visual representation after fusion | Cases where cross-modal alignment must survive heavy compression | Needs a projection when student and teacher widths differ |
| Features plus embedding plus logits | All of the above | The combination G2D and the localisation study used | More loss weights to tune |
Which student architecture will the appliance NPU or DSP run?
A MobileNet-style student built from depthwise-separable convolutions, ReLU or ReLU6 activations, no squeeze-and-excitation blocks, and channel counts picked for the specific accelerator, is the starting point the MLSys 2021 profiling favours.
An MLSys 2021 study that profiled seven edge AI platforms, including Edge TPU, a Rockchip NPU and the Kendryte KPU, showed that a network with fewer operations and memory accesses can still run slower when it mismatches the platform. Adding a squeeze-and-excitation block to a MobileNetV2 block barely changed CPU time but cost 15× more time on the KPU and 51× more on the Edge TPU. Only ReLU and ReLU6 had negligible latency impact on every platform, while Swish raised latency everywhere except the KPU because inference stacks could not fuse it. On every platform except the CPU, the MobileNetV1 block was the fastest building block in every layer.
Depthwise convolution is not uniformly cheap either. On the Rockchip NPU, depthwise latency grew quadratically with channel count, which the authors attribute to the NPU running depthwise layers as ordinary convolutions. The same study recommends larger kernels rather than the smallest one for depthwise convolutions on NPU, DSP and VPU targets. Channel counts interact with hardware padding: the Hexagon NN library pads tensors to multiples of 32 channels, producing step-shaped latency curves, so the study recommends keeping the largest channel count in each step.
The vendor toolchain sets hard rules on top of these tendencies. Arm’s Ethos-U Vela compiler (NXP i.MX fork) schedules a depthwise convolution on the NPU only if its weights are 8-bit, its stride is within the supported range and its feature maps are not 32-bit. Most Ethos-U85 operators also need constant tensor shapes and quantisation parameters on every input, output and weight tensor. On NXP’s i.MX ONNX Runtime stack, the graph partitioner hands portions of the model to registered execution providers by capability, with the CPU provider on Cortex-A cores as the default and Neutron and VSI NPU providers for acceleration; NXP notes that Neutron support in ONNX Runtime exists only on the i.MX 95.
The fusion block itself needs the same scrutiny. In the G2D comparison, late fusion gave the best results among traditional fusion methods, with cross-attention close behind.
A 2026 image-denoising study shows this design for the chip: its student was built for fallback-free execution on MediaTek APU and Qualcomm Hexagon NPUs, exported to TensorFlow Lite with the NCHW-to-NHWC layout change fused into the graph, and checked for numerical parity against the PyTorch reference.
Worked example: a depthwise-separable multimodal student at 6 KB
TinyM2Net-V3 (2024) is a published case of an image-plus-audio model compressed with knowledge distillation and 8-bit quantisation for resource-constrained edge devices. The authors based the teacher on TinyM2Net-V2 and designed the student from scratch, aiming for a model of a few kilobytes that fits the L1 and L2 caches of the target hardware. They varied filter sizes and dense-layer widths and replaced the second and third CNN layers with depthwise-separable versions; the chosen student reached 93% accuracy at 6 KB, 26× smaller than its teacher. In a second case study, a 58 KB student reached about 90.9% accuracy, a 296× size reduction for a 1.4% accuracy drop.
The first case used 52.4 KB of L1 and 40 KB of L2 (10% of available L2), the second 47.5 KB of L1 and 178 KB of L2 (44%), and neither needed off-chip DRAM for weights or activations. The models were quantised with TensorFlow Lite full-integer post-training quantisation, because both targets (the GAP8 processor and an Arm Cortex-A72) support int8 arithmetic.
There is one honest wrinkle. TinyM2Net-V3 distilled through temperature-scaled soft targets rather than feature matching. Our reading is that logit-only transfer can be enough for a small classification task with a simple label space, and that the case for feature and embedding matching grows as the student must preserve cross-modal alignment the task depends on. The architecture lesson transfers regardless: depthwise-separable layers compressed the student without significant accuracy loss.
How do we weight audio and video losses so neither branch starves?
Start with equal weight on each branch’s terms, scale every temperature-softened KL term by T², build training batches balanced across modalities, and then adjust the weights on a validation set by watching each branch’s standalone accuracy.
The T² factor comes from Hinton, Vinyals and Dean (2015): gradients from soft targets scale as 1/T², so multiplying them by T² keeps the hard and soft targets contributing in roughly the same proportion when the temperature changes. Without that prefactor, the distillation signal would become negligible at high temperatures.
AVNet (2026) supervises its fusion branch with cross-entropy plus a KL term from a frozen audio teacher on audio-only batches, from a frozen video teacher on video-only batches, and from both teachers plus their averaged ensemble on audio-visual batches, with a distillation weight of 1.0 throughout. Training batches are assigned 25% audio-only, 25% video-only and 50% audio-visual, so the student sees each operating mode. Modality dropout with learned null embeddings forces the fusion module to produce useful output when one stream is missing.
Published loss weights, side by side
| Study | Loss terms | Weights used |
|---|---|---|
| Hinton, Vinyals and Dean (2015) | Hard-target cross-entropy plus soft-target loss | Temperatures 1, 2, 5 and 10 tried; 0.5 relative weight on hard targets |
| AVNet (2026) | Cross-entropy per branch plus KL from audio, video and ensemble teachers | T = 4, λ = 1.0 |
| Hybrid cross-attention AVE model (2026) | Cross-entropy plus KL on softened outputs | T = 4, α = 0.8 |
| UMT (ICML 2023) | Task cross-entropy plus MSE feature distillation for audio and RGB | 1 and 50, same weight for both modalities |
| Audio-visual SELD (2025) | Response distillation plus feature distillation | 0.5 each, 20-epoch warm-up for the feature term |
| G2D (ICCV 2025) | Student loss plus feature loss plus logit loss | α = 1, β = 1 gave the best overall result |
The weights in this table range from 0.5 to 50. Those weights multiply losses of very different magnitude (a KL divergence on softened probabilities versus an MSE on feature maps), so a weight only means something relative to the scale of its own loss term.
Two adaptive methods re-weight or suppress modalities during training to counter imbalance. Gradient-Blending (2019) re-weights per-modality losses according to each modality’s overfitting and generalisation behaviour. Its offline version computes the weights once and keeps them fixed, while the online version recomputes them every few epochs; offline was slightly less accurate but much simpler to implement. The weight estimates stayed stable when computed on subsets of the training data, which keeps the cost down. G2D goes further with sequential modality prioritisation, which zeroes out the dominant modality’s gradients so the weaker one can train to convergence, and this complete suppression beat partial gradient modulation on every dataset tested.
We would build the validation set so that some cases are hard for audio and others hard for video, because a set dominated by one kind of difficulty will reward a student that leans on the other modality. If one branch’s standalone accuracy falls much faster than the other’s as training proceeds, that is the signal to rebalance.
Do we quantise before or after distilling?
Distil in floating point first, then quantise the trained student; move to quantisation-aware fine-tuning, with the teacher still supplying soft targets, only if post-training quantisation costs too much accuracy.
The distil-then-quantise order has been tested in audio-visual work. A 2026 study of a hybrid cross-attention event-recognition model cut the fusion network’s trainable parameters from 6.86 million to 2.81 million (59.06%) through distillation, with accuracy falling from 84.19% to 82.05%. Dynamic INT8 post-training quantisation then reduced the model from 10.71 MB to 2.04 MB at 81.20% accuracy, 2.99 points below the teacher. The SAS on-device detector (2026) reports that INT8 post-training quantisation cut active weight size by 4× with under 0.5 F1 points of degradation.
The 2026 cross-attention result has a limit for NPU targets. Its dynamic quantisation converted only the linear layers to INT8, quantising activations at run time and leaving the remaining operations in floating point.
INT8 affects speed and accuracy differently. INT8 gave more than 11× speedup over FP16 on the Rockchip NPU in the MLSys 2021 profiling study. The same study found that INT8 can sharply reduce the accuracy of some models on some platforms, and recommends removing the building blocks that cause the loss from the design space when INT8 is the target. Where the hardware allows mixed precision, the survey keeps sensitive layers (layer normalisation, first and last layers) in FP16 and dense linear layers in INT8.
How do we check on the device that the student still uses audio and video?
Run the deployed, quantised student on the target hardware three times over the same test set (both streams, audio blanked, video blanked) and compare each branch’s confidence against its unimodal teacher. A fused student that barely changes when one stream is removed has collapsed onto the other.
On-device modality and deployment checklist
- Confirm numerical parity between the PyTorch student and the exported model on a host CPU before touching the device, as the 2026 denoising study did.
- Read the execution provider assignment for every node, since ONNX Runtime partitions the graph between CPU and NPU providers by capability.
- Ablate each stream on the device; if the student was trained with modality dropout and null embeddings, substitute the learned null embedding rather than zeros, which is how AVNet handles a missing modality at inference.
- As one example of a fusion gain on a different task, SAS (2026) reported F1 of 0.86 audio-only and 0.95 with audio and video on its held-out test set.
- Compute a confidence ratio per modality, as defined in G2D: the weaker modality’s average confidence divided by its unimodal teacher’s, where a low ratio means the modality is being overshadowed.
- Repeat the ablation on the INT8 build, not only the float build, because quantisation error need not fall evenly on both branches.
- Measure latency and memory end to end on the target SoC, since a reduced operation count does not guarantee lower real latency.
Steps 3 to 5 have a training-time counterpart in OGM-GE (CVPR 2022), which monitors the discrepancy between each modality’s contribution to the learning objective and modulates the gradients accordingly. Running the same comparison after deployment closes the loop. Profiling end-to-end latency on embedded hardware before a procurement commitment is a wider question than this page covers; here the latency check exists to confirm that the student design held up on silicon.
What the published work does not yet show
SAS describes its NPU-optimised variant as under development, with distillation proposed rather than performed. AVNet lists a real-time pipeline for embedded automotive hardware as future work.
The 2026 cross-attention study compresses only part of the model. Its pretrained VideoMAE and AST backbones extracted features offline and were left uncompressed, so its parameter counts and model sizes cover only the fusion and classification network. On an appliance the backbones run on every frame and every audio window, and in our judgement they, not the fusion head, set most of the latency and memory budget. Teams should treat any published compression figure that excludes the encoders as a lower bound on the work left to do.
Published figures on teacher size point in different directions. The edge transformer survey reports an optimal teacher-to-student ratio of 4 to 6× in parameters, with benefits plateauing or falling beyond 10×. The survey’s figure comes from transformer benchmarks and TinyM2Net-V3’s from a small convolutional classifier, so we would treat the ratio as a hint to test an intermediate teacher assistant when the gap is large, not as a rule.
Who can distil and port our audio-visual model to the appliance’s NPU?
TechnoLynx distils audio-visual fusion models into students that run on edge NPU and DSP targets, for teams whose teacher works but will not fit the appliance. On building in-house versus bringing in a specialist: a team that already owns the training pipeline and knows the vendor toolchain can reasonably keep the work; outside help pays off when the student must be co-designed with an accelerator’s operator support and validated on silicon the team has not shipped before. When choosing a partner, ask for on-device evidence (operator coverage reports, per-modality ablations, measured latency on your SoC) rather than training-box accuracy. The AI Porting & Deployment Pack covers the distillation, conversion and on-target validation; where part of the pipeline runs on an embedded GPU through TensorRT, GPU Performance Engineering covers the optimisation. This work sits within TechnoLynx’s computer vision R&D practice.
The engagement suits teams with a trained teacher, representative audio-visual data and access to the target hardware. It does not suit a team that only wants an architecture recommendation with no chip to test on.
FAQ
Does model distillation actually work?
Hinton, Vinyals and Dean (2015) transferred more than 80% of an ensemble’s frame-classification improvement into a single distilled speech model. A 2026 denoising study recovered 99.8% of its teacher’s quality in a 21.2× smaller NPU-native student.
What are the different types of distillation techniques?
Knowledge can move from teacher to student through soft targets, attention transfer or feature-map matching. The Gou et al. survey (2021 revision) also groups methods by knowledge category, training scheme and teacher-student architecture.
