Embedded software engineering for an edge-AI device changes the order of work. The model, the operator support of the on-device inference runtime and the memory budget all have to be checked before firmware is written, because they decide whether the model fits and loads at all. If a model uses an operator the target runtime cannot execute, then depending on the runtime, that part of the graph falls back to the CPU and adds latency, or the load aborts on the device. Why the work runs on the device in the first place is a separate question, covered in our piece on edge vs cloud placement. What follows is what that placement does to the engineering.
How Does Developing Embedded Software for an Edge-AI IoT Device Differ from Building a Cloud ML Service?
Edge AI deployment is a multi-stage engineering process that covers model conversion, operator compatibility handling, runtime integration and precision verification. Deploying large models on resource-constrained hardware is not a scaled-down version of cloud serving: one 2026 survey describes it as a qualitatively different problem, governed by a hierarchy of binding constraints that interact with every upstream design decision. The same survey warns that optimising a model on server hardware and then deploying it to edge hardware does not necessarily give the expected efficiency.
At the microcontroller end, the gap is starker. Lin et al. describe MCUs as having no DRAM, no operating system, SRAM smaller than 256 kB and read-only flash. The TensorFlow Lite Micro authors (David et al., MLSys 2021) note that many embedded platforms lack basic features that mainstream platforms take for granted, such as memory management and library support. Their framework therefore assumes the model, the input and the output arrays are already in memory, and leaves file loading and peripheral access to the application.
A model that performs in a training environment is a different artefact from one that performs on the deployment target, and closing the gap between the two is engineering work, not a hand-off. On an edge device, firmware written against a model that later turns out not to fit, or not to convert, gets rewritten.
Memory, Power and Deadlines Decide the Model, Not the Firmware
Memory sets the hardest limit. A large embedded system has a few megabytes of flash and at most a megabyte of SRAM, and a small one has a few hundred kilobytes or less, split between ROM and RAM. On the STM32F746, a Cortex-M7 part with 320 KB of SRAM and 1 MB of flash, SRAM limits the size of activations and flash limits the size of the model. For scale: according to the Ariel-ML authors, even a single convolutional layer of ResNet-34 needs more than 400 KiB of RAM after quantization.
Reducing parameters alone misses the real bottleneck, which is activations. Lin et al. show that MobileNetV2-1.4 is 4.2× smaller than ResNet-50 at the same ImageNet accuracy, yet its peak memory is 2.3× larger, so a “mobile-efficient” architecture can make the SRAM problem worse. Google’s LiteRT for Microcontrollers guidance puts the requirement plainly: the model has to fit in the device’s memory alongside the rest of the program, both as a binary and at runtime. Bit width is the other lever: the Coral Edge TPU requires 8-bit quantized tensors, and our guide to choosing an edge AI accelerator and quantization costs covers the accuracy costs.
The Runtime Has a Footprint Too
The inference runtime’s own code also counts against the memory budget. The LiteRT for Microcontrollers core runtime fits in 16 KB on an Arm Cortex-M3. Integrating IREE grew firmware size by roughly 1.1 to 2.5× and RAM use by at least 1.2× in the Ariel-ML measurements. IREE also needed a 16 kB stack for its VM call chain, where RIOT-ML needed 2 kB. On a part with a few hundred kilobytes of SRAM, a 16 kB stack is a design decision, not a detail.
TFLM chose an interpreter to allow edge AI model updates in the field without re-exporting code, which can be infeasible with code generation. Lin et al. argue the opposite way for the tightest parts: interpreter-based runtimes spend extra memory, the most expensive resource on an MCU, on model meta-information, and generic runtimes carry kernels a given model never uses.
The Tensor Arena Is a Fixed Budget
TFLM and ExecuTorch plan memory ahead of time rather than allocating at runtime. TFLM requires the application to supply one contiguous memory arena, allocates everything from it during initialisation, and avoids allocation afterwards so that heap fragmentation cannot break long-running applications. A bare-bones setup from the TFLM documentation looks like this:
// Buffer for the tensor arena:
size_t tensor_arena_size = 2048;
uint8_t tensor_arena[tensor_arena_size];
// Interpreter using the shared tensor arena above:
tflite::MicroInterpreter interpreter(
tflite::GetModel(my_model_data), ops_resolver,
tensor_arena, tensor_arena_size);
ExecuTorch takes the same approach: its core runtime creates no heap-allocated memory, and all memory is supplied by the user, which also lets tensor data be placed in specific regions such as SRAM or DRAM. ExecuTorch’s memory planning fixes the arena size at export time. In the ExecuTorch paper’s Raspberry Pi Pico 2 example, model optimization through int8 quantization and operator fusion shrank the memory-planned arena from 101.2 KiB to 3.8 KiB, putting total RAM at about 11 KiB within the board’s 520 KiB of SRAM. Binary size responds to the same discipline: ExecuTorch’s selective build, which links only the kernels a model uses, can bring binary size down from MiB to KiB.
Power and Real-Time Deadlines
Model size also shows up in power use. Google’s LiteRT for Microcontrollers documentation notes that large, complex models can raise the duty cycle, so the processor spends more time working and less time idle, which increases power draw and heat. For battery devices, that ties model complexity directly to the power budget.
Deadlines add the last constraint. New MCU hardware (Arm Cortex-M, 32-bit RISC-V, ESP32) is moving to multicore designs, and using several cores for inference is becoming necessary to stay within real-time constraints. The gain is bounded: Ariel-ML’s multicore scheduler reached up to 1.6× speedup on the dual-core RP2040 and RP2350, close to the limit set by bus contention. On these dual-core parts, a latency budget that needs more than that from the second core has to be met in the model instead.
Why Does Operator Support on the Inference Runtime Have to Be Checked Before Firmware Is Written?
Each runtime supports only a subset of operators, and that subset differs across runtimes. The TFLM paper records that TensorFlow had more than 1,400 operations while TensorFlow Lite supported only about 130. The same authors note that users often find out whether their operations are compatible with the target framework only after building and training the model. Support also does not stop at the operator’s name: an operator may be supported for some data types and not others, exclude certain parameter ranges, or work only in combination with other operations.
Support varies by chip, not only by runtime. Across CPU, GPU and NPU plugins the NPU has the most limited operator set, and different RKNN chips (for example RV1106 and RK3566) have different operator sets and quantization capabilities. The set of unsupported operators differs across runtimes, and substitution strategies differ too. Three structural traps sit next to the operator list:
- Fixed shapes. Edge inference runtimes generally prefer fixed-shape compilation, and converting dynamic shapes is one of the deployment steps that most often needs manual work. TFLM does not support dynamic shapes at all.
- Tensor layout. The RK3588 NPU is optimised for NHWC, and passing a standard NCHW ONNX model straight to RKNN-Toolkit2 produces incorrect output, not an error.
- Accelerator-specific requirements. The Coral Edge TPU fully accelerates only models with 8-bit quantized tensors, sizes and parameters constant at compile time, and tensors with more than three dimensions allowed a size greater than one only in the three innermost.
The silent layout failure is the strongest reason to check early. A wrong output that still loads is far more expensive to find once firmware, sensors and pre-processing are wrapped around it.
Worked Example: One Upsampling Layer and a Pinned Runtime
A dehazing study on a Raspberry Pi 4 shows how an operator the pinned on-device runtime cannot register stops deployment. The authors’ first TensorFlow Lite exports emitted TILE version 3, because a Keras UpSampling2D layer lowered to a dynamic-shape chain (SHAPE → STRIDED_SLICE → TILE) rather than to a static resize. The interpreter on the device could not register that operator and aborted inside allocate_tensors(). The limit came from the device’s environment: 32-bit Raspberry Pi OS on Python 3.7 pinned the interpreter to tflite-runtime 2.11. The fix was a model change, not a firmware change. The authors rebuilt the network with an explicit nearest-neighbour resize, which brought the operator set down from ten distinct operators to six with no change in the float outputs (maximum absolute difference 0.0).
The runtime version that matters is the one that will actually run on the device, and its operator registry belongs in the design inputs alongside the SRAM figure.
What Happens When a Model Op Is Unsupported on the Target Edge Runtime
Several accelerated runtimes partition the graph and send what they cannot accelerate to the CPU. The details differ, and they decide how much of the model the accelerator runs.
| Runtime / toolchain | Behaviour on an unsupported operator |
|---|---|
| Coral Edge TPU compiler | Partitions the graph at the first unsupported operation; that operation and everything after it runs on the CPU, even supported operations later in the graph. |
| Arm Vela (Ethos-U microNPU) | Parts of the model that cannot be accelerated are left unchanged and run on the CPU with an appropriate kernel. |
| ONNX Runtime | Partitions the graph by querying each execution provider’s capabilities; the default execution provider runs whatever the specialised ones cannot. |
| ExecuTorch | The partitioner sends only supported subgraphs to a delegate; unsupported operators run on the portable CPU fallback, such as XNNPACK. |
| LiteRT CompiledModel API | Handles delegate setup and falls back automatically when an op is unsupported. |
| tflite-runtime on a pinned device | In the dehazing case, a TILE operator version the pinned interpreter could not register aborted allocate_tensors(); the model did not load. |
The share of operators on the CPU understates the cost. Coral’s documentation warns that even a small fraction of a model running on the CPU can slow inference by an order of magnitude compared with a version that runs entirely on the Edge TPU. ExecuTorch measurements on the Vulkan delegate add detail: four unsupported operator types split a ViT model into 25 Vulkan partitions; operators left on the CPU accounted for about 29% of ViT and 22% of Swin-T latency, while copying tensors across graph breaks cost only 5% to 6%. Fallback also wastes hardware: according to the Parallax authors, existing frameworks handle fallbacks poorly, leaving CPU cores idle and causing high latency and memory spikes.
Mitigations for an Unsupported Operator
The first option is to replace the operator with an equivalent the target supports. Unsupported operators need equivalent substitution or graph rewriting before deployment, and some toolchains automate part of this: Olive’s QNN preprocessing pass replaces or decomposes unsupported operators into supported subgraphs to reduce fallback on Qualcomm’s HTP backend. The dehazing case above is the manual version: an architecture edit that kept the maths and changed the lowering.
The second option is to extend the runtime, which has its own costs. LiteRT allows extra operators, but the runtime environment then has to be modified too, which can limit standard deployment options. Google’s own table shows the interpreter binary growing from 796 KB with built-in ops only to 23.0 MB once TensorFlow ops are added. Custom operators not built for device deployment can also perform worse on constrained devices than in a server environment.
The third option is smarter scheduling of the fallback itself. On three smartphones, Parallax reported 9% to 46% latency reductions in heterogeneous mode without model changes, at an average 26.5% memory overhead. That trade was measured on smartphones.
Export Success Is Not Runtime Success
In one 2026 study of LLM-agent-driven conversion, the final RKNN deployment of YOLO-World passed verification, yet analysis showed four operators had fallen back to CPU execution. The same model on ONNX Runtime’s QNN execution provider split into six subgraphs because Einsum was not supported. The RKNN deployment still passed verification.
A passed verification therefore does not show that the accelerator ran the whole model. The deployment pipeline described in that study starts by confirming that the original model runs and produces reference outputs, then checks that the converted model’s output shapes and precision match. Device diversity complicates this: the ExecuTorch authors note that AOT-compiled models may require developers to query device capabilities and ship hardware-specific model files.
Checks to Run Before the First Line of Firmware
- Confirm the original model runs in its training framework and record reference outputs on fixed test inputs.
- Pin the exact runtime version, toolchain and drivers that will exist on the device, and convert against those.
- Check every operator, data type and parameter range against the runtime’s support for the specific chip, not the chip family.
- Fix input shapes and convert the layout the accelerator expects (for example NHWC) before conversion.
- Budget memory: model binary in flash, plus tensor arena, runtime and stack in SRAM, alongside the application.
- Convert, run on the target, compare against the reference outputs, and confirm which partitions actually ran on the accelerator.
- Measure latency and duty cycle on the device against the real-time deadline and the power budget.
Steps 1 to 5 come before any application code exists. Skipping them does not save time. It moves the discovery to the point where firmware, sensor drivers and enclosure decisions all depend on a model that does not fit.
What the Evidence Does Not Settle
The fallback costs above come from Coral’s guidance for the Edge TPU, ExecuTorch measurements of vision transformers on its Vulkan delegate, Parallax measurements on smartphones, and YOLO-World conversions for RKNN and QNN. Partitioning also differs: the Edge TPU compiler partitions once, while ONNX Runtime splits the graph by each execution provider’s capability. Treat those figures as evidence that fallback is expensive, not as a prediction for a specific network. For the wider stack these devices sit in, see the tech stack for edge computing.
FAQ
What Type of Memory Is Firmware Typically Stored In?
On microcontrollers, the model sits in read-only flash, which limits its size, while activations live in SRAM.
Which Is Better, IoT or Embedded Systems?
They are not alternatives: an IoT device is normally an embedded system with network connectivity, so building one involves embedded software engineering.