Put the decision that cannot wait on the node. Send work that needs more context or a bigger model to the gateway or the cloud, and send upstream only what the next tier needs: anomalies, events and summaries, not raw streams. Keep the split point adjustable after deployment, because the measured trade-offs between latency, energy and accuracy shift with network conditions and anomaly rates. When a fleet’s sites share sensor types, link conditions and latency budgets, this edge inference pattern becomes one reference edge computing architecture that you tune per site through configuration instead of redesigning for each site.
Where should each inference step run: node, gateway or cloud?
Placement is a trade between three tiers, and one three-tier study measured each of them. In a three-tier predictive-maintenance study on mining machinery, the ESN-PdM framework (arXiv, 2024) measured average inference latency of 3.33 ms on the sensor node, 146.67 ms on the gateway and 641.71 ms in the cloud. Accuracy moved the other way: over 90% on sensor and gateway, 99% in the cloud. Running inference on the sensor cut node energy per inference cycle by about 44%, mainly because sending the compressed signal over the radio cost far more energy than running the model locally.
Neurosurgeon (ASPLOS 2017) found that the cloud’s compute advantage often fails to show up end to end because data transfer takes most of the time, especially over 3G and LTE. Its layer-level partitioning improved end-to-end latency by 3.1× on average and cut mobile energy by 59.5% on average compared with cloud-only execution.
Hierarchical inference combines tiers instead of picking one. Hierarchical inference runs a small model on every sample and offloads only the samples it is likely to get wrong. On the Coral Micro and Raspberry Pi with CIFAR-10, a 2024 measurement study found that this approach cut latency by up to 73% and device energy by up to 77%. The same study found that pairing a smaller on-device model with offload beat a larger on-device model that met the accuracy target by itself.
In existing hierarchical-inference work, confidence values decide how much work leaves the device. In a cellular-IoT logistics system (arXiv, 2021), edge decisions with confidence below 0.75 went to a deeper model at the fog gateway for re-evaluation, and the authors describe that threshold as a tunable trade between decision confidence and response time. Sending those decisions to the fog model raised F1 by 7.36% on average. Fixed split points also suffer when bandwidth fluctuates. Splitwise partitions LLM inference between edge devices and the cloud. On a 10 Mbps link it shifted work towards the edge, and it reported 53–61% lower P95 latency than cloud-only execution. That workload is far heavier than sensor scoring, but the lesson about static split points carries over. Partitioning also covers more than one kind of problem, and our breakdown of partitioning inference across edge and cloud as three distinct problems separates them before you pick a pattern.
Placement decision table
| Tier | Run here when | Measured reference | Watch for |
|---|---|---|---|
| Node (MCU) | The decision is time-critical or the link is unreliable, and the model fits SRAM and flash | 3.33 ms latency, about 44% lower node energy per cycle than offloading (ESN-PdM, 2024) | Lower accuracy; complex tasks may not fit at all |
| Gateway | Scoring needs several sensors or a larger model, and local-network latency is acceptable | 146.67 ms latency, accuracy over 90% (ESN-PdM, 2024) | Limited memory and often no accelerator, so an optimised runtime is needed |
| Cloud | Accuracy is critical, or the analysis spans the whole fleet | 641.71 ms latency, 99% accuracy (ESN-PdM, 2024) | Depends on the backhaul; transfer overhead dominated end to end in Neurosurgeon’s mobile tests |
Can our existing nodes and gateways run their share without new hardware?
They can when the trigger model fits the node’s flash and SRAM and the node runs a narrow trigger model, not the whole analysis. Typical microcontrollers have 192–512 KB of SRAM and 256 KB–2 MB of flash, and the model’s weights and code have to fit in that flash. Peak memory during an event, including activations, also has to stay inside SRAM, or inference falls back to slower off-chip memory. On an ARM Cortex-M0+ node with 256 KB of flash and 32 KB of SRAM, a one-hidden-layer autoencoder took 6,080 bytes, about 2% of firmware. The ESN-PdM sensor model fitted on an ESP32 after full integer quantisation, at about 170 times smaller than the cloud model. Gateways have more room: their processors range from an ARM9 to dual- or quad-core ARM parts.
Complex tasks are where on-device inference runs out. On the ESP32 and Arduino Nano, an image-classification task that a tiny model could not handle accurately was better offloaded, because a larger local model would have been slower still. A model that scores well in training still has to be ported, quantised and validated on the hardware that will run it, and closing that gap is engineering work, not a hand-off.
Keeping the fleet working when the link drops for minutes or hours
Resilience covers three separate designs: local control logic, a transport layer that survives intermittent backhaul, and a store-and-forward queue sized for real outages. Each one fails in its own way, so design each one deliberately.
Local decision logic so irrigation never waits for the cloud
Irrigation actuation should run on local threshold logic over buffered sensor readings, because LoRaWAN cannot guarantee when a command from the cloud will arrive. On a Class A device, the network server can only send a downlink after the device has sent an uplink. Measured LoRaWAN round trips ranged from 1.12 s to 6.59 s. One study of event-triggered control concluded that LoRaWAN suits systems that can tolerate blackout periods and delays of up to 10 s, and that the protocol cannot send time-critical data while a duty-cycle restriction is in force.
We would start the on-node logic from a moisture threshold with hysteresis, plus a rule for what to do when readings go stale. Event-triggered control fits this well: the node checks a triggering condition at each sample and transmits only when the measurement error exceeds a threshold. Valve nodes on mains power can use Class C, which keeps the receiver listening. The cost is that local logic does not see context held only at the centre, such as a forecast or a fleet-wide allocation decision. Keep that context as a parameter the centre updates, not as something actuation waits for. One building-monitoring guide makes the general point: deciding which functions must stay local is more useful than pretending connectivity never fails.
Transport that survives intermittent cellular without a reconnect cost spike
Load spikes on reconnect come from clients reconnecting together and draining backlogs at line rate. The fix is flow control: cap the catch-up rate so the backlog drains at a pace consumers can absorb. Actility, a LoRaWAN network vendor, makes the same point about gateways that buffer during WAN outages: they need graceful debuffering to avoid packet storms when the connection comes back. When many gateways lose the same backhaul and reconnect at once, give the reconnect timer exponential backoff with randomised jitter. Doubling alone keeps clients that failed together in sync.
On QoS: for a forwarder that sends from a durable spool, QoS 1 is almost always the right setting. QoS 2 adds a second round trip per message for a guarantee the consumer can get more cheaply. To get effective exactly-once delivery, give each message a stable idempotency key (device, metric and source timestamp) and write it with an upsert at the consumer. Set the MQTT 5 session expiry longer than the outage you designed for. Keep live messages on a high-priority lane and drain the backlog under a token-bucket limit.
Sizing the store-and-forward buffer and choosing what to evict
The buffer has to be local, persistent and sized for realistic outage windows. A best-effort in-memory queue is not enough. A persistent session in default Mosquitto is a short-outage tool. Mosquitto’s default queue holds 1,000 messages, so a subscriber receiving 20 messages per second fills it in 50 seconds. Azure IoT Edge keeps offline messages for a default time to live of 7,200 seconds, so any design outage longer than two hours needs a different setting and more disk. For outages of minutes up to a few hours, a local broker bridge works well. For outages of days, or when you need to prioritise or drop data selectively, put a durable spool in front of the forwarder.
Size the buffer with arithmetic, not a guess. One published sizing example takes a gateway publishing 400 messages per second at about 500 stored bytes each, which comes to 720 MB per hour. With 1,500 messages per second of backfill capacity, clearing a 72-hour backlog took about 26 hours, because live traffic keeps arriving while the backlog drains. The same source suggests sizing the spool partition at no more than 60–70% of usable flash. Nodes need a buffer of their own too: both the logistics node and the ESN-PdM sensor carry external storage for logging when there is no connectivity.
When storage fills before a sync window, the options are block, drop-newest or drop-oldest. Dropping the oldest is usually right for telemetry, dropping the newest keeps a complete prefix for audit trails, and blocking acquisition can crash control logic. For anomaly-driven fleets we would add a priority on top of recency: keep flagged events and periodic summaries, and shed routine readings first. Which rule is right depends on what the data is used for downstream, and that is not always known at design time. Platforms support this: AWS IoT Greengrass stream manager, for example, lets you set per-stream priorities so critical data is exported first. Then test the cycle on purpose. Remove only the WAN link, make a recognisable change during the outage, restore the link, and reconcile timestamps, duplicates and replay order at the destination.
For a battery-powered irrigation node, TechnoLynx would move moisture-anomaly detection onto the node itself, with an uplink only when the local trigger fires. TechnoLynx’s AI Porting & Deployment Pack takes this work from the node’s latency, airtime and power constraints, through fitting the on-node trigger model and validating it against an always-uplink baseline, to tuning the local/central split once it is in operation.
Do HVAC controls and portfolio analytics need raw sensor streams centrally?
HVAC control loops can run at the building gateway on raw local readings, while the centre receives summarised occupancy events. At least one vendor’s hardware already works this way. Terabee’s BACnet gateway, for example, takes people-counter data over MQTT or LoRa, does aggregation and data fusion on the gateway, and passes simple outputs to the building management system over BACnet/IP. A commissioned local BACnet path can keep running through an external WAN outage while the building LAN, BMS and gateway services stay up, so facilities staff keep live values. This is the general edge pattern: the edge runs the loop that must not wait, and the core receives the telemetry, aggregates and decisions it needs for coordination and analysis.
For portfolio reporting and anomaly detection, the minimum central feed we would start from is per-zone occupancy events and interval aggregates, setpoints, a few plant variables and local fault flags. Treat summary sufficiency for portfolio-level anomaly detection as a design hypothesis to check on a pilot, not a settled fact. We would expect some fault types to need higher-resolution data than summaries carry. For those, make raw-stream upload something you switch on per site and per period while an investigation runs. Outages will also delay summaries and leave reports stale until the backlog clears.
The split threshold can be tuned centrally after deployment without re-flashing every sensor, provided devices read it as configuration. The ESN-PdM framework, for example, uses a command service to change a node’s inference mode and push model updates. TechnoLynx would design this kind of remotely configurable two-tier controller, with telemetry linking missed detections to each threshold version, and run staged threshold tuning before fleet-wide use.
How to prove the new split still meets the alert SLA before fleet rollout
Measure it on a pilot. Timestamp every stage from the local flag to the dispatch alert on a representative subset of nodes, and compare the measured worst case, not the average, against the SLA. TechnoLynx’s AI Porting & Deployment Pack covers this validation step. It instruments the pilot nodes and backend for stage-by-stage timing, tunes the gateway pre-filter and confirmation logic, and analyses worst-case and percentile latency against the SLA before supporting a phased rollout.
End-to-end latency under a split is the sum of on-node scoring and pre-filter time, queuing and transmission time over the radio link, and backend confirmation and dispatch time. If a node holds a borderline event for local confirmation before escalating, the whole confirmation window adds to the normal transmit delay. The link term varies widely. NB-IoT delays range from tens of milliseconds to several tens of seconds depending on radio conditions and network load. On LoRaWAN, a confirmed uplink without its ACK waits a retransmission timeout, which backoff can stretch further. Clock synchronisation between edge and cloud matters as well: without it, cross-tier timestamps cannot be subtracted.
The pilot only proves something if it includes the conditions that break things: poor signal, peak traffic, the weakest hardware class and a deliberate outage. Canary releases follow the same logic: deploy to a small group first, check health signals, then expand. If the pilot’s worst case sits comfortably inside the SLA, including a margin for backend dispatch processing, move to a phased rollout. If it gets close, tighten the local confirmation threshold or shorten the pre-filter dwell time before scaling further. See also our guides to decomposing end-to-end latency budgets and instrumenting edge inference runtimes for placement decisions.
What update rate can a LoRaWAN duty-cycle budget actually support?
Work backwards from airtime. The regional duty cycle caps transmit time per sub-band, each frame’s time on air at your spreading factor and payload sets how many frames fit, and that figure, minus headroom for retries and joins, is the fastest update rate you can promise. After each frame, a device has to stay silent in the sub-band for Ts = Ta(1/d − 1), where Ta is time on air and d is the duty cycle. In the EU 868 band at 1%, that gives each end device at most 36 seconds of transmission per hour in each sub-band.
Worked example: EU868 at 1% duty cycle
One measurement study put LoRaWAN uplink time on air between 61 ms and 2.7 s for 10–50 byte payloads across SF7 to SF12. The table applies the duty-cycle formula to both ends of that range. The Things Network’s sandbox caps uplink airtime at 30 seconds per node per day, which on that network binds well before the 1% duty cycle does.
| Time on air per frame | Max frames per hour per sub-band (36 s ÷ Ta) | Silence after each frame (Ta × 99) | Frames per day under a 30 s/day cap |
|---|---|---|---|
| 61 ms (short payload, low SF) | about 590 | about 6 s | about 490 |
| 2.7 s (larger payload, SF12) | about 13 | about 267 s (4.5 min) | about 11 |
Assumptions: one sub-band at 1%, no retransmissions, no acknowledged frames. If a stack enforces the limit per channel, it divides the sub-band budget: eight channels in a 1% sub-band leave 0.125% each. The network can tighten things further, because DutyCycleReq sets an aggregated duty cycle of 1/2^MaxDutyCycle. Confirmed frames drain capacity, because the gateway’s downlinks also carry duty-cycle off-periods. Other regions work differently. The FCC sets no duty cycle in the 902–928 MHz band but imposes a 400 ms dwell time on uplinks. Payload ceilings also bind: RP002-1.0.1 caps the MACPayload at 230 bytes.
For parking sensors, the realistic availability promise is the change-rate at rush hour that this budget supports, not the sampling rate. A sensor that reports only state changes sends nothing while the bay is stable. Rush hour is when changes cluster, so size for the peak change rate, not the daily average. Batching several results into one frame saves the fixed cost of each frame: the header adds 13 bytes at every spreading factor, and ETSI lists per-packet time overhead from 40 ms at SF7 to 1,100 ms at SF12. The price of batching is staleness, so the batch interval becomes part of the update promise you make to drivers.
We would decide per gateway class whether HVAC fault scoring runs on the gateway or in the cloud, weighing uplink cost and volume against compute headroom, latency need and accuracy. A two-tier pre-filter fits where neither extreme pays for itself. Through the AI Porting & Deployment Pack, TechnoLynx builds that cost model per gateway class, implements the local scorer for the weakest gateway class in the fleet, and validates backhaul savings and latency against the central-only baseline.
Can one split-inference reference design roll out to every site?
Yes, if the sites are similar enough. We would standardise fixed partition points, a common aggregation protocol and a common update and monitoring pattern, and tune only parameters such as thresholds and bandwidth caps per site. TechnoLynx’s AI Porting & Deployment Pack is the route for porting the edge share of that reference design onto each node and gateway class and deploying it across the fleet.
Edge orchestration guidance reaches the same conclusion from the operations side. Start with 10–20 sites, standardise site profiles and “golden” configurations so sites are not treated as snowflakes, then put change windows and KPIs in place. Customising each site solves short-term exceptions but creates long-term support debt and makes incidents harder to diagnose. Rollouts should tolerate the network: updates queue and sync when links return, staging protects critical traffic, and critical apps keep running when the central controller is unreachable. Handle model versions and rollback the same way you handle application versions, with validation site by site before fleet-wide expansion.
Ongoing maintenance then rests on tooling to update, observe and recover sites at scale. The baseline signals are app health, resource saturation, network stability, and version and configuration drift. Updating MCU-class nodes is the hard part: their software is rarely updated once programmed, and updating many remote devices can be difficult or impossible. That is one more reason to keep the tunable threshold at the gateway or in configuration, not in node firmware. If a subset of sites differs materially (no reliable uplink, different sensor hardware), we would create a second variant instead of stretching one design over both. Expect each new site to still need a validation pass.
How much uplink do you save by sending events instead of raw streams?
It depends on how rare the anomalies are and how much data each record carries. Where the sensor is a camera, as on a crop harvester, TechnoLynx’s Computer Vision R&D work builds the on-device filter that decides which frames are worth sending.
The measured results are large but come from controlled or simulated setups. A 2026 study in Scientific Reports sent only anomalous vibration samples over LoRa and measured a 98.19% cut in data volume and a 98.10% cut in transmission energy with DBSCAN, against 97.32% for both with Isolation Forest. That test used a single 3D-printer vibration scenario, a fixed LoRa configuration and a fixed power supply, so battery behaviour was not part of it. An APSIPA 2019 study of edge anomaly filtering cut observations sent by about 90%, 76% and 99.5% across three datasets. The saving was largest for high-dimensional data with a low anomaly rate. A predictive filter that transmits only when a reading deviates from its forecast suits agriculture, where consecutive readings barely change, though that work was validated in simulation only.
Choose the reduction method by what the downstream task needs. Send summary statistics when the task only needs the mean, variance and trend. Use event-triggered transmission when the signal is mostly static. Gate transmission on an anomaly trigger when the monitored activity contains only sparse anomalies. When you need both trend monitoring and forensic detail, combine aggregates in normal operation with full-resolution bursts on a trigger. Two costs come with this. False positives erode the saving: Isolation Forest’s higher false-positive rate made it use slightly more energy than DBSCAN. Aggressive filtering also throws away data you may later want for retraining, so a small sampled slice of raw data is worth budgeting for. Whatever the scheme, the resulting message rate and payload still have to fit the duty-cycle and payload limits worked through above.
For vehicle fleets, we would keep the time-critical hazard signal on the vehicle and batch heavier analytics centrally across vehicles. Central cost then follows aggregate load, not one dedicated call per vehicle at peak concurrency. TechnoLynx’s Inference Cost-Cut Pack addresses that central batched path, starting by establishing which hazard functions are time-critical and what the fleet’s peak concurrency looks like.
Who can help us design and roll out split inference across our sensor fleet?
TechnoLynx provides IoT and embedded AI engineering that covers deciding what runs on the node, gateway and cloud, fitting the edge models to the hardware you already have, and proving latency and backhaul savings on a pilot before rollout. When choosing a partner, ask for measured pilot results against your SLA, not projections. Where the gateway or central tier needs new accelerator hardware, LynxBenchAI’s published results show how candidate devices run the public PyTorch stack. They do not measure your own workload.
This is not the right engagement for a single site where the vendor’s gateway firmware already does what you need, or for teams whose real problem is radio coverage and gateway placement, which a LoRaWAN network operator or integrator handles better. Teams with strong embedded and MLOps skills can often run the design themselves, using the store-and-forward and duty-cycle arithmetic above.
The two numbers to watch after go-live
A split-inference design rests on two assumptions you cannot fully check before go-live: that the threshold sends the right samples upstream, and that the buffer outlasts the outages the fleet actually sees. For the buffer, oldest message age is the single most useful signal: it is the real lag of your data, and drain rate against arrival rate shows whether a backlog will ever clear. For the threshold, the signal is missed detections per threshold version, collected as part of every staged change. A fleet that tracks both has what it needs to keep moving its split point. A fleet that tracks neither has effectively fixed its split point at deployment, even if the setting is still configurable.
