Continuous security monitoring for a fielded AI fleet rests on three mechanisms working together. The first is a device registry that records, for every unit, the model version it should run and the version it reports running. The second is telemetry that devices buffer locally and forward when they reconnect. The third is a reconciliation loop that keeps comparing the two and acts on any gap. The signals worth alerting on are version or hash changes, error spikes and connectivity gaps, plus failed integrity checks, which should trigger immediate action.
This is the operational half of AI engineering for IoT and embedded devices. The model has shipped, and now someone has to know what each unit is actually doing, including the units that have not spoken to the backend in a week.
How can we see which model version every fielded device is running?
Keep two records per device and compare them: the desired state, meaning the version the operator assigned, and the reported state, meaning what the device says it is running. AWS IoT’s Device Shadow service is built on exactly this split. Applications write a desired object, the device writes a reported object, and AWS IoT publishes a delta object listing where the two differ.
TechnoLynx builds this layer for deployed fleets as the Production AI Monitoring Harness. The harness gives you a per-device view of model versions, alerts on failed updates and reporting that survives connectivity gaps. The engagement starts by framing which version and status signals your devices need to surface, then integrates a telemetry and device-management platform with your OTA update and rollback reporting.
AWS IoT and Azure IoT Edge already expose the raw material. In Azure IoT Edge, the agent’s reported properties include the status of the last desired properties it saw, the status of each running module and a copy of the desired properties currently running. That copy is what tells you whether a device applied the latest deployment or is still on a previous manifest. Each module reports a version, a build and a commit, along with a runtime status of running, stopped, failed, backoff or unhealthy. The status code for the last desired properties separates success (200) from an invalid configuration (400), an invalid schema version (412), empty desired properties (417) and a general failure (500).
What the registry records matters as much as having one. InTechHouse makes the point that an edge model version has to trace back past the training run to the optimisation pipeline that produced the exact binary, because two models built from the same checkpoint with different quantisation or pruning settings can behave differently in production. Red Hat describes packaging patterns where the model and the inference runtime have independent release cycles and versioning histories. In that case the registry needs both versions, not one. How to lay out that registry across several fleets and firmware lines is covered in structuring model registries for multi-fleet, multi-firmware deployments.
With desired and reported versions in one place, a dashboard showing which units are on the current model and which are on an old one is a group-by. Netdata’s fleet guidance applies the same grouping to OTA rollouts, watching each firmware-version cohort for regressions before a rollout completes. That guidance also names the single highest-leverage check: whether you can see which devices are reporting right now without writing a query.
How do you monitor AI models in production when they run on edge devices rather than in the cloud?
The monitoring moves to the device. Each unit computes what it can locally and sends summaries to a central tier that it reaches by its own outbound connection. Red Hat states the constraint plainly: for most edge use cases, inference must not need internet connectivity to function, while telemetry and monitoring data flow from the edge back to the core. Netdata describes the resulting architecture as edge-resident rather than centralised, outbound-only rather than pull-based, and store-and-forward rather than always-connected. Pull-based systems that have to reach into each device cannot work for most fleet endpoints sitting behind NAT and firewalls.
Bandwidth and privacy both push the same way. InTechHouse recommends computing drift-relevant statistics on the device, such as feature distributions and confidence histograms, and transmitting only those compact aggregates rather than raw images or sensor readings. Explicit filtering rules, such as sending only flagged events above a confidence threshold, also limit what could be exposed if a transmission channel were compromised. The central tier still matters, because aggregating across the fleet is what separates a systemic issue affecting many devices from a single-device hardware fault. The Siemens Industrial Edge architecture on Azure shows one concrete version of this: Siemens AIIS sends metrics through an OpenTelemetry Collector to the AI Model Monitor, which aggregates them from the edge devices and sends them to Azure Monitor.
Model behaviour adds a layer of monitoring on top of firmware monitoring. Promwad notes that firmware rollback is triggered by hard failures such as boot loops and watchdog timeouts, while a model can run without exception and still produce degraded output that no fault code reveals. Drift detection is its own subject. The thresholds and escalation behind it are covered in safe fleet-wide model updates with canary rollouts and tiered drift alerts. This page stays with version, update and liveness signals.
Getting an alert when an update fails, not a call from the site
A failed update should arrive as an event from the device, routed straight to whoever is on call. The ETSI EN 303 645 baseline describes the pattern for consumer IoT. A device that detects an update could not be delivered or applied can generate a log entry and notify a trusted peer, such as an administrator, over a secure channel. ETSI also points out that telemetry from many devices lets a manufacturer notice that updates are failing on authenticity checks. One failure might be a flaky unit, but many failures on the same authenticity check point at the signing pipeline or at tampering.
Underneath the alert you need update state, not just a final version. The IETF’s RFC 9019 firmware update architecture gives its status tracker the job of showing which devices have received an update and which are still pending. It also requires the boot process to expose the current firmware version, the update status and whether errors occurred. Balena’s fleet tooling shows a device as Updating while it moves to a new release and records update status and its duration, with Done meaning the device has reached its target state. Those fields are only populated on supervisor 16.7.0 or later.
The distinction to design for is pending versus failed. InTechHouse argues that an offline device should have the update queued and applied automatically on reconnect, not be treated as a deployment failure. An explicit failure report or a failed integrity check should trigger an immediate alert. Where security and model-quality alerts share a fleet, route them apart. A tampering signal belongs with the security on-call, and a drift signal belongs with the ML team.
What happens if a model or firmware update is interrupted or fails on a field device?
A well-designed device returns to a known good version and reports what happened. ETSI EN 303 645 names watchdogs and dual-bank flash or recovery partitions as ways to bring a device back to a known good version or its factory state after a failed automatic update. The same standard asks that devices stay locally functional through a loss of network and recover cleanly after a loss of power.
For models specifically, InTechHouse describes the receiving device validating the update’s signature and integrity before installing it, then running a brief validation check before fully activating the new version, with automatic rollback if defined thresholds are not met. Defining those rollback conditions, such as a detection rate floor, a latency bound or an error-rate ceiling, in advance is what makes a staged rollout a safety mechanism. Promwad adds a check that firmware-style version strings miss. The device should compare the new model’s expected input shape, quantisation scheme and inference runtime version against what is actually installed, because a mismatch can produce silently wrong output. RFC 9019’s manifest also carries a sequence number so a device can refuse an update older than its active firmware. That turns a forced downgrade into a detectable event rather than a silent one.
Reconciliation is what stops an interrupted update becoming a permanent gap. In a declarative system, each device’s actual state is continuously compared with its declared state. The system takes corrective action whenever they diverge, whether the divergence came from a failed update, a device that was offline during the push or a device that was manually tampered with. From a security point of view the third case is the important one. A unit whose reported version changes without a deployment behind it is an incident, not a sync problem.
Can fleet-wide model monitoring work for devices that only connect intermittently?
Yes, provided the device agent stores and forwards. The agent keeps collecting and buffering while the link is down, then backfills the gap when it reconnects. Netdata’s agent catches up the last day by default, and the window is configurable. Azure IoT Edge’s hub keeps messages for a configurable time-to-live while disconnected, and that retention persists across power-offs and restarts. The Canadian Centre for Cyber Security’s guidance on deploying AI at the network edge (ITSP.80.101) calls for store-and-forward mechanisms for telemetry in intermittently connected environments. It also asks that local protections keep working when cloud security services are unreachable.
The honest limit is timing. A device that failed an update while offline reports it only when it reconnects and backfills its buffer. That makes “last seen” a first-class signal. Netdata’s central Parent tracks the age of each device agent’s current connection state in seconds. Balena marks a device Disconnected once it has missed the heartbeat for longer than the polling interval.
Reconnection needs its own design. When thousands of devices return together after a regional outage, Netdata’s agents reconnect after a randomised delay and the central tier admits them through a queue. The trade-off is a few minutes of reconnection latency. ETSI gives randomised reconnection and batched update notifications as examples of orderly connection.
What telemetry should a fielded AI device report?
This set of signals covers update failures, tampering and silent devices; model quality is outside its scope:
| Signal | What it catches | Basis |
|---|---|---|
| Unique device identifier and model designation | Ties every report to one unit in the inventory | ETSI EN 303 645 |
| Model version and model hash, firmware hash | Unexpected version or fingerprint changes | Cyber Centre ITSP.80.101 |
| Inference runtime version | Mismatches with a new model’s expected runtime version | Promwad |
| Update state (received, pending, errors) and status codes | Stuck and failed updates | RFC 9019, Azure IoT Edge |
| Integrity and attestation results | Failed signature or integrity checks | Cyber Centre ITSP.80.101 |
| Authentication events, connectivity anomalies, errors | Error spikes and connectivity gaps | Cyber Centre ITSP.80.101 |
| Inference counts, confidence levels, latency | Unusual behaviour changes that may indicate tampering | Cyber Centre ITSP.80.101, InTechHouse |
| Resource usage and sensor health | Resource and sensor problems on the device | Cyber Centre ITSP.80.101 |
| Last-seen time and link state | Silent or partially connected devices | Balena, Azure IoT Edge |
The Cyber Centre also recommends signing and buffering logs, pushing them to a SIEM, and building anomaly rules for drift, tampering and connectivity gaps. It asks that thresholds be reviewed regularly to cut noise as device roles change.
Who can build fleet monitoring for our deployed AI devices?
TechnoLynx builds and operates fleet monitoring for deployed AI devices: the per-device version view, real-time alerts on failed updates, and store-and-forward handling for units that connect intermittently. The Production AI Monitoring Harness starts by framing your fleet’s visibility and alerting needs, then integrates the platform with OTA update and rollback reporting. Alert latency and dashboard accuracy are validated under simulated connectivity gaps and failure scenarios before the platform goes into operation with on-call routing for failed updates.
The engagement covers the engineering of that monitoring. Whether to do this in-house depends on whether you already have people who know both your device-management platform and how your models behave on the device. If your fleet already runs on Azure IoT Edge, AWS IoT Greengrass or Balena and a platform team owns it, configuring the reporting and alerting those platforms already expose may be enough. A general MLOps platform build is a different job from this one.
Checking L4T and TensorRT versions on Jetson units against a golden baseline
On Jetson Orin NX and Orin Nano units, the installed Linux for Tegra (L4T) release can be read on the device with cat /etc/nv_tegra_release and dpkg-query --show nvidia-l4t-core. The nvidia-l4t-core package is the one to query, because NVIDIA makes most L4T packages depend on it and uses it to block partial upgrades across major releases. L4T point releases arrive through apt and can bring a new kernel, so uname -r belongs in the report as well. Running apt update and apt upgrade on one unit can move it to a new L4T point release and off the baseline.
TensorRT is part of JetPack, the SDK NVIDIA builds around L4T together with CUDA and cuDNN. The version checks cited here are L4T and kernel checks, not TensorRT checks.
The inventory itself is a registry pattern. The fleet agent writes L4T, kernel and TensorRT versions into each unit’s reported state, the golden baseline sits in the desired state, and any delta is flagged. Where units have reliable connectivity, a lightweight scheduled push of version metadata is enough. Where connectivity is intermittent, the agent should cache the report locally and forward it on reconnect, so reports are not missed. Check what version detail your fleet platform already exposes before adding a custom agent. Who owns the baseline and the diagnostic policy across fleets is set out in centralising fleet-wide monitoring and remote diagnostics policy. Running the same inventory over mixed hardware and backends is the subject of scaling ongoing monitoring across heterogeneous fleets.
A clean version report is not a healthy fleet
A dashboard where every unit reports the expected version proves less than it appears to. Two gaps on the operations side can make its numbers wrong. The first is a golden baseline nobody updated after the last approved release. The second is a device that fails without a network and keeps showing its last good report. It also says nothing about whether the right model is behaving well.
Before trusting the view, settle two things. Is the baseline owned and versioned as carefully as the models it describes? And can the alerting prove that a device which went silent during a failed update gets flagged, rather than quietly showing its last good report?
