OTA signing verification on a production device proves one thing: the image was signed by a key the device trusts and has not been altered since. It does not prove that anyone approved that build for release. This article sets out four checks that run continuously, not once: statistically grounded sampling of production against the dossier, a scripted log-to-dossier comparison, an honest treatment of hardware that cannot attest, and a traceability matrix with a named owner and defined update triggers.
Per-device identity keys are typically injected during manufacturing test, often at a contract manufacturer’s facility, and a failure at that step can compromise every device from that production run. A platform shipped with imprint keys instead of production root keys is effectively running on well-known default keys.
How many units, from how many batches, should we check against the dossier?
Set the sample sizes from an established lot acceptance sampling scheme, sample every batch rather than a few chosen ones, and sample more heavily on the first batches from any new line or contract manufacturer. The defensible move is to borrow acceptance sampling from manufacturing quality practice and state that transfer in the dossier itself.
For inspecting the proportion of nonconforming items in a lot, ISO 2859-1 is the most common sampling standard in industry. A single sampling plan is the pair (n, c): draw n units, and reject the lot if more than c of them are nonconforming. Plans are set up with reference to two quality levels. The acceptable quality level (AQL) is the defect rate the producer wants accepted with high probability, and the lot tolerance percent defective (LTPD) is the defect rate the consumer finds unacceptable. Each has a risk attached: producer’s risk is the probability of rejecting a lot at the AQL, consumer’s risk the probability of accepting a lot at the LTPD, and the NIST/SEMATECH handbook gives typical values for both between 0.2 and 0.01. The operating characteristic curve, which plots acceptance probability against the lot’s defect rate, is the primary tool for displaying and investigating a plan’s properties.
One caution matters more than the rest. A 2024 analysis of ISO 2859-1 plans under the JCGM 106:2012 framework found that, for a lot of 1,200 units, the small sample sizes the standard recommends for large AQLs leave consumer’s risk close to 1 when a lot is only marginally accepted. For a security property, treat your customer and the auditor as the consumer. Set a tight AQL for “secure boot enabled with the production key” and accept the larger samples that follow, because one fleet-wide misconfiguration costs more than inspection time.
FDA’s drug CGMP regulations require that samples represent the batch and that the sampling plan result in statistical confidence. FDA’s quality system regulation §820.250(b) requires sampling plans to be written, based on a valid statistical rationale, and reviewed when changes occur. FDA’s process validation guidance expects heavier sampling during process performance qualification, continued through process verification until routine sampling levels can be set.
Two failure shapes drive the batch count. A systematic fault, such as a programming station loaded with the wrong key hash, affects a whole batch, and even a small per-batch sample catches it, provided every batch is sampled. A sporadic per-unit fault, such as an interrupted OTP write, needs larger samples within each batch, because a clean small sample says little about rare defects. Sampling across many batches addresses the first shape; sample size within a batch addresses the second.
Per sampled unit, the check should cover the settings the dossier claims: the public key hash fused into one-time-programmable memory, the secure boot enable flag, the anti-rollback setting and the debug interface state. For its Series 2 devices, Silicon Labs documents a security readconfig command, run against a device’s serial number, that checks its secure boot configuration. Keys used for update integrity must be unique per device under ETSI EN 303 645, so the check should also confirm that two sampled units do not share a critical security parameter.
Worked example: what a clean sample does and does not tell you
The probability that a sample of n units contains no misconfigured unit is 0.99^n. For n = 50 that is about 0.61, so a clean sample of 50 would miss a 1% defect rate more often than not; for n = 300 it falls to about 0.05. The arithmetic shows why sampling gives confidence, never certainty, and why the plan has to be redone whenever the line, the CM or the provisioning process changes. The comparison also depends on secure boot bring-up data being recorded the same way in every batch; a field one batch omits cannot be checked for that batch.
Automating OTA signing verification against the dossier each cycle
Yes, the comparison can be scripted, as long as the dossier exists as structured reference values rather than prose, and every automated flag carries a link back to the record that raised it. The mechanism is the same one attestation verifiers use: evidence collected from the device is compared with reference values that represent its intended configured state. NIST SP 800-128 describes configuration monitoring as checking systems against agreed baseline configurations, and notes that automation capabilities such as SCAP can carry out those assessments.
In practice the dossier turns into a reference set keyed by SKU and firmware version: expected root key hash, boot stage versions, secure boot and rollback flags, debug state, and the expected signing key for each release. The tool ingests per-unit logs keyed by serial number and batch ID, compares each field with the reference set for that unit’s SKU, and emits one flag per deviation. Each flag records the log entry, its timestamp, the batch ID, the unit serial and the dossier version it was compared against; without those fields a flag is an opinion, and a reviewer cannot reproduce it. The logs themselves need integrity protection. The OCP firmware signing guidance says signing-server audit logs should be tamper evident, with capacity for at least a 90-day window, and NIST SP 800-193 says logged firmware events should be recorded with assurances of integrity and access control.
Low-volume runs do not need the tooling: a documented manual spot check with the same audit trail is defensible, and the investment pays off once sample volume makes manual diffing a recurring cost.
ETSI EN 303 645 V3.1.3 (2024) defines device requirements but no testing or certification method, leaves documentation requirements to the testing regime, and points to ETSI TS 103 701 as a way to assess conformance. Germany’s BSI provides an assessment template that a manufacturer can use for self-assessment and a test laboratory can fill with individual test-case results. EN 303 645 also requires a recorded justification for each recommendation considered not applicable or not fulfilled, with Table B.1 as a schema to record them. Mapping your reference set to those provision identifiers makes each automated flag traceable to the clause it concerns.
Factory sampling has a field-side complement. EN 303 645 notes that telemetry from many devices lets a manufacturer notice updates failing authenticity checks. A spike of rejected updates on one batch is a signal worth correlating with that batch’s sampling results.
Our gateways can’t do full remote attestation: what can we push over OTA?
Whether a deployed gateway can do real remote attestation depends on its hardware, not only its firmware. Attestation needs a secret key that no software on the device can read except the attestation code, and that implies some level of hardware support. TechnoLynx audits deployed gateway hardware revisions against those attestation requirements, designs and implements the OTA-deliverable secure boot plus anti-rollback fallback for units that can take it, runs fault-injection and glitching tests against that fallback to characterise what risk remains, and produces the retrofit-versus-replace inventory split for rollout planning. TechnoLynx’s IoT and embedded engineering work on this starts with the hardware revision audit, because every later decision depends on it.
What real attestation needs from the hardware
Remote attestation comes in three forms: software-based, which relies on strict timing measurements; hardware-based, which runs attestation in a tamper-resistant component; and hybrid, which uses a hardware/software co-design without complex hardware. In the IETF’s TPM-based remote integrity verification draft, a verifier distrusts a device if the quote signature is wrong, the nonce does not match, the signed PCR values cannot be reconstructed from the event log, important log entries differ from known-good reference values, or the evidence is older than the appraisal policy’s freshness threshold. RFC 9334 names the roles: the attester produces evidence, the verifier appraises it, and the relying party decides whether to consider the attester trustworthy. A gateway with no protected attestation key cannot produce evidence that survives the first of those checks, whatever firmware it runs.
Between full hardware attestation and nothing sits a narrower band. Hybrid schemes build on a minimal trust anchor, typically ROM holding the attestation code and key with a memory protection unit limiting access to them. Software-based attestation was surveyed in 2021 for legacy and resource-constrained IoT devices without specialised hardware. Both are worth evaluating per hardware revision, but they are weaker guarantees and should be described that way to customers.
The OTA fallback and its limits
For units with a ROM root of trust and tamper-resistant memory for a counter, the fallback is secure boot with anti-rollback. A monotonic counter in tamper-resistant memory makes the device reject any image at or below the current version even when its signature is valid, which closes the downgrade-to-a-vulnerable-signed-build path. Read the vendor fine print. Silicon Labs notes that its anti-rollback counter does not prevent flashing an older signed bootloader image to the device by other means, which is why it pairs secure boot with page-lock options that block re-flashing validated pages outside the secure engine.
Researchers used electromagnetic fault injection, with no hardware modification, to bypass signature checks and execute unsigned code on production gateway ECUs from Volkswagen and BMW. Their mitigation was to stop unauthenticated flash executing early in boot with the MPU, a feature present on the tested controllers but not used by the manufacturers. Fault-injection testing of the fallback on each SoC characterises the residual risk a physical attacker leaves.
Fused keys are a second gap: they cannot be replaced in the field. When the verification key is fused at manufacture, a stolen signing key can be unrecoverable for units already in the field. Revocation alone is often not enough, because embedded devices may not fetch revocation data or support certificate rollover. Crypto-agile designs pre-provision a successor key so signing can switch to it. Whether your installed base has one is a question the hardware audit should answer.
Retrofit or replace: a triage by hardware revision
| Hardware revision has | What is achievable | Action |
|---|---|---|
| TPM or equivalent hardware holding a protected attestation key | Remote attestation with TPM-signed, nonce-bound evidence | Enable attestation and appraise against reference values |
| ROM root of trust and rollback-capable storage, no protected attestation key | Secure boot plus anti-rollback via OTA; hybrid attestation only if ROM/MPU support it | Push the OTA fallback, lock validated flash pages and debug, and test it under fault injection |
| No hardware-protected key storage | No attestation key that software cannot read, whatever the firmware | Classify as replace in the retrofit-versus-replace split |
The split between the rows comes from the hardware audit, not an estimate. EN 303 645 requires the model designation to be recognisable on the device or through an interface. NIST SP 800-128 treats automated discovery as generally more effective than manual collection for keeping component inventories accurate. Query deployed units for their hardware revision rather than trusting shipment records, then prioritise replacement by the exposure of the non-retrofittable population. The design-time side of this, choosing silicon so the problem does not recur, is covered in anchoring OTA signing and attestation to a hardware root of trust.
Signed is not authorised: explaining the gap to leadership and auditors
The one-sentence version for a board: a valid signature proves which key signed a build, not that anyone approved that build to ship. NIST SP 800-193 draws the line exactly: authentication verifies an image’s source and integrity, while authorisation is the permission to perform the update. RFC 9019 makes the same distinction for IoT updates, noting that not every entity able to sign a manifest has the same permissions. Rejecting unsigned firmware stops images not signed by a trusted key. It leaves open every path where a trusted key signs something it should not have: a mistake, weak governance, an insider, or an intrusion into the build or signing systems.
Two incidents that make the point
Kaspersky’s analysis of Operation ShadowHammer describes trojanised ASUS Live Update installers distributed between June and November 2018. They were signed with legitimate “ASUSTeK Computer Inc.” certificates and hosted on ASUS’s own update servers, which Kaspersky gives as part of why they went undetected for so long. Over 57,000 Kaspersky users installed the backdoored version.
The second case concerns firmware signing keys. BleepingComputer reported that source code leaked after a breach at MSI contained image signing private keys for 57 MSI products and Intel Boot Guard private keys for 116 MSI products. Because the matching public keys are believed to be built into the hardware, Boot Guard can no longer be trusted on affected devices if those keys cannot be changed. For leadership, the lesson is that a valid signature from a legitimate key was no defence: ShadowHammer’s updaters carried legitimate signatures, and the leaked MSI keys let an attacker sign firmware that passes Boot Guard. The failure sits upstream, in who could cause a signature to be made.
The control that closes it, and the evidence that shows it is closed
Closing the gap needs a release-authorisation control separate from authentication. The options are multi-party M-of-N approval so that no single engineer can sign and ship, NIST’s recommendation to consider two-person approval of code for signing, or device-side policy that rejects updates unless signed by both the firmware author and the device operator. The ASSURED research architecture goes further and has the OEM sign a separate authorisation token, carrying constraints such as the applicable device model, inside each update envelope. The backend side, telling “validly signed” apart from “authorised for release”, is the subject of separating valid signatures from release authorisation. Limiting the damage when a signed bad image does get through is covered in hardening firmware against signed-but-malicious updates.
An evidence set that keeps them apart contains:
- A signing event log tying every released firmware version to its key, approvers and timestamp.
- Approver records showing the release was approved before signing and deployment.
- HSM validation certificate references for the modules protecting signing keys, since “we use an HSM” without a certificate reference is not evidence.
- Key lifecycle records: generation date, rotation history and revocation status for every key, including the offline root.
- A tested demonstration that the device rejects a validly signed but outdated image, not just a design claim that it should.
- Signing request logs recording requester, project, hash signed, signature and public key, reviewed regularly for anomalous key use.
Add one negative test aimed at this exact gap: sign a deliberately unapproved build through the normal pipeline, present it to sampled production units, and record whether the authorisation control rejects it. A passing signature check alongside a rejected installation is the clearest proof that the second control exists.
Keeping the traceability matrix current when chips, keys and SKUs change
The traceability matrix stays current only if specific events force an update, a named person owns each update, and the underlying evidence is regenerated by the pipeline rather than recalled by memory. NIST SP 800-128 frames this as configuration change control: changes are formally identified, reviewed, analysed for security impact, tested and approved before implementation, with each role’s responsibilities and authority set out at every step. Firmware is itself a configuration item under that guidance, and the information kept for a configuration item includes ownership and version numbers.
| Trigger | What goes stale | Regenerate | Sign-off |
|---|---|---|---|
| Supplier chip revision change | Root of trust features, OTP layout, attestation capability row | Hardware audit entry, sampling reference set, sampling plan review | Hardware owner for the SKU |
| OTA signing key rotation | Signing log lineage, expected key per release, key lifecycle records | Reference set key fields; rollback and rejection tests | Signing service owner |
| New firmware version | Version floor, boot stage hashes, approver record | Reference values and signing event entry | Release owner |
Key rotation is easy to miss as a trigger, because nothing physical changes. The OCP firmware signing guidance cites NIST SP 800-57 as suggesting an originator usage period of one to three years for private signature keys, and EN 303 645 requires secure management processes across the whole lifecycle of critical security parameters. Separating an offline root key from a rotatable online signing key means rotations happen on the online key while the hardware root stays tied to the root’s lineage. Record each rotation as a formal change event, or every evidence item referencing the old key goes stale without anyone noticing.
Name the owner of the evidence pack for each SKU and firmware release. The worked secure-update evidence example in the Supply Chain Security Handbook assigns retention ownership to the supplier release owner and the buyer’s lifecycle monitoring owner, and records excluded configurations with a dated follow-up decision. Automation reduces the manual work to ownership decisions and sign-off, but it does not remove the need for someone accountable. SP 800-128 also asks that older approved baselines be kept for review or rollback, so the matrix should be versioned, never overwritten.
Build this evidence with the same pipeline that builds and signs the firmware, rather than assembling it by hand before an audit. That evidence is an engineering output, not certification or regulatory sign-off. How to package those artefacts for a specific auditor or procurement reviewer is covered in building a defensible evidence pack for audit and procurement review.
Each check catches a different failure
None of these checks substitutes for another. Production sampling catches a line that drifted from the dossier but says nothing about whether a release was approved. The authorisation control catches a rogue signed build but cannot see a batch programmed with imprint keys. Attestation, where the hardware allows it, tells you what a deployed unit booted, and for gateways that cannot attest the honest position is a documented fallback with tested limits, not a claim of equivalence. The question left to settle is which of the four your current evidence actually covers, and which one an auditor would find missing first.
