22 Safe Over-the-Air Updates
22.1 Start With the Story
An over-the-air update demo can succeed once and still be unsafe. The important story asks what happens when the package is wrong, the download stops, the device reboots halfway, the new firmware cannot report health, or the rollback path is not understood by support. OTA evidence is about recovery, not just delivery.
Read this chapter as a rehearsal for change under stress. A prototype update path should record image identity, compatibility checks, health confirmation, rollback behavior, and the conditions that force another test before release.
22.2 OTA as Controlled Device Path
An OTA system is safe only when the team can control which package reaches which device and can prove what happened after installation. The download is only one step in that path.
Inspect Update Question and device can accept in Figure 22.1 for ota as controlled device path. At the decision point in ota as controlled device path, follow the change from Update Question to device can accept on it. continue/pause/abort identifies the later check.
Read Update Question with device can accept in Figure 22.1 for ota as controlled device path. Review it by keeping Update Question, device can accept, and continue/pause/abort as separate entries. Combining Update Question with device can accept hides accountability. Carry Update Question into the evidence for ota as controlled device path.
A reviewable prototype names the package, checks device eligibility, installs through a recoverable state machine, observes first-boot health, and keeps a rollback route available until the update is confirmed.
For a cold-room monitor, the package record might name an ESP32-C3 build, source revision, partition table, dependency record, package hash, signature or approval record, target hardware revision, current-version range, and expected health checks. A device with the wrong sensor board, too little storage, weak battery, incompatible bootloader layout, or unsupported current version should reject the package before writing anything. That rejection is useful evidence, not a failed demo.
The update path should keep local behavior visible while the package moves through download, verification, staging, reboot, health check, confirmation, and rollback. If the gateway drops during download, the device should continue local sensing and report retry state. If the new image boots but cannot read the SHT31 probe, restore the previous image or hold the device in a safe recovery state with a clear reason. If the update succeeds, report package id, firmware version, health checks, queue state, and last fault so the rollout can continue deliberately.
Even a small prototype should separate the release owner, package store, rollout rule, device update agent, and health telemetry. That separation lets a reviewer challenge the update path without guessing whether a failure came from build content, targeting policy, network delivery, device storage, boot confirmation, or post-update behavior.
This is why OTA belongs in the prototype phase. It exposes package identity, device eligibility, field recovery, and fleet visibility before the team depends on remote updates to rescue deployed devices. A simple local update fixture is enough for early evidence if the record already shows the safety controls a later device-management platform must preserve.
22.3 Confirm Health Before Success
Do not mark an update successful at reboot. Mark it successful after the device proves the capabilities that matter for the product.
Start with a lab fixture that can offer the right package, an incompatible package, a corrupt package, and an interrupted download. For ESP-IDF, record OTA partition slots, esp_ota_mark_app_valid_cancel_rollback, image size, project version, secure boot or signing choice, and the health signal used before confirmation. For MCUboot-based Zephyr or Nordic builds, record slot layout, image version, signing key identity, boot status, swap state, and how the firmware confirms the new image. For Linux gateways, record the systemd unit, package source, checksum, service health check, and rollback command.
Make the rollout gate numerical enough to act on. A canary batch might continue only if devices report package id, new version, successful sensor read, queue recovery, gateway publish, stable reset count, and battery or power state within the expected window. It should pause if missing reports, repeated retries, unexpected resets, rollback reasons, or support tickets cross the threshold. The point is to make continue, pause, abort, and rollback ordinary outcomes of the same review path.
- Before install: verify model, hardware revision, current version, package signature, storage space, power state, and rollback slot.
- During install: report download progress, verification result, write result, reboot reason, and any interruption.
- After first boot: confirm sensor access, actuator safety state, queue recovery, network reconnect, sleep behavior, and version telemetry.
- Before rollout expansion: compare update failure rate, rollback rate, crash/reset rate, battery impact, and support reports against the gate.
Keep secrets out of the device and records. Public verification material, package hashes, package ids, and approval references can be recorded. Private signing keys, update-control credentials, and device tokens need protected storage and clear ownership. A prototype that hard-codes update credentials may work in the lab while teaching the wrong release habit.
22.4 Rollback Is Update Logic
Rollback must be designed into the update state machine. A device needs to know when a new image is pending, when it is confirmed, and when it should return to the previous image because boot, health, or communication checks failed.
The hard cases are usually partial: power loss during write, valid firmware on the wrong hardware revision, an update that boots but cannot read a sensor, or a fleet segment that passes the lab gate but fails on weak networks. OTA evidence should make those partial failures visible before the rollout reaches the whole fleet.
Under the hood, the update agent needs persistent state that survives reboot. It should record attempted package id, previous version, candidate version, slot or partition, verification result, reboot reason, health-check status, confirmation state, retry count, and rollback reason. If power fails during a write, the bootloader and agent should know whether to resume, reject, or keep the previous image. If the new image boots but never confirms health, the boot path should return to known-good behavior or hold the device in a controlled recovery state.
Compatibility is also a firmware boundary. A package built for a larger flash layout, different sensor board, changed calibration schema, incompatible radio stack, or new storage migration can boot and still be wrong for a device. The update record should state target product, hardware revision, current-version range, bootloader or partition scheme, configuration migration, and minimum power or storage conditions. The device update agent should check those facts before staging the package.
Telemetry closes the loop. A rollout dashboard or simple review table should distinguish rejected, downloaded, verified, staged, booted, confirmed, retried, rolled back, missing, and unresolved devices. Without those states, a fleet can look quiet while some devices are stuck in retry loops, have reverted to an old image, or stopped reporting during a weak-network update window.
22.4.1 Why the Bootloader Is the Risk Surface
A bootloader is the first code a microcontroller runs after reset. It initializes memory and peripherals, then hands control to the main application — and on most parts it is also the code that accepts a new image and writes it to flash. Fundamental chip behavior such as clock source, clock rate, watchdog timer, and brown-out detection is normally set through separate non-volatile configuration fuses, not the application binary. A package that gets a target’s fuse or partition assumptions wrong can leave a device unable to boot before the transferred payload is even considered.
This is why bricking is not a vague fear. If an update path writes directly over the only copy of the bootloader or active image and the transfer is interrupted, corrupted, or aimed at the wrong hardware revision, there may be no known-good code left to boot into. A device in that state is unrecoverable over the air, and lost outright if it is physically inaccessible. The recoverable-slot and boot-confirmation requirements earlier in this section exist to prevent exactly that: the update agent stages the new image in an inactive slot or partition, and the bootloader only switches which image it boots from after health checks confirm the new one.
Two established OTA architectures make different trade-offs with this risk. An on-chip bootloader that handles the update itself can reuse the device’s existing radio or serial hardware and keeps size and cost down, but a corrupted bootloader on that same chip can brick the device with no fallback path. A dedicated off-chip programmer — a companion chip or module that owns the update protocol separately from the application processor — keeps the update path isolated from application firmware risk, at the cost of extra size, complexity, and unit cost. Record which architecture a target device uses in target_bootloader_or_partition_scheme; it changes what “recoverable” means for that hardware.
22.5 Learning Objectives
By the end of this chapter, you will be able to:
- Treat OTA as a controlled update path rather than a single firmware feature.
- Define update package records with version, target, compatibility, integrity, authenticity, and rollback expectations.
- Separate build, signing, distribution, rollout policy, device agent, health checks, and telemetry responsibilities.
- Review update safety controls such as protected transport, signed packages, anti-rollback rules, boot slots, power and space checks, and post-update health confirmation.
- Plan staged rollout gates and handoff records that make a prototype update safe to repeat.
22.6 Prerequisites
This chapter builds on:
- Managing Libraries and Version Control for IoT Prototypes, where dependency and build inputs are recorded.
- Testing and Debugging IoT Software Prototypes, where normal runs, fault runs, and regression records are created.
- Choosing Architecture Patterns for IoT Software Prototypes, where update, storage, communication, and observability responsibilities are separated.
22.7 OTA Is a Lifecycle Evidence Path
An OTA feature is not complete when a device can download a file and reboot. The review question is broader: can the team prove that the right package reaches the right device, at the right time, with enough health evidence to confirm or roll back the update?
Start with four evidence questions:
A device that reboots into a new version has not necessarily accepted a safe update. Confirmation should wait until the post-update health checks prove the device can still sense, decide, communicate, store, sleep, and recover in the ways the prototype requires.
22.8 OTA System Boundaries
An OTA system crosses more boundaries than device firmware. A reviewable prototype names each boundary and records which evidence it owns.
Inspect Build Record and Rollout Policy in Figure 22.2 for ota system boundaries. Before accepting ota system boundaries, follow the change from Build Record to Rollout Policy on it. Use confirm or rollback as the boundary.
Read Build Record with Rollout Policy in Figure 22.2 for ota system boundaries. Audit it by keeping Build Record, Rollout Policy, and confirm or rollback as separate entries. Both Build Record and Rollout Policy need evidence. Carry Build Record into the evidence for ota system boundaries.
Use these boundaries in the review:
The prototype can start with a simple update server or local fixture, but the record should already name the package, target devices, expected safety controls, and health signals. That prevents the demo path from becoming an unreviewed production path.
22.9 Package and Compatibility Records
Every update needs a package record that can be checked by both people and devices. The record does not have to use a specific platform format, but it should make compatibility and rejection rules explicit.
package_id=
package_type=full-image | delta | configuration | model | script | data
target_product=
target_hardware_revision=
target_bootloader_or_partition_scheme=
current_version_range=
new_version=
source_revision=
dependency_record=
build_command=
package_size=
package_hash=
signature_or_approval_record=
minimum_power_or_storage_state=
expected_health_checks=
rollback_available=yes | no | limited
release_owner=
release_notes=Review the record before testing the update. If the package target is vague, the device may accept a build meant for another board, another partition layout, another sensor configuration, or another dependency set.
22.10 Update Safety Controls
Safety controls make a bad update detectable and recoverable. The right set depends on prototype risk, but a review should not skip the question.
Inspect OTA SERVER and Deploy in Figure 22.3 for update safety controls. Before accepting update safety controls, follow the change from OTA SERVER to Deploy on it. connectivity, and application metrics limits the claim.
Read OTA SERVER with Deploy in Figure 22.3 for update safety controls. Review its phases with OTA SERVER establishing the start, Deploy changing maturity, and connectivity, and application metrics marking commitment. The later connectivity, and application metrics stage does not erase the OTA SERVER gate. The conclusion in update safety controls now has a named boundary.
Review these controls:
Public verification material can live on the device; private signing material should not. Keep credentials, signing keys, and update-control secrets out of firmware source and out of deployed devices unless the design explicitly requires and protects them.
22.11 Device Update State Machine
OTA failure handling is easier to review when the device has explicit update states. The state machine should make rejection and recovery as visible as success.
Inspect Idle and Reboot in Figure 22.4 for device update state machine. At the decision point in device update state machine, put Idle and Reboot into the same reading of it. The figure ties this to no install.
Read Idle with Reboot in Figure 22.4 for device update state machine. Move through it starting at Idle, crossing Reboot, and closing on no install. A failure at Reboot changes the route from Idle. This supplies device update state machine with a concrete retest point.
Use clear states:
22.12 Health Checks and Rollback Evidence
A health check is not “the device turned on.” It is a small, targeted proof that the update did not break the prototype’s essential behavior.
Common health checks include:
Rollback evidence should record the failed check, previous version, attempted package, device state, and whether the device returned to service. A rollback without a reason is hard to distinguish from an ordinary reboot.
22.13 Rollout Gates
Rollout is a decision process, not a single button. Each gate should have a small enough blast radius that the team can learn, pause, and recover.
Inspect BUILD PHASE and Production in Figure 22.5 for rollout gates. Before carrying rollout gates forward, locate BUILD PHASE beside Production on it. The figure ties this to and Fix.
Read BUILD PHASE with Production in Figure 22.5 for rollout gates. Inspect its stages from BUILD PHASE through the later Production milestone to and Fix. The later and Fix stage does not erase the BUILD PHASE gate. Reopen rollout gates whenever Production changes.
Review gates before a wide update:
An OTA review is healthier when pausing or aborting is a normal gate outcome, not an emergency exception. Define the stop rule before the rollout begins.
22.14 Cold-Room OTA Review
A team is preparing an update for a cold-room monitor prototype. The update changes retry behavior after gateway loss and adds a compact post-reset fault record. The testing chapter already proved the new retry state on a bench board and an integration rig.
22.14.1 Update Questions
The OTA review asks:
First, Which devices should accept this package? Next, Can the device reject a package built for the wrong board or current version? Then, Can it download and stage the package without stopping local temperature monitoring? After that, Can the new firmware prove that probe read, alarm state, local queue, gateway report, and post-reset fault record still work? Finally, Can the device return to the previous image if health checks fail?
22.14.2 Evidence Runs
The team records these runs:
22.14.3 Handoff Decision
The package is allowed into a canary batch only after the team can reproduce both a successful confirmation and a forced rollback. The broad rollout remains blocked until canary health telemetry confirms that update state, local sensing, gateway reporting, and fault records behave as expected.
prototype=cold-room-monitor
update_question=retry-state update must not stop local monitoring
package_id=
source_revision=
target_hardware_revision=
current_version_range=
new_version=
normal_update_evidence=download, verify, stage, reboot, health confirm
fault_evidence=incompatible package, interrupted download, forced health failure
rollback_evidence=previous image restored and rollback reason reported
canary_gate=success, retry, rollback, and missing-report signals reviewed
handoff=best-practices chapter should keep update records in release review22.15 OTA Handoff Record
Every reviewed OTA path should leave a handoff record for future releases:
release_name=
package_id=
package_type=
source_revision=
dependency_record=
target_devices=
excluded_devices=
eligibility_rules=
safety_controls=
lab_evidence=
fault_evidence=
rollout_gates=
pause_or_abort_rules=
health_signals=
rollback_path=
unresolved_devices=
next_rerun_condition=
review_owner=Rerun the OTA path whenever the bootloader, partition layout, update agent, package format, signing process, communication dependency, storage migration, or health-check criteria changes.
22.16 Knowledge Check
22.17 Common Failure Patterns
22.18 Summary
First, OTA review starts with package identity, device eligibility, recovery path, and rollout evidence. Next, Build, signing or approval, distribution, rollout policy, device agent, and health telemetry are separate boundaries. Then, Safety controls should cover authenticity, integrity, compatibility, anti-rollback policy, power and storage gates, recoverable boot, health confirmation, and telemetry. After that, A device update state machine should make rejection, retry, confirmation, and rollback visible. Finally, Staged rollout gates let the team learn, pause, abort, and record before a wider update becomes hard to reverse.
22.19 Key Takeaway
OTA design is a reliability feature: include identity, integrity checks, rollback, staged rollout, telemetry, and a recovery path before relying on remote updates.