22 Safe Over-the-Air Updates

22.1 Start With the Story

This section: Start With the Story — An over-the-air update demo can succeed once and still be unsafe.

An over-the-air update demo can succeed once and still be unsafe. The important story asks what happens when the package is wrong, the download stops, the device reboots halfway, the new firmware cannot report health, or the rollback path is not understood by support. OTA evidence is about recovery, not just delivery.

Read this chapter as a rehearsal for change under stress. A prototype update path should record image identity, compatibility checks, health confirmation, rollback behavior, and the conditions that force another test before release.

22.2 OTA as Controlled Device Path

This section: OTA as Controlled Device Path — An OTA system is safe only when the team can control which package reaches which device and can prove what happened after installation.

An OTA system is safe only when the team can control which package reaches which device and can prove what happened after installation. The download is only one step in that path.

Figure 22.1 applies an update gate to the cold-room monitor’s retry change.

+

The rollout path in Figure 22.1 begins with a narrow retry change for a cold-room monitor. A package record names the ESP32-C3 build, hash, and approval. Eligibility checks hardware revision, current version, signature, storage, power, and rollback slot before any write. Lab, canary, pilot, and wider gates then use first-boot health: sensor read, alarm state, local queue, gateway report, and post-reset fault record. Reboot is not confirmation. If the new image cannot read the probe, restore the prior image or hold a safe state and record the failed check. Expansion depends on health, rollback, crash, battery, and support evidence at each gate.

A reviewable prototype names the package, checks device eligibility, installs through a recoverable state machine, observes first-boot health, and keeps a rollback route available until the update is confirmed.

For a cold-room monitor, the package record might name an ESP32-C3 build, source revision, partition table, dependency record, package hash, signature or approval record, target hardware revision, current-version range, and expected health checks. A device with the wrong sensor board, too little storage, weak battery, incompatible bootloader layout, or unsupported current version should reject the package before writing anything. That rejection is useful evidence, not a failed demo.

The update path should keep local behavior visible while the package moves through download, verification, staging, reboot, health check, confirmation, and rollback. If the gateway drops during download, the device should continue local sensing and report retry state. If the new image boots but cannot read the SHT31 probe, restore the previous image or hold the device in a safe recovery state with a clear reason. If the update succeeds, report package id, firmware version, health checks, queue state, and last fault so the rollout can continue deliberately.

Even a small prototype should separate the release owner, package store, rollout rule, device update agent, and health telemetry. That separation lets a reviewer challenge the update path without guessing whether a failure came from build content, targeting policy, network delivery, device storage, boot confirmation, or post-update behavior.

This is why OTA belongs in the prototype phase. It exposes package identity, device eligibility, field recovery, and fleet visibility before the team depends on remote updates to rescue deployed devices. A simple local update fixture is enough for early evidence if the record already shows the safety controls a later device-management platform must preserve.

22.3 Confirm Health Before Success

This section: Confirm Health Before Success — Do not mark an update successful at reboot.

Do not mark an update successful at reboot. Mark it successful after the device proves the capabilities that matter for the product.

Start with a lab fixture that can offer the right package, an incompatible package, a corrupt package, and an interrupted download. For ESP-IDF, record OTA partition slots, esp_ota_mark_app_valid_cancel_rollback, image size, project version, secure boot or signing choice, and the health signal used before confirmation. For MCUboot-based Zephyr or Nordic builds, record slot layout, image version, signing key identity, boot status, swap state, and how the firmware confirms the new image. For Linux gateways, record the systemd unit, package source, checksum, service health check, and rollback command.

Make the rollout gate numerical enough to act on. A canary batch might continue only if devices report package id, new version, successful sensor read, queue recovery, gateway publish, stable reset count, and battery or power state within the expected window. It should pause if missing reports, repeated retries, unexpected resets, rollback reasons, or support tickets cross the threshold. The point is to make continue, pause, abort, and rollback ordinary outcomes of the same review path.

• Before install: verify model, hardware revision, current version, package signature, storage space, power state, and rollback slot.
• During install: report download progress, verification result, write result, reboot reason, and any interruption.
• After first boot: confirm sensor access, actuator safety state, queue recovery, network reconnect, sleep behavior, and version telemetry.
• Before rollout expansion: compare update failure rate, rollback rate, crash/reset rate, battery impact, and support reports against the gate.

Keep secrets out of firmware source and review records. Public verification material, package hashes, package ids, and approval references can be recorded. Keep release-signing private keys off deployed devices; when a device needs Wi-Fi or update credentials or a device-specific token, provision them into protected device storage with clear ownership and rotation. A prototype that hard-codes update credentials may work in the lab while teaching the wrong release habit.

22.4 Rollback Is Update Logic

This section: Rollback Is Update Logic — Rollback must be designed into the update state machine.

Rollback must be designed into the update state machine. A device needs to know when a new image is pending, when it is confirmed, and when it should return to the previous image because boot, health, or communication checks failed.

The hard cases are usually partial: power loss during write, valid firmware on the wrong hardware revision, an update that boots but cannot read a sensor, or a fleet segment that passes the lab gate but fails on weak networks. OTA evidence should make those partial failures visible before the rollout reaches the whole fleet.

Under the hood, the update agent needs persistent state that survives reboot. It should record attempted package id, previous version, candidate version, slot or partition, verification result, reboot reason, health-check status, confirmation state, retry count, and rollback reason. If power fails during a write, the bootloader and agent should know whether to resume, reject, or keep the previous image. If the new image boots but never confirms health, the boot path should return to known-good behavior or hold the device in a controlled recovery state.

Compatibility is also a firmware boundary. A package built for a larger flash layout, different sensor board, changed calibration schema, incompatible radio stack, or new storage migration can boot and still be wrong for a device. The update record should state target product, hardware revision, current-version range, bootloader or partition scheme, configuration migration, and minimum power or storage conditions. The device update agent should check those facts before staging the package.

Telemetry closes the loop. A rollout dashboard or simple review table should distinguish rejected, downloaded, verified, staged, booted, confirmed, retried, rolled back, missing, and unresolved devices. Without those states, a fleet can look quiet while some devices are stuck in retry loops, have reverted to an old image, or stopped reporting during a weak-network update window.

22.5 Learning Objectives

This section: Learning Objectives

By the end of this chapter, you will be able to:

• Treat OTA as a controlled update path rather than a single firmware feature.
• Define update package records with version, target, compatibility, integrity, authenticity, and rollback expectations.
• Separate build, signing, distribution, rollout policy, device agent, health checks, and telemetry responsibilities.
• Review update safety controls such as protected transport, signed packages, anti-rollback rules, boot slots, power and space checks, and post-update health confirmation.
• Plan staged rollout gates and handoff records that make a prototype update safe to repeat.

22.6 Prerequisites

This section: Prerequisites

This chapter builds on:

• Managing Libraries and Version Control for IoT Prototypes, where dependency and build inputs are recorded.
• Testing and Debugging IoT Software Prototypes, where normal runs, fault runs, and regression records are created.
• Choosing Architecture Patterns for IoT Software Prototypes, where update, storage, communication, and observability responsibilities are separated.

22.7 OTA Is a Lifecycle Evidence Path

This section: OTA Is a Lifecycle Evidence Path — An OTA feature is not complete when a device can download a file and reboot.

An OTA feature is not complete when a device can download a file and reboot. The review question is broader: can the team prove that the right package reaches the right device, at the right time, with enough health evidence to confirm or roll back the update?

Start with four evidence questions:

22.8 OTA System Boundaries

This section: OTA System Boundaries — An OTA system crosses more boundaries than device firmware.

An OTA system crosses more boundaries than device firmware. A reviewable prototype names each boundary and records which evidence it owns.

Figure 22.2 separates build identity, trust, rollout policy, and device health.

+

The boundaries in Figure 22.2 give each OTA stage an owner and an artifact. The build record binds source, target, and package; signing or approval establishes the trust decision. Distribution keeps the package available, while rollout policy chooses target devices and pause or abort gates. The device agent checks eligibility, downloads, verifies, writes, and boots the candidate. Health checks confirm it or trigger rollback. Telemetry closes the loop by sending the result to the rollout decision. A download log alone cannot prove that the right device received a trusted image or that local sensing recovered after reboot, so retain evidence across all six stages.

Use these boundaries in the review:

22.9 Package and Compatibility Records

This section: Package and Compatibility Records — Every update needs a package record that can be checked by both people and devices.

Every update needs a package record that can be checked by both people and devices. The record does not have to use a specific platform format, but it should make compatibility and rejection rules explicit.

package_id=
package_type=full-image | delta | configuration | model | script | data
target_product=
target_hardware_revision=
target_bootloader_or_partition_scheme=
current_version_range=
new_version=
source_revision=
dependency_record=
build_command=
package_size=
package_hash=
signature_or_approval_record=
minimum_power_or_storage_state=
expected_health_checks=
rollback_available=yes | no | limited
release_owner=
release_notes=

Review the record before testing the update. If the package target is vague, the device may accept a build meant for another board, another partition layout, another sensor configuration, or another dependency set.

22.10 Update Safety Controls

This section: Update Safety Controls — Safety controls make a bad update detectable and recoverable.

Safety controls make a bad update detectable and recoverable. The right set depends on prototype risk, but a review should not skip the question.

Figure 22.3 shows how server policy and device checks limit an update rollout.

+

The server side keeps the firmware revision, signing or integrity decision, rollout schedule, and success metrics together. A canary group receives the package before wider groups so errors can be observed while exposure is limited. The device must still verify eligibility and package integrity, apply the update, then report health after reboot. Monitoring closes the loop: a successful download is not the same as a healthy application. Keep the prior valid image and the rollback condition clear before expanding the rollout. The figure’s percentage stages are policy examples, not evidence that this prototype has passed them.

Review these controls:

22.11 Device Update State Machine

This section: Device Update State Machine — OTA failure handling is easier to review when the device has explicit update states.

OTA failure handling is easier to review when the device has explicit update states. The state machine should make rejection and recovery as visible as success.

Figure 22.4 follows the device from eligibility to health confirmation or rollback.

+

Idle means the device is still running its current image. Eligibility checks target and policy before a download begins; retry behavior must remain visible if the transfer stalls. Verification checks the package before it is staged in a safe slot. After reboot, the health state tests the behavior needed to confirm the new image. If that check fails, rollback returns to a known-good version rather than treating the reboot as success. Record the transition and its result so the update decision can distinguish a transport failure, a trust failure, and an application failure.

Use clear states:

22.12 Health Checks and Rollback Evidence

This section: Health Checks and Rollback Evidence — A health check is not “the device turned on.” It is a small, targeted proof that the update did not break the prototype’s essential behavior.

A health check is not “the device turned on.” It is a small, targeted proof that the update did not break the prototype’s essential behavior.

Common health checks include:

Rollback evidence should record the failed check, previous version, attempted package, device state, and whether the device returned to service. A rollback without a reason is hard to distinguish from an ordinary reboot.

22.13 Rollout Gates

This section: Rollout Gates — Rollout is a decision process, not a single button.

Rollout is a decision process, not a single button. Each gate should have a small enough blast radius that the team can learn, pause, and recover.

Inspect Figure 22.5 and ask what evidence permits a candidate to expand from one rollout population to the next.

+

Read Figure 22.5 from the build through each larger group. Lab rigs come first, then staff devices, a pilot batch, and wider use. Each group supplies proof for install, boot, health, app use, and recovery before risk grows. A breached limit leaves the growth path at once. It leads to pause, stop, rollback, and repair, not to a note that the next gate may ignore. The final record links one release to each group, its measures, its stop rules, and the recovery choice.

Review gates before a wide update:

22.14 Cold-Room OTA Review

This section: Cold-Room OTA Review — A team is preparing an update for a cold-room monitor prototype.

A team is preparing an update for a cold-room monitor prototype. The update changes retry behavior after gateway loss and adds a compact post-reset fault record. The testing chapter already proved the new retry state on a bench board and an integration rig.

prototype=cold-room-monitor
update_question=retry-state update must not stop local monitoring
package_id=
source_revision=
target_hardware_revision=
current_version_range=
new_version=
normal_update_evidence=download, verify, stage, reboot, health confirm
fault_evidence=incompatible package, interrupted download, forced health failure
rollback_evidence=previous image restored and rollback reason reported
canary_gate=success, retry, rollback, and missing-report signals reviewed
handoff=best-practices chapter should keep update records in release review

22.15 OTA Handoff Record

This section: OTA Handoff Record — Rerun the OTA path whenever the bootloader, partition layout, update agent, package format, signing process, communication dependency, storage migration, or health-check criteria changes.

Every reviewed OTA path should leave a handoff record for future releases:

release_name=
package_id=
package_type=
source_revision=
dependency_record=
target_devices=
excluded_devices=
eligibility_rules=
safety_controls=
lab_evidence=
fault_evidence=
rollout_gates=
pause_or_abort_rules=
health_signals=
rollback_path=
unresolved_devices=
next_rerun_condition=
review_owner=

Rerun the OTA path whenever the bootloader, partition layout, update agent, package format, signing process, communication dependency, storage migration, or health-check criteria changes.

22.16 Knowledge Check

This section: Knowledge Check

22.17 Common Failure Patterns

This section: Common Failure Patterns

22.18 Summary

This section: Summary — First, OTA review starts with package identity, device eligibility, recovery path, and rollout evidence.

First, OTA review starts with package identity, device eligibility, recovery path, and rollout evidence. Next, Build, signing or approval, distribution, rollout policy, device agent, and health telemetry are separate boundaries. Then, Safety controls should cover authenticity, integrity, compatibility, anti-rollback policy, power and storage gates, recoverable boot, health confirmation, and telemetry. After that, A device update state machine should make rejection, retry, confirmation, and rollback visible. Finally, Staged rollout gates let the team learn, pause, abort, and record before a wider update becomes hard to reverse.

22.19 Key Takeaway

This section: Key Takeaway — OTA design is a reliability feature: include identity, integrity checks, rollback, staged rollout, telemetry, and a recovery path before relying on remote updates.

OTA design is a reliability feature: include identity, integrity checks, rollback, staged rollout, telemetry, and a recovery path before relying on remote updates.

22.20 What’s Next

This section: What’s Next