18 CI/CD Fundamentals for IoT
18.1 Start With the Change That Must Ship
Imagine changing one line that controls a pump. A fast test can catch a simple mistake, but the real device must still behave safely. Continuous integration and continuous delivery (CI/CD) is a repeatable path that builds, checks, and records each change. Firmware means the software stored inside the device.
Start with the risk created by the change. Run quick code checks first, then a model, and then the real target when timing or hardware matters. Save the exact build and test result. Test a broken dependency, a false sensor value, and a failed device check. A green early stage cannot replace a later physical check.
Use one question at each stage. Did the source change match the request? Did the build use known tools? Did quick checks reject common mistakes? Did unit tests cover the changed rule? Did the model use honest device limits? Did the real board meet timing and power needs? Did the candidate keep the evidence from every stage? Did a person approve the remaining risk?
Fail early when possible. Keep later checks for facts that need real hardware. A stage may pass and still leave a known gap. Show that gap beside the green result.
This simple path does not define every release rule. The right stages depend on the product and harm a failure could cause. The Practitioner section builds the evidence chain. Under the Hood examines automation, target tests, and release controls.
Imagine changing one line of firmware for a sensor device and then being asked a practical question: can this exact change go to real hardware? A CI/CD pipeline is the evidence trail that answers. It links the source change to the build, the tests, the signed artifact, the release gate, and the retest trigger, so a reviewer can approve or hold the candidate without guessing which bytes are about to ship.
18.2 Overview: From a Change to a Releasable Candidate, Repeatably
Continuous integration and continuous delivery describe one disciplined path: every change to firmware, configuration, or service code is built and tested automatically, and the result is a candidate that is always ready to release behind a deliberate gate. Continuous integration is the front half — integrate small changes frequently so a machine builds and tests each one, catching conflicts and regressions while they are small. Continuous delivery is the back half — keep a tested candidate that can be approved, paused, rolled out in stages, or rolled back at any time.
The value is not "automatic release." The value is repeatable evidence and fast feedback: a record of what changed, what was built, which checks ran, what passed or failed, and which artifact is allowed to ship. IoT raises the bar over a web pipeline because the same source may have to build for several hardware targets, the output is a firmware image with a manifest and a signature rather than a container, real-device timing matters, and you cannot quietly hotfix a device in someone's wall. The pipeline has to make those hardware-shaped assumptions visible instead of hiding them behind a single green check.
If you only need the intuition, this layer is enough: continuous integration builds and tests every change automatically; continuous delivery keeps a releasable, gated candidate ready at all times. The pipeline's job is repeatable evidence and fast feedback, and for IoT that evidence must include the hardware targets, the signed artifact, and the recovery plan, not just "the build passed."
Before treating a green pipeline as permission to ship, use Figure 18.1 to locate the evidence that each stage contributes and the hardware boundary that each later check closes.
Read Figure 18.1 from left to right. The source change establishes what must be traced; build, static analysis, and unit tests reject inexpensive defects first; simulation exercises modeled peripherals; and hardware-in-the-loop checks the real target and timing. The final release decision therefore inherits a chain of increasingly physical evidence, connecting rapid integration feedback to an IoT candidate that is genuinely ready for a controlled rollout.
18.2.1 The One-Minute View
Integrate continuously
Every change is built and tested automatically so conflicts and regressions surface while they are small.
Gate before release
Continuous delivery keeps a tested candidate ready, released only through a deliberate gate.
Hardware changes the bar
Multiple build targets, signed firmware artifacts, and no easy field hotfix make IoT pipelines carry extra evidence.
18.2.2 Beginner Examples
Read these examples as a progression from the simplest observation to the boundary that still needs evidence. Begin with this case: a developer pushes a one-line firmware fix; the pipeline rebuilds for every supported board and runs the test suite before anyone reviews it. Then examine how a build can be green while the release gate still holds it because the security scan has not run: “all jobs passed” is not “approved to ship.” Finish by checking artifact identity—the image engineers tested and the image that reaches devices must be the same signed artifact, tied together by a digest rather than rebuilt at deploy time. Together, the examples connect an intuitive result to a bounded engineering claim.
18.2.3 Overview Knowledge Check
If you can separate continuous integration from continuous delivery, you have the core idea. Continue to Practitioner for the pipeline stages and the gates that protect a release.
18.3 Practitioner: Pipeline Stages and Automated Gates
A pipeline is a sequence of stages, each triggered by the last and each leaving evidence. A typical IoT flow runs: a commit triggers the pipeline; the candidate is built (often cross-compiled for several hardware targets); static analysis and unit tests run close to the change; integration and hardware-aware tests exercise the boundaries the change can affect; the artifact is packaged, versioned, and signed; a release gate compares the evidence against the change; and delivery proceeds in stages to the fleet. The sequence keeps fast checks early, so most failures stop the pipeline in seconds, and slow real-device checks later, where they are worth their cost.
18.3.1 Automated Quality Gates
Between stages sit automated gates: conditions the candidate must meet to proceed. The build must succeed for every claimed target; unit and static checks must pass; a security scan must run; lint or coverage thresholds, where the team sets them, must hold. A failed gate stops the pipeline and reports why, which is the mechanism that keeps a broken change from advancing. A gate is only honest if a skipped check is recorded as skipped with a reason and owner, never silently passed.
To see why a gate is a reviewable decision rather than a green badge, inspect Figure 18.2 before choosing which candidates may advance.
In Figure 18.2, begin with change scope, because it determines which evidence is relevant. Move next through test coverage and the intended release boundary, checking that skipped or limited tests remain visible. Finish at the retest trigger and decision: approve only within the supported boundary, hold when evidence is incomplete, and roll back or retest when a gate fails. This makes the pipeline’s fast feedback durable enough to govern a release.
18.3.2 Cadence: Run the Right Stage at the Right Time
Not every stage runs on every change. Keep developer feedback in seconds by running the build, static analysis, and unit tests on every commit; run integration and security scans on pull requests; run hardware-in-the-loop and extended integration nightly; and reserve full regression and the release gate for the candidate being prepared to ship. This is the testing pyramid's cadence applied to delivery: fast checks gate every change, expensive checks run when they are worth it.
18.3.3 Practitioner Knowledge Check
If you can place the stages, set the gates, and run them at the right cadence, you can stop here. Continue to Under the Hood for the parts of IoT delivery a web pipeline never has to handle.
18.4 Under the Hood: What Makes an IoT Pipeline Different
The deeper layer is the handful of mechanisms that separate an IoT pipeline from an ordinary software one: building for many targets, keeping the tested artifact identical to the shipped artifact, putting real hardware in the loop, and ending every candidate with a recovery decision.
18.4.1 Cross-Compilation and the Build Matrix
A single firmware change often must build for several boards, chips, or sensor configurations, each with its own toolchain and constraints. A build matrix compiles the candidate for every claimed target, so a change that passes on one and fails on another is caught in CI, not in the field. The build evidence must name the target and configuration, because "it built" is meaningless without saying for what.
18.4.2 The Tested Artifact Must Be the Shipped Artifact
A web service can often be rebuilt at deploy time with little consequence. A firmware candidate must not be. The image engineers tested, the image the gate approved, and the image devices receive have to be the same bytes — promoted, not rebuilt — tied together by a digest and a signature. Rebuilding after testing reintroduces the risk the tests were meant to remove, because the shipped image was never the one under test. This is also why the artifact's manifest and signature belong in the pipeline's evidence.
18.4.3 Real Hardware in the Loop
Some defects only appear with real timing, interrupts, and I/O. Bringing hardware-in-the-loop rigs or device farms into the pipeline lets the slow, real-device tests run on a schedule against the actual target, surfacing race conditions and timing faults that deterministic host-based unit tests never trigger. An intermittent failure on such a rig is usually a real bug, not a flaky job to rerun until it is green.
18.4.4 Every Candidate Ends with a Recovery Decision
Because a bad firmware release can brick devices you cannot reach, the gate's output is not just approve or reject. For any change that can affect boot, update, or recovery state, the record must name the rollback or recovery path and the retest trigger before the candidate moves on. Delivery itself should be staged — a small cohort first, expanding only while telemetry stays clean — so a bad release reaches few devices; the mechanics of staged rollout and rollback are covered in the linked chapters.
18.4.5 Common Review Findings
Review these failure modes in the order they can weaken or invalidate the result. Begin with Green pipeline treated as proof. The gate must compare evidence to the change, not just collect passes. Then examine Tested and approved artifact not tied together. Without a shared digest, the shipped image may never have been tested. Then examine Skipped checks hidden in logs. A skip belongs in the record with a reason and owner. Then examine Flaky boundary rerun to green. Rerunning a hardware-aware check until it passes hides a real, uncertain boundary instead of resolving it. Then examine No recovery decision. A candidate that can affect update or boot state is approved with no rollback or recovery recorded. Finish with No retest trigger. The same candidate cannot be reliably re-evaluated after a related change. This sequence connects each warning to the evidence a reviewer should demand before accepting the claim.
18.4.6 Under-the-Hood Knowledge Check
At this depth, IoT CI/CD is a discipline of traceable, hardware-aware evidence: build for every target, keep the tested artifact identical to the shipped one, put real hardware in the loop, and end each candidate with a recovery decision. A trustworthy pipeline answers which stage produced each result and proves the image that ships is the image that passed.
18.5 Gate a Thermostat Firmware Change
For these CI/CD fundamentals, a thermostat patch changes the relay cut-off from 28 °C to 30 °C before thousands of devices receive it. Figure 18.1 follows that change through source, build, test, package, approval, and release evidence. Figure 18.1 shows the repeatable CI path, while Figure 18.2 places a human decision where device risk needs it.
For these CI/CD fundamentals, the thermostat pipeline can prove that the same source made the tested firmware artifact. A unit test can check the 30 °C threshold. Hardware-aware CI/CD fundamentals require a test that shows whether the real relay opens under load. A staged rollout can then compare resets and temperature alarms before the wider CD step. These fundamentals show why passing one gate does not replace the evidence owned by the next.
18.5.1 Predict the Pipeline Stop
- Predict: Source tests pass, but the signed package digest differs from the tested artifact. Should deployment continue? Check: No. The CI/CD chain no longer proves that the released firmware is the tested firmware.
- Predict: A simulator passes while the target board resets during relay switching. Is the IoT release evidence complete? Check: No. The physical power and relay interaction needs hardware validation.
18.6 Summary
Read the chapter’s conclusions as one connected evidence argument: Continuous integration builds and tests every change automatically; continuous delivery keeps a tested, gated candidate ready to release. The value is repeatable evidence and fast feedback, not automatic release. IoT pipelines carry more than a web pipeline: multiple hardware build targets, signed firmware artifacts with manifests, real-device timing, and no easy field hotfix. A typical pipeline runs commit → build (per target) → static + unit → integration + hardware-aware → package + sign → release gate → staged delivery, each stage leaving evidence. Automated quality gates stop a candidate that fails a required check; a gate is honest only when a skipped check is recorded with a reason and owner. Run fast checks on every commit and reserve slow, real-device checks for nightly or release cadence, applying the pyramid’s cadence to delivery. A build matrix compiles for every claimed target; the tested, approved, and shipped artifact must be the same bytes, promoted and verified by digest rather than rebuilt. Hardware-in-the-loop in CI surfaces timing bugs; every candidate that can affect boot, update, or recovery must record a rollback path and retest trigger, with delivery staged to bound the blast radius.
An IoT CI/CD pipeline earns trust by making hardware-shaped evidence visible: it builds for every target, proves the shipped image is the tested image, runs real hardware in the loop, and never approves a firmware candidate without a recovery path. Ask of any release: which stage produced this decision, and is the image about to ship the exact one that passed the gate?
18.7 See Also
OTA Update Architecture for IoT
Turn an approved artifact into a safe, verified, recoverable device update.
Rollback & Staged Rollouts
Pace delivery in canary waves and define rollback before the first device updates.
Monitoring and CI/CD Tools for IoT
Read the telemetry that grades a staged rollout and signals when to halt.
Testing Pyramid & Challenges
See how the test pyramid and its cadence feed the pipeline's stages.