Prototyping · Study deck

Testing and Debugging Firmware

A greenhouse controller reads soil moisture and shows a watering reminder.

Voltage Vera is your guide for this deck.

testing

After studying this chapter

Learning objectives

You will be able to:

Convert prototype questions into host tests, simulator checks, board runs, integration tests, and field observations.
Choose the right test boundary for firmware logic, drivers, state machines, communication paths, storage, and update behavior.
Add observability through logs, counters, trace pins, packet captures, persisted fault records, and dashboards without hiding timing or power problems.
Use a repeatable debug loop that isolates one variable at a time and records the result.

Major section

In 60 Seconds · Start With the Story

A greenhouse controller reads soil moisture and shows a watering reminder.

The demo works while every part is present.
The real learning begins when the measuring unit is unplugged, the stored setting is damaged, or power ends halfway through a save.
This prototype evidence does not certify the final product or every field condition.

Major section

Overview: Test the Boundary That Can Lie · Happy Paths Need Real Faults

Firmware tests are valuable when they stress the boundary most likely to hide the fault.

Pure logic can be tested on a host.
Pin timing, bus errors, sleep current, radio retries, and update recovery need hardware or a faithful integration rig.
The purpose is not to run every possible test.

A useful firmware test route keeps the prototype question, selected boundary, normal run, fault run, debug observation, regression record, and handoff decision connected.

Major section

Observability Before Debugging

Debugging is slow when the only evidence is a failed demo.

Good observability also changes what can be trusted.
A reboot with a stored reset reason, last state, queue depth, and version number is a reproducible event.
A reboot described as "it restarted once" is only a story.
Observation has a cost.

Major section

Observability Before Debugging (continued)

On ESP32, a record might include esp_reset_reason(), a compact fault code in non-volatile storage, the current OTA slot, queue depth, and the last sensor state.

In Zephyr, it might include structured logging, shell commands, watchdog reason, and a retained crash or reset counter.
Serial prints can change interrupt timing, UART buffers can hide scheduling bugs, network logs can alter radio duty cycle, and verbose messages can exhaust flash or heap.
Regression evidence should close the loop.

Major section

Observability Before Debugging (continued)

If adding logs makes the bug disappear, that is a finding: the bug is probably sensitive to timing, buffering, memory pressure, or task scheduling.

After the cause is isolated, the final record should connect the reproduced symptom, the smallest credible boundary, the instrument used, the one change made, the rerun result, and the condition that must reopen the test.
That condition might be a new sensor library, FreeRTOS task split, MQTT client update, bootloader change, OTA partition change, low-power mode, or gateway firmware update.
Without that trigger, the team can accidentally reuse old evidence after the prototype has become a different system.

Major section

Test Evidence Ladder

Host tests are excellent for pure logic, parsers, filters, and state machines.

Board and integration tests are necessary when timing, drivers, buses, radios, storage, power, or update behavior can change the result.
Each wider boundary answers questions the cheaper one cannot: parsers and state machines can run on a host, but buses, interrupts, timing, and power require hardware.
Normal and fault runs remain paired at every layer.
It proves assumptions only as far as the model is faithful.

The firmware testing ladder widens from source checks to field observation while pairing normal and fault runs at every layer.

Major section

Let Risk Set the Ladder Height · Choose the Test Boundary

A boundary is where the test enters and observes the system.

Good tests name their boundary in the record.
A vague "device test passed" is hard to reuse. "Board run proved missing sensor creates FAULT_SENSOR_MISSING within one sample period and local control remains active" is evidence.
Poor boundaries create false confidence: the test passes because the hard part was mocked away, or it fails because the setup is measuring a different system than the prototype uses.

Major section

Observability and Debug Fixtures

Debugging improves when the prototype is observable before it fails.

Observability is not just more logs.
That evidence lets the team check that changing retry behavior solved the observed delay without hiding stale readings.

Observability fixtures for IoT debugging

Major section

Do Not Let Observability Change the Bug · Debug With a Repeatable Loop

Serial prints, network logging, and verbose diagnostics can change timing, power, memory, and radio behavior.

Debugging should narrow the unknowns.
A useful debug session changes one thing, observes one outcome, and leaves a record.
If the setup changed, the result is not directly comparable.

Major section

Cold-Room Test Review

A team is prototyping a cold-room monitor.

The firmware reads a temperature probe, tracks alarm state, queues samples while the gateway is unavailable, and reports status when connectivity returns.
The demo worked on the bench, but the team needs evidence before a pilot.
The team does not rewrite the entire communication path.

Major section

Cold-Room Test Review (continued)

Host harness: Tests conversion math, threshold hysteresis, payload formatting, and state transitions with fixed sample sequences.

The initial gateway-blocked run shows that the queue fills and publish retries begin to delay sensor reads.
It first adds queue-depth counters and a retry-state trace, then confirms the delay appears only when publish retries run in the sensing path.
The final evidence record includes the host state-machine test, bench missing-probe run, gateway-blocked integration run, queue-depth log excerpt, and firmware commit used for the repeat run.

Major section

How to Use: Write a Firmware Regression Card · Regression and Handoff Record

Revise it if the record depends on memory or a one-off demo setup.

A fix is incomplete until it becomes regression evidence.
The team should know which command, board setup, fixture, version, and observation must be repeated when code, dependencies, configuration, hardware, or update behavior changes.
The fields in the Regression record fields diagram make a fix repeatable.
Setup records the board, fixture, inputs, and injected fault.

Major section

Common Failure Patterns

Happy-path test suite: The prototype proves only normal reads, normal publishes, and normal dashboard updates.

Debug output changes timing: Verbose serial or network logging shifts ISR timing, sleep behavior, memory pressure, or radio use.
Switch to targeted counters, trace pins, sampled logs, or post-reset records.
The evidence cannot be reproduced.
Improve observability before the next pilot run.

Major section

Checkpoint: Summary · Key Takeaway

Testing an IoT prototype means matching evidence to the risk boundary.

Host tests are valuable for logic, but board, integration, and field observations are needed for drivers, timing, storage, networks, power, and update behavior.
Observability should make faults visible without changing the behavior being measured.
Debugging should reproduce, isolate, instrument, change one variable, rerun, and record.

Deck summary

Key takeaways

A greenhouse controller reads soil moisture and shows a watering reminder.

Firmware tests are valuable when they stress the boundary most likely to hide the fault.
Debugging is slow when the only evidence is a failed demo.
On ESP32, a record might include esp_reset_reason(), a compact fault code in non-volatile storage, the current OTA slot, queue depth, and the last sensor state.
If adding logs makes the bug disappear, that is a finding: the bug is probably sensitive to timing, buffering, memory pressure, or task scheduling.

Retrieval practice

Recall check 1 of 4

Voltage Vera says: answer from memory, then check your reasoning.

Q1Per this chapter's discussion of observability costs, if a team adds serial logging to debug a fault and the fault disappears, what should the team conclude?

AThe bug is fixed because logging suppresses it; no timing or memory investigation is needed.

BThe logging code must have a syntax error because otherwise its presence cannot change the result.

CLogging can shift timing, buffering, memory or scheduling; a vanished bug is not proven fixed.

DKeep verbose logging enabled because its timing changes stabilize the fault path during operation.

Show answer

Answer: C The chapter states: 'Observation has a cost.

Retrieval practice

Recall check 2 of 4

Voltage Vera says: answer from memory, then check your reasoning.

Q2Per this chapter's test evidence ladder, what can 'static and source checks' catch, and what can't they prove?

AStatic checks prove runtime behavior on the target board, replacing the bench sensor run.

BStatic checks catch bus timing and missing-sensor faults because those are configuration errors.

CStatic checks matter only after an integration rig has already passed runtime faults.

DStatic checks catch syntax, unsafe patterns, missing records and config, not runtime behavior.

Show answer

Answer: D The chapter's test evidence ladder lists 'Static and source checks' as: 'Catch syntax, formatting, unsafe patterns, missing records, and obvious configuration mistakes.

Retrieval practice

Recall check 3 of 4

Voltage Vera says: answer from memory, then check your reasoning.

Q3Per this chapter's test-boundary choices, which boundary should a test use when it must prove bootloader, OTA package, rollback, configuration compatibility, or post-update health checks?

AStorage boundary

BUpdate boundary

CCommunication boundary

DState-machine boundary

Show answer

Answer: B see answers page

Retrieval practice

Recall check 4 of 4

Voltage Vera says: answer from memory, then check your reasoning.

Q4A prototype passes host tests for payload formatting, but a bench run shows the device sometimes stops sensing when the gateway is unavailable. What is the best next review action?

ARun a gateway-loss fault test and observe queue, retry state and sensing cadence through recovery.

BPromote host payload tests alone while deferring the gateway fault seen on the bench.

CRepeat isolated sensor-driver tests while leaving gateway retries unexamined.

DAdd broad function-entry logs and rerun, risking a timing change that hides the fault.

Show answer

Answer: A Host tests are useful, but they do not prove cross-layer behavior.

Print reference

Answers 1 of 2

Answer key.

C · If adding logs makes the bug disappear, that is a finding: the bug is probably sensitive to timing, buffering, memory pressure, or task scheduling.
D · The chapter's test evidence ladder lists 'Static and source checks' as: 'Catch syntax, formatting, unsafe patterns, missing records, and obvious configuration mistakes.
B · The chapter's boundary-choice checklist states: 'Update boundary: Use when a test must prove bootloader, OTA package, rollback, configuration compatibility, or post-update health checks.' This is distinct from the storage boundary, which covers local queue and flash behavior rather than the update/rollback path itself.

Print reference

Answers 2 of 2

Answer key.

A · Host tests are useful, but they do not prove cross-layer behavior.