Technical detail · Testing and Debugging Firmware
Under the Hood: The cost of observing firmware
Non-volatile storage keeps data when power is removed, so a saved fault can survive a reboot. An OTA slot is a location in flash holding a firmware image; recording the slot identifies which image was running. Good observability also changes what can be trusted. A reboot with a stored reset reason, last state, queue depth, and version number is a reproducible event. A reboot described as “it restarted once” is only a story. On ESP32, a record might include esp_reset_reason(), a compact fault code in non-volatile storage, the current OTA slot, queue depth, and the last sensor state. In Zephyr, it might include structured logging, shell commands, watchdog reason, and a retained crash or reset counter.
A UART buffer temporarily holds serial data as it is sent or received. Heap memory is the pool used for dynamic allocations while code runs. Observation has a cost. Serial prints can change interrupt timing, UART buffers can hide scheduling bugs, network logs can alter radio duty cycle, and verbose messages can exhaust flash or heap. When the symptom is timing-sensitive, prefer a GPIO trace pin, a counter sampled at the end of a run, a logic-analyzer capture, or a packet capture from Wireshark, mosquitto, tcpdump, or the gateway rather than printing from every function entry. If adding logs makes the bug disappear, that is a finding: the bug is probably sensitive to timing, buffering, memory pressure, or task scheduling.