Design Methodology · Study deck

End-to-End Testing: Field Conditions and Evidence

A battery device can pass on a bench and fail after a cold radio retry.

Blueprint Bina is your guide for this deck.

testingvalidationtradeoff
iotclass.org

After studying this chapter

Learning objectives

You will be able to:

  • Design IoT tests for power, radio, timing, and sensor faults.
  • Assemble release evidence from injected failures and recovery.
  • Explain: The mathematical gist: A 2400 mAh, 3.6 V cell stores 8.64 Wh by nameplate, but a 120 mA pulse through 3 Ω sags 0.360 V.
  • Explain: Defect status records severity, impact, and a follow-up result; regression checks guard the failure path after a fix.
iotclass.org

Major section

Phoebe's Field Notes: Why a Battery-Budget Test Needs More Than mAh × V

The mathematical gist: A 2400 mAh, 3.6 V cell stores 8.64 Wh by nameplate, but a 120 mA pulse through 3 Ω sags 0.360 V.

  • After an explicit 80% derate, 1920 mAh at an 80 µA average gives 24,000 hours, or 2.74 years.
  • A 6 dBi antenna also changes a 20 dBm EIRP-limited radio from 20 to 14 dBm conducted power, so the installed antenna belongs in the same test.

Numbers to remember

14 dBmEIRP conducted power
iotclass.org

Major section

Release Gate Evidence

Defect status records severity, impact, and a follow-up result; regression checks guard the failure path after a fix.

  • The environment item asks whether test conditions match the release context, while security and update checks cover credentials, OTA, rollback, and response.
  • Operations readiness adds monitoring, a support runbook, staged rollout, and a rollback plan.
  • In the A release gate approves, holds, waives, or redesigns only after traceability, defects, regression proof, release environment, recovery readiness, and ownership are explicit diagram, traceability maps critical requirements to tests or accepted waivers.

Key terms

Critical defects
Critical defects are closed with a repeated failing scenario or equivalent regression check; remaining defects have severity, impact, owner, and decision.
Known failures
Known failures are hidden in notes or deferred without review.
A release gate approves, holds, waives, or redesigns only after traceability, defects, regression proof, release environment, recovery readiness, and ownership are explicit.
A release gate approves, holds, waives, or redesigns only after traceability, defects, regression proof, release environment, recovery readiness, and ownership are explicit.
iotclass.org

Major section

Release Gate Evidence (continued)

Those records support an approve, hold, waive, or redesign choice with an owner.

  • A clean lab result cannot replace an open field or recovery check.
  • Critical requirements map to passed tests or accepted waivers.
  • Important claims have no test, no owner, or no result.
  • Known failures are hidden in notes or deferred without review.
iotclass.org

Major section

Release Gate Evidence (continued)

Critical defects are closed with a repeated failing scenario or equivalent regression check; remaining defects have severity, impact, owner, and decision.

  • Fixes added or updated repeatable tests that run in the right pipeline.
  • A defect was fixed manually but has no guard against returning.
  • The product is approved from a clean lab run only.
iotclass.org

Major section

Plan Tests Before Running Them

A test plan does not need to be long.

  • It needs to prevent ambiguity when results arrive.
  • Results cannot be reproduced because the environment was not recorded.
  • Disconnect network for defined periods during sampling, transmit, update, and idle states.
  • Pass if all queued readings are delivered once, timestamps remain ordered, and power returns to sleep budget.

Why it matters

Running a test because the tool exists, not because a requirement needs evidence.

iotclass.org

Major section

Metrics Without False Confidence

Metrics should guide review, not replace judgment.

  • A large number can still be meaningless if the test does not assert the right behavior.
  • Assertions are weak, negative paths are absent, or hardware effects are outside the test.
  • Skipped, quarantined, or non-critical tests hide the risky areas.
  • Helps track triage load and repeated failure areas.

Key terms

Late field failures
Late field failures are excluded from the metric.
iotclass.org

Major section

Incremental Examples

A first validation pass can protect isolated firmware or service logic.

  • GoogleTest, pytest, or Vitest checks that malformed sensor payloads are rejected, timestamps are ordered, units are normalized, and the alarm state machine moves through normal, warning, alarm, mute, and clear states.
  • The evidence is a CI run with test vectors and expected outputs.
  • A gateway reconnect claim needs integration and system evidence.
  • A Mosquitto or EMQX broker can force disconnects, duplicate deliveries, retained sessions, and delayed acknowledgements while firmware logs, MQTT broker logs, packet captures, and dashboard records are saved together.
The validation route can be checked by asking which evidence belongs at each stage before a release decision.
The validation route can be checked by asking which evidence belongs at each stage before a release decision.
iotclass.org

Major section

Incremental Examples (continued)

Wireshark or tcpdump shows packet behavior; OpenTelemetry, CloudWatch, Azure Monitor, or Grafana shows backend timing.

  • If duplicate messages appear only after reconnect, the regression check should repeat that failure path instead of only rerunning happy-path unit tests.
  • A field release for OTA firmware needs hardware, cloud, and operations evidence.
  • The release decision should say which firmware build, device cohort, monitoring alerts, and rollback trigger are covered.
iotclass.org

Major section

Incremental Examples (continued)

Risk review then asks what interrupted downloads, low battery, signature checks, boot slots, or rollback could invalidate that claim.

  • The test plan chooses the fixture, data, and pass rule before an evidence run saves logs, versions, measurements, and defects.
  • The last stage records approve, hold, or redesign with the owner and release scope.
  • Together they explain why The validation route can be checked by asking which evidence belongs at each stage before a release decision matters to: Hold an OTA Release Gate.
iotclass.org

Major section

Debate: IoT System Design Trade-off Scenarios

A debate is useful only when its assumptions and exit path are visible.

  • Firmware means the program stored on a device to control its hardware.
  • A real-time operating system means software that schedules device tasks within known timing rules; it is shortened to RTOS.
  • This exercise supports a bounded decision, not a universal winner.
iotclass.org

Major section

Debate: IoT System Design Trade-off Scenarios (continued)

The scenarios below expose architecture, data, support, ethics, power, and compatibility trade-offs.

  • Scenario: A startup is building a fleet management platform expected to scale from 100 to 100,000 vehicles over 3 years.
  • Scenario: An energy company needs to store and query 10 years of smart meter data (15-minute intervals, 1 million meters).
  • Scenario: A fitness wearable company has anonymized health data from 10 million users.
iotclass.org

Deck summary

Key takeaways

The mathematical gist: A 2400 mAh, 3.6 V cell stores 8.64 Wh by nameplate, but a 120 mA pulse through 3 Ω sags 0.360 V.

  • Defect status records severity, impact, and a follow-up result; regression checks guard the failure path after a fix.
  • Those records support an approve, hold, waive, or redesign choice with an owner.
  • Critical defects are closed with a repeated failing scenario or equivalent regression check; remaining defects have severity, impact, owner, and decision.
  • A test plan does not need to be long.
iotclass.org

Retrieval practice

Recall check 1 of 5

Blueprint Bina says: answer from memory, then check your reasoning.

Q1A battery sensor recovers on the bench after one reboot. What release evidence is still needed?

ATreat a bench reboot as proof of installed-path link, power and update recovery under field faults.
BCheck only power-on state after reboot because it establishes duplicate control, freshness, and rollback.
CIgnore duplicate events after reconnect and count the link’s return as release proof.
DInject link/power/update faults on site; log recovery, duplicates, stale data and rollback.
Show answer

Answer: D Field recovery spans interacting failure paths.

iotclass.org

Retrieval practice

Recall check 2 of 5

Blueprint Bina says: answer from memory, then check your reasoning.

Q2What selects cross-layer failure modes that a system test plan must exercise?

ARequirement Claim
BCross-layer Risk Review
CLayered Test Plan
DVersioned Evidence Run
Show answer

Answer: B Risk review belongs beside the claim because it selects which boundaries and failure modes the plan must exercise.

Q3A battery sensor has strong unit tests for its state machine, but field pilots show duplicate cloud messages after intermittent network loss. What evidence is missing?

AAdd mocked state-machine tests without a real reconnect or backend path.
BWaive duplicate delivery without assigning an owner or mitigation.
CTest reconnect, queues, message IDs, and backend handling under network faults.
DExplain duplicates in marketing without tracing their system cause.
Show answer

Answer: C IoT field behavior often crosses layers.

iotclass.org

Retrieval practice

Recall check 3 of 5

Blueprint Bina says: answer from memory, then check your reasoning.

Q4A release gate shows one critical requirement marked as skipped because the environmental chamber was unavailable. What is the most defensible release decision?

AApprove release because the dashboard is mostly green and the missing chamber test is only one row.
BHold release or issue a scoped waiver with rationale, owner, expiry, monitoring, and rollback plan.
CDelete the requirement from the traceability matrix and recalculate the pass rate.
DUse a unit-test coverage number as a substitute for environmental evidence.
Show answer

Answer: B Release decisions must expose evidence gaps, defects, waivers, and residual risk instead of hiding them behind a pass rate.

iotclass.org

Retrieval practice

Recall check 4 of 5

Blueprint Bina says: answer from memory, then check your reasoning.

Q5This chapter's smart-meter storage scenario asks you to choose between InfluxDB/TimescaleDB and PostgreSQL for 10 years of data from 1 million meters at 15-minute intervals. Which discussion points does the chapter explicitly list for that decision?

AUse recent/historical query needs, compression, team expertise and BI integration.
BChoose by firmware language and assume it determines historical query and compression fit.
CChoose by utility ownership rather than its BI tools, data retention and query mix.
DChoose by meter enclosure size because it predicts decade-scale storage and integration requirements.
Show answer

Answer: A The chapter lists four discussion points for the time-series-vs-SQL scenario: query patterns (recent vs historical), compression requirements, the team's existing expertise, and integration with BI tools.

iotclass.org

Retrieval practice

Recall check 5 of 5

Blueprint Bina says: answer from memory, then check your reasoning.

Q6The chapter's cattle health monitor scenario constrains the design to one CR2032 battery and a 5-year life target. Which three features does the chapter say compete for that power budget?

AGPS (high), temperature (low) and heart rate (medium).
BWi-Fi, touchscreen and voice alerts, all counted as low power
CSolar, satellite and camera imaging instead of the listed sensors
DBattery life, GPS and cellular data, omitting the heart-rate feature
Show answer

Answer: A The chapter's scenario names exactly three features competing for one CR2032 battery's power budget over a 5-year target life: GPS location (called out as high power), temperature sensing (low power), and heart rate monitoring (medium power).

iotclass.org

Print reference

Answers 1 of 2

Answer key.

  1. D · Field recovery spans interacting failure paths.
  2. B · Risk review belongs beside the claim because it selects which boundaries and failure modes the plan must exercise.
  3. C · IoT field behavior often crosses layers.
  4. B · Release decisions must expose evidence gaps, defects, waivers, and residual risk instead of hiding them behind a pass rate.
iotclass.org

Print reference

Answers 2 of 2

Answer key.

  1. A · The chapter lists four discussion points for the time-series-vs-SQL scenario: query patterns (recent vs historical), compression requirements, the team's existing expertise, and integration with BI tools.
  2. A · The chapter's scenario names exactly three features competing for one CR2032 battery's power budget over a 5-year target life: GPS location (called out as high power), temperature sensing (low power), and heart rate monitoring (medium power).
iotclass.org