Testing & Validation · Study deck
System Validation
Picture a greenhouse node that displays a neat moisture value on a desk.
Test Tessa is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- distinguish verification (built to spec) from validation (solves the real problem) and explain why IoT needs both
- name the four core test levels (unit, integration, system, acceptance) and place them on the testing pyramid by speed and cost
- schedule test levels by cadence (commit, pull request, nightly, weekly, pre-release) rather than running everything on every change
- explain why code coverage measures quantity not quality, and what boundary/negative/error-path cases add
Major section
Start With the Story: The Device Worked Until It Left the Bench · Overview: Verify the Build, Validate the Mission
In the field, a wet plug changes the reading and a weak link delays the warning.
- The parts still run, but the grower cannot rely on the result.
- Firmware is the code stored on a device.
- It defines the boundary of current evidence.
Major section
Practitioner: Test Levels, the Pyramid, and When to Run What
A real strategy is built from named test levels, each answering a different question.
- Unit tests check individual functions or modules in isolation, often with hardware mocked.
- Integration tests check that components work together across an interface — firmware to cloud, device to app, one service to another.
- Fast, cheap, repeatable unit tests form the wide base; slower integration tests sit in the middle; expensive end-to-end tests are the narrow top.
- The upward trade-off is slower, costlier, broader evidence.
Major section
Practitioner: Test Levels, the Pyramid, and When to Run What (continued)
System (end-to-end) tests exercise the complete path from sensor to dashboard.
- Acceptance tests confirm the system meets user-facing requirements.
- The Testing PyramidThe pyramid is a heuristic for distributing effort by cost and speed.
- Milliseconds, essentially free; run on every commit.
- Seconds to minutes; run on pull requests.
Major section
Practitioner: Test Levels, the Pyramid, and When to Run What (continued)
That relationship supports frequent low-level gates while keeping a deliberate set of system and acceptance checks for whole-path claims.
- Missing contract tests let field-name and unit mismatches slip through.
- Slow and expensive; keep the set small.
- Over-investing here makes the suite too slow to run often.
Major section
Practitioner: Test Levels, the Pyramid, and When to Run What (continued)
A common guideline is roughly two-thirds to four-fifths unit tests, with integration and end-to-end making up the remainder — treat the exact percentages as a target, not a law.
- Slowest, highest confidence; run before release.
- Passing acceptance in the lab is not the same as field validation.
- The point is to keep developer feedback in seconds while still exercising the slow, real-world paths on a schedule.
Major section
Under the Hood: Coverage Quality, Metrics, and Validation That Continues
The deeper layer is about the gap between a passing suite and a trustworthy system.
- Coverage Is Quantity, Not QualityA suite can execute 90% of the lines while proving almost nothing if it runs the same happy-path inputs over and over.
- Breadth of code exercised by tests.
Major section
Under the Hood: Coverage Quality, Metrics, and Validation That Continues (continued)
Coverage measures which code ran, not whether the right cases were tried.
- A high coverage number with no boundary or failure cases is a confidence illusion.
- HIL, SIL, and the Bugs Units Cannot SeeSome defects only appear with real timing and real I/O.
- Stability of the suite over runs.
Major section
Under the Hood: Coverage Quality, Metrics, and Validation That Continues (continued)
Conformance Is Not InteroperabilityTwo distinct claims are easy to conflate.
- Interoperability testing checks that it actually works with other independent implementations of that specification.
- Coverage and pass-rate numbers become useful only when they remain connected to faults and field behavior.
- Long detection means bugs travel far before discovery.
Major section
Under the Hood: Coverage Quality, Metrics, and Validation That Continues (continued)
The shape of the box, not just its center, is what tells you the sensor is seeing a genuinely different distribution rather than a shifted average.
- Field failure rate then tests whether the earlier measures predicted the mission that mattered.
- Consistently high, with flaky tests investigated.
- Bland–Altman limits of agreement.
Major section
Under the Hood: Coverage Quality, Metrics, and Validation That Continues (continued)
Mean 10, SD 8 gives mean-2SD = -6, an impossible battery life — inspect the distribution before using a normal interval.
- No single arrow proves quality; the ordered flow shows where a reassuring local metric can be contradicted by later evidence and sent back into the validation plan.
- d = -27.2, s = 34.8 → 95% limits -95.4 to 41.0.
- Close means can still hide a wide, unacceptable spread.
Major section
Under the Hood: Coverage Quality, Metrics, and Validation That Continues (continued)
Bench says: a mean, a correlation, or a pass/fail count on the measurement.
- A stronger check, such as the Shapiro–Wilk test, gives a p-value for the normality assumption itself; treat "the data is normal" as something to test, not assume.
- Reporting correlation when the claim is really about agreement.
- Soak testing runs the system continuously for long periods to expose slow leaks, memory growth, and drift.
Major section
Under the Hood: Coverage Quality, Metrics, and Validation That Continues (continued)
When two people, tools, or automated classifiers are meant to reach the same categorical judgment on device or field data, check the agreement statistic, not just the correlation.
- Field adds: whether the data is even normal, how wide two methods really disagree, and whether raters agree or merely trend together.
- Validation Does Not End at LaunchLab conditions — stable power, clean RF, a perfect network — do not represent the field, so systems validated only in the lab routinely fail once deployed.
- Common PitfallsReview these failure modes in the order they can weaken or invalidate the result.
Major section
Under the Hood: Coverage Quality, Metrics, and Validation That Continues (continued)
Short, because fast suites run often.
- Regression testing re-runs prior tests after every change so a fix in one place does not silently break another.
- Finish with Letting test plans rot: Requirements drift; tests for removed features pass meaninglessly while new behavior goes untested unless plans are owned and updated.
- This sequence connects each warning to the evidence a reviewer should demand before accepting the claim.
Major section
Summary
Verification confirms the system was built correctly against its specification; validation confirms the built system solves the real problem in real conditions.
- Shift-left testing reflects the rule of thumb that defects grow roughly an order of magnitude more expensive at each later stage.
- The test levels are unit, integration, system (end-to-end), and acceptance, surrounded by specialized IoT levels: hardware-in-the-loop, environmental, security, and field or soak testing.
- The testing pyramid distributes effort by speed and cost: many fast unit tests, fewer integration tests, and a small set of slow end-to-end tests; treat the percentages as a target, not a law.
Deck summary
Key takeaways
In the field, a wet plug changes the reading and a weak link delays the warning.
- A real strategy is built from named test levels, each answering a different question.
- System (end-to-end) tests exercise the complete path from sensor to dashboard.
- That relationship supports frequent low-level gates while keeping a deliberate set of system and acceptance checks for whole-path claims.
- A common guideline is roughly two-thirds to four-fifths unit tests, with integration and end-to-end making up the remainder — treat the exact percentages as a target, not a law.
Retrieval practice
Recall check 1 of 3
Test Tessa says: answer from memory, then check your reasoning.
Q1In IoT quality assurance, what does verification check?
Show answer
Answer: C Verification is 'did we build it right' against the spec; validation is 'did we build the right thing' for the real-world problem.
Retrieval practice
Recall check 2 of 3
Test Tessa says: answer from memory, then check your reasoning.
Q2A device has 100% unit-test coverage and the cloud backend has 95%. After deployment, data is lost because the device sends a JSON field named 'temperature' but the cloud parser expects 'temp'. Which testing gap let this reach production?
Show answer
Answer: C Each side works in isolation, so unit tests pass on both.
Retrieval practice
Recall check 3 of 3
Test Tessa says: answer from memory, then check your reasoning.
Q3During nightly CI, a hardware-in-the-loop test passes about 80% of runs and fails the other 20% with no code changes between runs. The rig's reset and stimulus timing have been checked. Which timing-related cause should the team investigate first?
Show answer
Answer: D Intermittent failures on real hardware usually expose timing or concurrency bugs that deterministic host-based unit tests never trigger.
Print reference
Answers
Answer key.
- C · Verification is 'did we build it right' against the spec; validation is 'did we build the right thing' for the real-world problem.
- C · Each side works in isolation, so unit tests pass on both.
- D · Intermittent failures on real hardware usually expose timing or concurrency bugs that deterministic host-based unit tests never trigger.