Testing & Validation · Study deck
System Validation
Picture a greenhouse node that displays a neat moisture value on a desk.
Test Tessa is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- distinguish verification (built to spec) from validation (solves the real problem) and explain why IoT needs both
- name the four core test levels (unit, integration, system, acceptance) and place them on the testing pyramid by speed and cost
- schedule test levels by cadence (commit, pull request, nightly, weekly, pre-release) rather than running everything on every change
- explain why code coverage measures quantity not quality, and what boundary/negative/error-path cases add
Major section
Start With the Story: The Device Worked Until It Left the Bench · Overview: Verify the Build, Validate the Mission
In the field, a wet plug changes the reading and a weak link delays the warning.
- The parts still run, but the grower cannot rely on the result.
- Firmware is the code stored on a device.
- It defines the boundary of current evidence.
Major section
Practitioner: Test Levels, the Pyramid, and When to Run What
A real strategy is built from named test levels, each answering a different question.
- Unit tests check individual functions or modules in isolation, often with hardware mocked.
- Integration tests check that components work together across an interface — firmware to cloud, device to app, one service to another.
Major section
Practitioner: Test Levels, the Pyramid, and When to Run What (continued)
System (end-to-end) tests exercise the complete path from sensor to dashboard.
- Acceptance tests confirm the system meets user-facing requirements.
- The Testing Pyramid The pyramid is a heuristic for distributing effort by cost and speed.
- The upward trade-off is slower, costlier, broader evidence.
Major section
Practitioner: Test Levels, the Pyramid, and When to Run What (continued)
Milliseconds, essentially free; run on every commit.
- Fast, cheap, repeatable unit tests form the wide base; slower integration tests sit in the middle; expensive end-to-end tests are the narrow top.
- Seconds to minutes; run on pull requests.
- Missing contract tests let field-name and unit mismatches slip through.
Major section
Practitioner: Test Levels, the Pyramid, and When to Run What (continued)
That relationship supports frequent low-level gates while keeping a deliberate set of system and acceptance checks for whole-path claims.
- Slow and expensive; keep the set small.
- Over-investing here makes the suite too slow to run often.
- Slowest, highest confidence; run before release.
Major section
Practitioner: Test Levels, the Pyramid, and When to Run What (continued)
A common guideline is roughly two-thirds to four-fifths unit tests, with integration and end-to-end making up the remainder — treat the exact percentages as a target, not a law.
- Passing acceptance in the lab is not the same as field validation.
- The point is to keep developer feedback in seconds while still exercising the slow, real-world paths on a schedule.
- Bench says: unit tests, static analysis, and a build check pass on every commit, in seconds.
- Spend Coverage Where Failure Hurts Coverage is a budget, not a single global number.
Major section
Under the Hood: Coverage Quality, Metrics, and Validation That Continues
The deeper layer is about the gap between a passing suite and a trustworthy system.
- Coverage Is Quantity, Not Quality A suite can execute 90% of the lines while proving almost nothing if it runs the same happy-path inputs over and over.
- Breadth of code exercised by tests.
Major section
Under the Hood: Coverage Quality, Metrics, and Validation That Continues (continued)
Coverage measures which code ran, not whether the right cases were tried.
- A high coverage number with no boundary or failure cases is a confidence illusion.
- HIL, SIL, and the Bugs Units Cannot See Some defects only appear with real timing and real I/O.
- Stability of the suite over runs.
Major section
Under the Hood: Coverage Quality, Metrics, and Validation That Continues (continued)
Conformance Is Not Interoperability Two distinct claims are easy to conflate.
- Interoperability testing checks that it actually works with other independent implementations of that specification.
- Coverage and pass-rate numbers become useful only when they remain connected to faults and field behavior.
- Long detection means bugs travel far before discovery.
Major section
Under the Hood: Coverage Quality, Metrics, and Validation That Continues (continued)
The shape of the box, not just its center, is what tells you the sensor is seeing a genuinely different distribution rather than a shifted average.
- Field failure rate then tests whether the earlier measures predicted the mission that mattered.
- Consistently high, with flaky tests investigated.
- Bland–Altman limits of agreement.
Major section
Under the Hood: Coverage Quality, Metrics, and Validation That Continues (continued)
Field adds: whether the data is even normal, how wide two methods really disagree, and whether raters agree or merely trend together.
- No single arrow proves quality; the ordered flow shows where a reassuring local metric can be contradicted by later evidence and sent back into the validation plan.
- Mean 10, SD 8 gives mean-2SD = -6, an impossible value — not normal.
- Close means can still hide a wide, unacceptable spread.
Major section
Under the Hood: Coverage Quality, Metrics, and Validation That Continues (continued)
d = -27.2, s = 34.8 → 95% limits -95.4 to 41.1.
- A stronger check, such as the Shapiro–Wilk test, gives a p-value for the normality assumption itself; treat "the data is normal" as something to test, not assume.
- Reporting correlation when the claim is really about agreement.
- Bench says: a mean, a correlation, or a pass/fail count on the measurement.
Major section
Under the Hood: Coverage Quality, Metrics, and Validation That Continues (continued)
When two people, tools, or automated classifiers are meant to reach the same categorical judgment on device or field data, check the agreement statistic, not just the correlation.
- Validation Does Not End at Launch Lab conditions — stable power, clean RF, a perfect network — do not represent the field, so systems validated only in the lab routinely fail once deployed.
- Soak testing runs the system continuously for long periods to expose slow leaks, memory growth, and drift.
- Regression testing re-runs prior tests after every change so a fix in one place does not silently break another.
Major section
Under the Hood: Coverage Quality, Metrics, and Validation That Continues (continued)
Short, because fast suites run often.
- Common Pitfalls Review these failure modes in the order they can weaken or invalidate the result.
- Finish with Letting test plans rot.: Requirements drift; tests for removed features pass meaninglessly while new behavior goes untested unless plans are owned and updated.
- This sequence connects each warning to the evidence a reviewer should demand before accepting the claim.
Major section
Summary
Shift-left testing reflects the rule of thumb that defects grow roughly an order of magnitude more expensive at each later stage.
- The test levels are unit, integration, system (end-to-end), and acceptance, surrounded by specialized IoT levels: hardware-in-the-loop, environmental, security, and field or soak testing.
- The testing pyramid distributes effort by speed and cost: many fast unit tests, fewer integration tests, and a small set of slow end-to-end tests; treat the percentages as a target, not a law.
- Coverage measures quantity, not quality; test design needs boundary values, equivalence partitions, and negative and error-path cases.
Deck summary
Key takeaways
In the field, a wet plug changes the reading and a weak link delays the warning.
- A real strategy is built from named test levels, each answering a different question.
- System (end-to-end) tests exercise the complete path from sensor to dashboard.
- Milliseconds, essentially free; run on every commit.
- That relationship supports frequent low-level gates while keeping a deliberate set of system and acceptance checks for whole-path claims.
Retrieval practice
Recall check 1 of 3
Test Tessa says: answer from memory, then check your reasoning.
Q1In IoT quality assurance, what is the difference between verification and validation?
Show answer
Answer: A Verification is 'did we build it right' against the spec; validation is 'did we build the right thing' for the real-world problem.
Retrieval practice
Recall check 2 of 3
Test Tessa says: answer from memory, then check your reasoning.
Q2A device has 100% unit-test coverage and the cloud backend has 95%. After deployment, data is lost because the device sends a JSON field named 'temperature' but the cloud parser expects 'temp'. Which testing gap let this reach production?
Show answer
Answer: B Each side works in isolation, so unit tests pass on both.
Retrieval practice
Recall check 3 of 3
Test Tessa says: answer from memory, then check your reasoning.
Q3During nightly CI, a hardware-in-the-loop test passes about 80% of runs and fails the other 20% with no code changes between runs. What is the most likely root cause and the right response?
Show answer
Answer: A Intermittent failures on real hardware usually expose timing or concurrency bugs that deterministic host-based unit tests never trigger.
Print reference
Answers
Answer key.
- A · Verification is 'did we build it right' against the spec; validation is 'did we build the right thing' for the real-world problem.
- B · Each side works in isolation, so unit tests pass on both.
- A · Intermittent failures on real hardware usually expose timing or concurrency bugs that deterministic host-based unit tests never trigger.