7 System Validation

7.1 Start With the Story: The Device Worked Until It Left the Bench

Turn the Mission Into a Test Claim

Picture a greenhouse node that displays a neat moisture value on a desk. In the field, a wet plug changes the reading and a weak link delays the warning. The parts still run, but the grower cannot rely on the result.

Firmware is the code stored on a device. An analog-to-digital converter is a part that turns a measured voltage into a number; it is often shortened to ADC. A test claim should connect those parts to the mission: under named soil, weather, power, link, and user conditions, the system reports within a stated error and time.

Prove the narrow build rules first, then challenge the full claim. Change the input, lower power, delay the network, restart each part, remove a record, and ask a real user to act. Keep expected and observed results, conditions, owner, and next decision.

Passing these cases does not prove all future fields or users. It defines the boundary of current evidence. The deeper sections separate checks of the build from checks of the mission and show how unit, link, system, and field tests fit together.

Picture a soil-moisture node that passes every desk check: the firmware boots, the ADC returns numbers, the cloud receives payloads, and the dashboard draws a line. Then it goes into a greenhouse, condensation reaches the connector, the radio fades near a metal rack, and the farmer still does not trust the alert. The build was verified, but the mission was not yet validated.

Start simple: write down the real claim before choosing a test. If the claim is "this calculation follows the spec," verification can answer it. If the claim is "this helps someone make the right decision in the field," validation has to include the environment, user, network, device state, and evidence record that make the claim believable.

Test Tessa

“A demo proves it can work once; a test proves it keeps working — bring the field, not the bench.”

Through this module, Tessa runs her Field Check on each result: what the bench showed, what the field adds, and what evidence earns her stamp.

7.2 Overview: Verify the Build, Validate the Mission

Two words do most of the work in this module, and they are not synonyms. Verification asks whether the system was built correctly against its specification: does the firmware compute the right value, does the message match the schema, does the function return what the contract promised. Validation asks a harder question: does the built system actually solve the intended problem once it is in someone's home, field, or factory. A device can pass every verification test and still fail validation because the specification itself was wrong for the real world.

Both are required, and for connected hardware the stakes are unusual. A web service that ships a bug can patch it in minutes for every user at once. A deployed IoT device may run for years in a harsh environment, behind intermittent connectivity, owned by someone who never applies an update. The deploy-and-forget reality is why the industry leans on shift-left testing: the widely cited rule of thumb is that a defect grows roughly an order of magnitude more expensive to fix at each later stage, so a problem caught at the desk is cheap and the same problem caught in the field can mean truck rolls, recalls, or a device that simply stays broken.

Before choosing test techniques, inspect Figure 7.1 to locate where a failure can enter the connected system and why one green layer cannot validate the whole mission.

Read Figure 7.1 across hardware, firmware, network, cloud, and application. Each layer introduces a different failure mechanism, while the links let faults propagate. That stack explains why verification must be distributed and why validation still has to observe the complete field outcome.

Why IoT Testing Is Different

Traditional software runs on known servers with a reliable network and instant updates. An IoT product runs across a stack that can each fail independently — sensor and circuit, firmware, radio link, cloud service, mobile app — and a failure in any layer can propagate through the whole system. It also lives in physical conditions a unit test cannot reproduce: temperature extremes, vibration, power brown-outs, RF interference, and users who install it in places the designer never imagined. Because the device may never be reachable again, the testing has to be more thorough before it ships, not after. Read Figure 7.1 in that order.

The One-Minute View

Beginner Examples

Read these examples as a progression from the simplest observation to the boundary that still needs evidence. Begin with a thermostat that passes every unit test in a California lab but fails in Alaska because its antenna degrades below freezing: verification passed, validation did not. Then consider how poorly secured devices can be recruited into IoT botnets; weak pre-release security testing has real downstream consequences. Finish by separating “all tests are green,” a verification claim, from “real users in real conditions got what they needed,” a validation claim. Together, the examples connect an intuitive result to a bounded engineering claim.

Overview Knowledge Check

If you can explain why a green test suite is not the same as a validated product, you have the core idea. Continue to Practitioner for the test levels and how to balance them.

7.3 Practitioner: Test Levels, the Pyramid, and When to Run What

A real strategy is built from named test levels, each answering a different question. Unit tests check individual functions or modules in isolation, often with hardware mocked. Integration tests check that components work together across an interface — firmware to cloud, device to app, one service to another. System (end-to-end) tests exercise the complete path from sensor to dashboard. Acceptance tests confirm the system meets user-facing requirements. Around these sit specialized levels that IoT demands: hardware-in-the-loop, environmental, security, and field or soak testing.

The Testing Pyramid

The pyramid is a heuristic for distributing effort by cost and speed. Fast, cheap, repeatable unit tests form the wide base; slower integration tests sit in the middle; expensive end-to-end tests are the narrow top. A common guideline is roughly two-thirds to four-fifths unit tests, with integration and end-to-end making up the remainder — treat the exact percentages as a target, not a law. The reasoning is behavioral, not numeric: a suite that runs in seconds gets run on every commit and catches bugs while they are cheap, while a suite that takes hours gets skipped, and skipped tests catch nothing.

Before assigning a test cadence, inspect Figure 7.2 to connect each level’s scope to its feedback time and cost rather than treating the pyramid as a fixed percentage recipe.

Read Figure 7.2 from the broad unit-test base through integration to the narrow end-to-end layer. The upward trade-off is slower, costlier, broader evidence. That relationship supports frequent low-level gates while keeping a deliberate set of system and acceptance checks for whole-path claims.

Run the Right Test at the Right Time

Cadence matters as much as coverage. Map each test level to a trigger so the fast checks gate every change and the expensive ones run when they are worth it: unit tests, static analysis, and a build check on every commit; integration and security scans on pull requests; hardware-in-the-loop and extended integration nightly; soak and full regression weekly; environmental, security, and field validation before release. The point is to keep developer feedback in seconds while still exercising the slow, real-world paths on a schedule. Read Figure 7.2 in that order.

Spend Coverage Where Failure Hurts

Coverage is a budget, not a single global number. Allocate it by the impact of failure: safety-critical paths warrant the highest coverage because a defect can cause injury; core logic and protocol implementations need high coverage because bugs there break the product or mishandle edge cases; utility code and hardware-abstraction layers can take lower targets because they are lower risk or are exercised in integration. Track coverage per feature, not as one average that hides an untested critical path behind an over-tested trivial one.

Practitioner Knowledge Check

If you can name the levels, place them in the pyramid, schedule them by cadence, and budget coverage by risk, you can stop here. Continue to Under the Hood for what coverage hides and how validation continues in the field.

7.4 Under the Hood: Coverage Quality, Metrics, and Validation That Continues

The deeper layer is about the gap between a passing suite and a trustworthy system. Three things separate teams that ship reliable hardware from teams that merely have green dashboards: they know what coverage does not measure, they track metrics that predict field behavior, and they treat validation as ongoing rather than a pre-launch checkbox.

Coverage Is Quantity, Not Quality

A suite can execute 90% of the lines while proving almost nothing if it runs the same happy-path inputs over and over. Coverage measures which code ran, not whether the right cases were tried. Test quality comes from deliberate design: boundary-value tests (minimum, maximum, just inside and just outside the limits), equivalence partitioning (one representative from each input class), negative and error-path tests, and error guessing informed by past failures. A high coverage number with no boundary or failure cases is a confidence illusion.

HIL, SIL, and the Bugs Units Cannot See

Some defects only appear with real timing and real I/O. Software-in-the-loop runs the firmware logic against a simulated environment on a host; hardware-in-the-loop runs the actual firmware on the target while sensors and actuators are simulated or stimulated, so the test exercises real interrupts, drivers, and timing. This is where race conditions, uninitialized state, and timing-dependent faults surface. A HIL test that fails intermittently — passing most runs and failing some — is usually not a flaky framework; it is revealing a genuine timing or concurrency bug that unit tests, which run deterministically on a host, were never positioned to find.

Conformance Is Not Interoperability

Two distinct claims are easy to conflate. Conformance testing checks that an implementation follows a specification's required behavior. Interoperability testing checks that it actually works with other independent implementations of that specification. Both can be necessary: a device can conform to a protocol on paper and still fail to interoperate with a particular peer because of optional features, version differences, or ambiguous corners of the standard. Treat a conformance pass and an interoperability pass as separate evidence.

Coverage and pass-rate numbers become useful only when they remain connected to faults and field behavior. Before comparing individual metrics, use Figure 7.3 to see how the evidence moves from development, through testing and deployment, into operational feedback.

Read Figure 7.3 from development to testing, deployment, and the field. Code coverage describes exercised structure, defect density and pass rate describe discovered behavior, and mean time to detect describes how quickly the evidence loop reacts. Field failure rate then tests whether the earlier measures predicted the mission that mattered. No single arrow proves quality; the ordered flow shows where a reassuring local metric can be contradicted by later evidence and sent back into the validation plan.

Metrics That Predict the Field

Statistically Validate the Measurement, Not Just the Average

A mean and a pass/fail count are not the whole story when the claim is about a measurement: that a sensor reads accurately, that two calibration methods agree, or that field data is even the shape statistics can act on. Four checks carry most of that weight, and each has a fast way to apply it and a fast way to be fooled by it.

Confirm the data is normally distributed before trusting standard deviation. Standard deviation only describes spread cleanly when the data is normally distributed: about 68.2% of values fall within one SD of the mean, 95.4% within two, and 99.7% within three. A quick sanity check is mean minus two SD: if that lands on a value the quantity cannot physically take — a battery life of -6 days from a mean of 10 and an SD of 8, for instance — the data is not normal, and SD-based claims built on it should stop there. A stronger check, such as the Shapiro–Wilk test, gives a p-value for the normality assumption itself; treat "the data is normal" as something to test, not assume.

Read spread and outliers with the interquartile range. A box-and-whisker plot reports the median with the first and third quartile points, the interquartile range (IQR) that holds the middle half of the data, and the outliers beyond it — a sturdier summary than mean and SD when a distribution is skewed. Real air-quality particulate-count deployments make the case well: comparable low-traffic environments (a library, a park walk) cluster with a tight IQR near the bottom of the scale, while a high-traffic junction or a car interior shows both a higher median and a long tail of outlier readings. The shape of the box, not just its center, is what tells you the sensor is seeing a genuinely different distribution rather than a shifted average.

Compare two measurement methods with Bland–Altman limits of agreement. When a new or lower-cost sensor needs validating against a reference instrument, plotting the mean of the two against their difference — after confirming the differences are themselves normally distributed — produces a limits-of-agreement interval that says how far apart the two methods can be expected to read. For a difference with mean d = -27.2 and standard deviation s = 34.8, the 95% limits are d − 1.96s = -95.4 and d + 1.96s = 41.1: the candidate method can read anywhere from 95.4 units below to 41.1 units above the reference on a single measurement, even though both are "working." Whether that spread is acceptable is a product decision, but it has to be made from the interval, not from the fact that the two means happen to be close.

Do not mistake correlation for agreement. Two raters, sensors, or classification passes can move together almost perfectly — high correlation — while still disagreeing on the actual value or category, because correlation only measures whether they trend together, not whether they match. Cohen's kappa measures agreement directly: a kappa of 0.25 signals only slight agreement between two independent scorers, even in a case where their outputs correlate strongly, because one rater is consistently offset from the other. When two people, tools, or automated classifiers are meant to reach the same categorical judgment on device or field data, check the agreement statistic, not just the correlation.

Validation Does Not End at Launch

Lab conditions — stable power, clean RF, a perfect network — do not represent the field, so systems validated only in the lab routinely fail once deployed. Field and beta validation at representative sites, with real network conditions and real user behavior, belongs as a mandatory gate before production. Soak testing runs the system continuously for long periods to expose slow leaks, memory growth, and drift. Regression testing re-runs prior tests after every change so a fix in one place does not silently break another. Reserve the lab for reproducible regression; use the field for real-world acceptance, and keep validating as conditions change.

Common Pitfalls

Review these failure modes in the order they can weaken or invalidate the result. Begin with Writing the test plan after development. Plans written afterward describe what was built, not what should have been; write them during requirements and sign them off before coding. Then examine Confusing coverage with quality. Line coverage with no boundary, negative, or error cases proves little; design the cases, do not just chase the number. Then examine Validating only in the lab. Controlled conditions hide the field variability that causes real failures; make field validation a gate. Then examine Ignoring flaky tests. An intermittent failure is usually a real timing or concurrency bug, not noise to retry away. Finish with Letting test plans rot. Requirements drift; tests for removed features pass meaninglessly while new behavior goes untested unless plans are owned and updated. This sequence connects each warning to the evidence a reviewer should demand before accepting the claim.

Under-the-Hood Knowledge Check

At this depth, testing and validation is a discipline of evidence: levels chosen and scheduled deliberately, coverage spent where failure hurts, quality designed rather than counted, and validation that follows the device into the field. A trustworthy review asks which level produced each result, what the coverage did not exercise, and whether the system was proven in the conditions where it will actually run.