Testing & Validation · Study deck
The Testing Pyramid
Imagine a team that can only test the full IoT system by walking to a lab bench, flashing a device, pairing a phone, waiting for cloud messages, and reading a dashboard.
Test Tessa is your guide for this deck.
After studying this chapter
Fast feedback and trustworthy evidence
A useful test strategy connects fast feedback with evidence for each product claim.
- The pyramid can assign checks according to feedback speed.A small firmware check can run after each change, while a sensor-to-dashboard test needs more time and equipment.
- Different test levels can answer different product questions.A unit result covers isolated logic; integration checks interfaces, and hardware-aware checks can expose real sensor assumptions.
- IoT adds evidence that a laptop cannot supply.Heat, weak links, attacks, and field conditions require checks beyond the fast software base of the pyramid.
- A green suite needs carefully bounded claims.Missing failure inputs, an untested interface, or changed hardware can leave a serious defect behind an apparently successful result.
Major section
Slow checks get skipped
The slow bench workflow explains why routine firmware checks belong near the pyramid’s base.
- A repeatable check can test a specific claim about the product.An isolated firmware rule can be checked without waiting for the sensor, radio, cloud, and dashboard to work together.
- Slow setup can delay feedback after small code changes.The chapter’s bench workflow includes flashing, phone pairing, and cloud messages before the team can inspect the dashboard.
- Frequent fast checks catch defects while fixes are cheaper.A suite that takes too long gets skipped, so its first useful failure arrives after the bug has become expensive.
- The cheapest suitable layer can preserve useful feedback.Small checks protect firmware logic, while slower whole-path tests remain necessary for behaviour that crosses the complete system.
Major section
The evidence pyramid
The pyramid connects test scope with cost; read upward from the wide unit base to validation and release.
- The wide base can represent many fast checks of isolated logic.These tests can run after each firmware change because they do not require the complete device-to-dashboard path.
- Integration can add evidence where components exchange data.Moving upward joins more parts, increasing setup and run time while exposing interface disagreements that isolated functions cannot reveal.
- Hardware-aware and system layers can test broader product behaviour.A real sensor can expose a start-up misread that the same firmware function never encounters during laptop tests.
- Validation and release can sit above the faster verification layers.The narrow top reserves physical and field capacity for claims that require real conditions and the complete product.
Major section
Verification and validation answer different questions
Following the design and solving the real problem require different observations.
- Verification can check whether the product follows its intended design.A passing isolated firmware test supports the rule it exercises, without establishing that the deployed product solves the user’s problem.
- Validation can check whether that design works for the real problem.Successful use at a real site answers a broader question than a software suite running on a developer’s laptop.
- IoT hardware can create gaps in ordinary software evidence.Sensor behaviour, radio timing, and the physical assembly cannot be copied perfectly into a laptop test environment.
- A broad base leaves capacity for expensive upper-layer checks.Keeping routine logic checks fast makes room for the hardware, environment, security, and field work that still needs real evidence.
Major section
Seconds at the base, minutes at the top
Test schedules reflect the time and equipment needed to answer each question.
- A fast suite can protect each commit with immediate feedback.The chapter shows thousands of unit tests finishing in seconds beside a handful of end-to-end tests taking minutes.
- The slower whole-path checks can run on a different schedule.Nightly end-to-end tests still matter, but their delay makes them a poor substitute for frequent checks of isolated firmware logic.
- Hardware evidence answers questions that software mocks can miss.The start-up sensor misread can appear only when a hardware-aware test reaches the interface hidden by the laptop setup.
- The proportions can express engineering judgement rather than a fixed rule.The aim is a useful fast base and a manageable slow top, with each layer answering questions the lower layers cannot.
Activity 1 · Match
✎ Place the evidence in the pyramid
I want each test result beside the question it can answer.
Match these checks to unit, integration, hardware-aware, system, or field validation: isolated firmware logic; agreement on a message field; real sensor start-up; sensor-to-dashboard path; use at a real site.
3 minutes · Pen and paper · Answer: Activity 1
Major section
A result needs a review record
The record links a question to follow-through; read from question and layer through observation, limit, action, and retest.
- The opening question determines the test layer and setup.An isolated function needs unit evidence, while a firmware-to-cloud interface question requires components to exchange the relevant message.
- The observation records what happened within that setup.A sensor-to-dashboard success is useful system evidence, but its scope should not silently expand to every hardware or field condition.
- The evidence limit states which claims remain open.A host-based firmware pass cannot establish behaviour at a real sensor interface that the test never exercised.
- The action, owner, and retest trigger make the result usable.When firmware, schema, hardware, or environment changes, the record identifies which earlier conclusion needs another check.
Major section
An inverted pyramid slows the feedback loop
The inverted pyramid delays the very feedback that routine changes need.
- An inverted pyramid can put too much routine work at the slow top.Many end-to-end tests and a thin middle make the suite expensive, flaky, and less likely to run after a small change.
- Realistic tests can lose value when the team rarely runs the suite.A full-system resemblance does not prevent delayed detection when developers skip a long or unreliable suite.
- Lower-level checks can answer many questions more cheaply.Moving isolated rules and interface checks downward restores faster feedback without discarding the whole-path behaviours that still need system tests.
- A useful shape depends on the product’s evidence needs.An integration-heavy strategy can be reasonable, while a base narrower than the slow top signals a feedback problem worth reviewing.
Major section
Symptoms can cross hardware and software boundaries
Interface evidence helps trace a visible symptom back to its source.
- A visible system symptom can originate at another layer.Integration checks at firmware-to-cloud or device-to-app boundaries help locate the disagreement hidden behind a dashboard failure.
- An interface record needs enough context to locate the cause.Naming only the final system failure leaves the team without evidence identifying which component handoff broke.
- Host-based firmware tests depend on their hardware assumptions.A mocked sensor, bus, or bootloader can hide the real interface behaviour touched by a firmware change.
- Hardware-aware checks can close questions left open by those assumptions.The sensor start-up example needs a real or simulated device boundary before a laptop success can support the hardware claim.
Major section
A perfect bench network misses field stresses
Network, physical, and security failures need checks that can actually expose their causes.
- Unreliable connectivity can change the evidence needed for delivery and updates.Loss, latency, and disconnection belong in integration and system tests because a stable bench network makes those operations look easier.
- Physical stresses need evidence from the real assembly.Temperature, vibration, and moisture can cause failures that a software-only test cannot reproduce or close.
- Security testing can ask questions beyond successful normal operation.Assets, trust boundaries, and threats determine abuse checks, because an ordinary functional pass does not describe what an attacker can do.
- Each stress belongs at a layer capable of exposing its fault.The pyramid can stay useful when network, physical, and security questions receive appropriate checks instead of unrelated green software results.
Major section
Changes can invalidate yesterday’s evidence
A recorded trigger prevents yesterday’s success from becoming an unsupported claim about today’s product.
- An old result depends on the setup that produced the observation.Firmware, schema, hardware, and environment changes can remove the assumptions that made an earlier green result meaningful.
- A retest trigger can tie each result to future changes.Recording the relevant behaviour or input boundary prevents a successful run from being reused indefinitely after the product changes.
- Functional success leaves the abuse question unanswered.Security evidence must address assets and threats even when normal sensor reporting still passes the existing suite.
- A trustworthy record makes evidence limits visible to reviewers.The owner and next action keep an untested or changed boundary from disappearing behind the team’s overall green dashboard.
Major section
Coverage cannot choose the right test cases
A high coverage result can coexist with an untested failure input.
- Line coverage records execution rather than the quality of test inputs.The chapter’s 92% result can coexist with a malformed-packet crash because the tests exercised only well-formed sensor data.
- Boundary cases can challenge the limits of accepted inputs.Minimum, maximum, and just-inside or just-outside cases can expose firmware behaviour that repeated happy-path examples never reach.
- Equivalence classes help select meaningfully different inputs.A representative from each input class can broaden the questions asked, rather than repeating similar valid packets to raise the coverage number.
- Negative cases test the failures the product must survive.Missing, malformed, stale, and out-of-range sensor inputs need deliberate checks before the suite supports confidence about bad data.
Activity 2 · Predict
✎ A green suite meets a malformed packet
I have a green coverage report and a device that crashes in the field.
The chapter’s firmware has 92% line coverage, but every input was well formed. Predict whether more happy-path tests settle the malformed-packet failure. Name the missing kinds of cases.
3 minutes · Pen and paper · Answer: Activity 2
Major section
A pass supports only the boundary tested
The tested boundary determines how far a passing result can support the claim.
- A unit pass does not establish agreement between message consumers.A changed consumer-facing message needs integration evidence at the interface, even when the local parser handles its own input correctly.
- A system smoke test leaves storage preservation unproven.Starting the complete path does not establish that a storage migration preserved the required retained state.
- A strong record names the question and the boundary covered.A claim that tests passed can hide the level and the changed behaviour that the run never exercised.
- Evidence must come from the interface affected by the change.Attaching the specific message or storage boundary check prevents an unrelated green result from closing that open question.
Major section
Retest triggers keep results tied to the product
Changes to inputs, messages, storage, and hardware can reopen different results.
- New input classes can reopen the coverage review.Boundary and negative cases need another look when fresh error conditions appear beyond the sensor inputs already tested.
- A changed message schema can reopen the integration result.The consumer and producer must still agree across the revised contract before the earlier interface conclusion is reused.
- Storage and boot changes can reopen the retained-state check.A revised storage layout, migration, or boot state can invalidate a previous result about preserved device state.
- A triggering change can call for a fresh observation.Firmware, hardware, schema, or environment changes belong beside the recorded result so reviewers can identify when another run is required.
Major section
Honest records expose skipped checks
The record must show which important checks did not run.
- A skipped boundary belongs in the visible result record.Burying an unrun storage-migration or interface check in logs lets the green summary imply evidence that does not exist.
- A successful scenario does not cover every related scenario.The observed path supports its own tested conditions, leaving other input classes and physical arrangements to be checked separately.
- A coverage percentage cannot replace deliberate failure-case design.A high number still leaves firmware vulnerable when malformed or out-of-range sensor inputs never reach the test suite.
- A bounded result can state both the observation and the remaining gap.Naming the level, untested boundary, and retest trigger gives a reviewer the evidence needed to judge the claim.
Major section
A balanced pyramid protects useful feedback
A balanced strategy keeps fast checks frequent and expensive evidence focused.
- The wide base makes frequent feedback practical for firmware changes.Many cheap unit checks catch isolated defects early, leaving scarce hardware and field capacity for questions those checks cannot answer.
- The upper layers connect software success with real product conditions.Hardware-aware, environmental, security, and field tests can supply evidence beyond a laptop or a stable bench network.
- The correct test layer depends on the claim being reviewed.A message contract needs integration evidence, while a retained-state migration needs its own specific boundary check.
- Careful cases and retest triggers keep a green suite credible.Boundary and negative inputs reveal missed behaviour, and recorded changes prevent old successes from outliving the assumptions behind their results.
Deck summary
Trust comes from the evidence behind the green result
A green dashboard earns trust through the cases, boundaries, and conditions behind its results.
- A suite can protect the product only when people run the checks.Fast checks on each change can avoid the delayed feedback from a costly inverted pyramid that developers increasingly skip.
- Verification and validation need different supporting observations.Following the design is a narrower claim than solving the real problem at a field site with the deployed assembly.
- Coverage needs boundary and failure cases to support confidence.The malformed-packet crash shows how well-formed inputs can leave dangerous behaviour untouched despite 92% line coverage.
- Each result needs a named level and a reason for retesting.A changed message, hardware interface, storage layout, or environment can require fresh evidence before the team reuses a previous success.
Retrieval practice
Recall check 1 of 3
Test Tessa says: answer from memory, then check your reasoning.
Q1Why does the testing pyramid put most testing effort into fast unit tests at the wide base rather than into end-to-end tests at the top?
Show answer
Answer: A The shape follows behavior: suites that run in seconds get run constantly and catch bugs while they are cheap, while slow suites get skipped and catch nothing.
Retrieval practice
Recall check 2 of 3
Test Tessa says: answer from memory, then check your reasoning.
Q2A team's IoT test suite is mostly slow end-to-end tests with very few unit tests, and developers increasingly skip running it. What does this 'inverted pyramid' cost them, and what is the right correction?
Show answer
Answer: A An inverted pyramid trades fast feedback for slow, brittle runs.
Retrieval practice
Recall check 3 of 3
Test Tessa says: answer from memory, then check your reasoning.
Q3An IoT firmware module reports 92% line coverage, yet a field unit crashes on a malformed sensor packet. Every test used well-formed inputs. What does this reveal about the coverage number, and what is the fix?
Show answer
Answer: A High coverage with only happy-path inputs is a confidence illusion.
Print reference
Answers
Answer key.
- A · The shape follows behavior: suites that run in seconds get run constantly and catch bugs while they are cheap, while slow suites get skipped and catch nothing.
- A · An inverted pyramid trades fast feedback for slow, brittle runs.
- A · High coverage with only happy-path inputs is a confidence illusion.
Print reference
Activity 1 answer
Model answer.
Match: Isolated logic: unit. Message agreement: integration. Real sensor start-up: hardware-aware. Sensor-to-dashboard path: system. Use at a real site: field validation. Each result supports the boundary actually exercised.
Print reference
Activity 2 answer
Model answer.
Predict: More happy-path tests do not settle the failure. Coverage says which lines ran. Boundary, equivalence, and negative or error-path cases must deliberately include inputs such as malformed, missing, stale, and out-of-range sensor data.