31 Simulation and Validation: Test Matrices and Release Gates

31.1 Start With the Decision

A release review needs more than a list of tests. A matrix must link each risk to a test level, result, and safe end state.

31.2 Route Overview

This is part 2 of 2. Review Simulation and Validation: Evidence Strategy for the preceding evidence.

31.3 Learning Objectives

• Build a validation matrix from risks and evidence levels.
• Define negative tests and release gates for IoT systems.

31.4 Chapter Roadmap

• Build the Validation Matrix
• Simulation, HIL, and Bench Roles
• HIL Is Not Always Bigger
• Fault Injection and Negative Testing
• Negative Tests Need Safe End States
• Automation and CI Evidence
• Release Evidence Gate
• Incremental Examples
• Gateway Reconnect Validation
• Try It Now
• Choose the Evidence Level
• Practice Checks
• Match Validation Level to Evidence
• Order Simulation Validation Flow
• Label Validation Evidence Ladder
• Concept Check: Evidence Scope
• Concept Check: Release Gate Decision
• Common Pitfalls
• 1. Testing Tools Instead of Requirements
• 2. Reporting Only Green Builds
• 3. Simulation Is Not Measurement
• 4. Skipping Fault Injection
• 5. Losing Traceability
• Summary
• References
• See Also
• What’s Next
• Key Takeaway

31.5 Build the Validation Matrix

A validation matrix connects requirements, risks, test levels, and evidence. It prevents the test suite from becoming a collection of disconnected scripts.

Validation matrix row:
- claim: device buffers readings while broker is unreachable
- level: virtual integration, then HIL bench
- input: broker unavailable during normal reporting window
- expected: local queue grows within limit, retry backoff starts, no reset occurs
- evidence: simulator log, HIL serial log, broker trace, queue counter
- residual risk: flash wear and power impact need bench measurement

31.6 Simulation, HIL, and Bench Roles

Simulation-driven validation works best when each method has a clear job.

Figure 31.1 shows why a useful HIL rig is defined by control and observability, not by physical size. The real candidate image sits inside an explicit harness boundary that owns stimulus, observation, and reset.

Trace Figure 31.1 through known input, device response, timestamped observation, known-start reset, and repeat. Preserve the image hash, harness version, stimulus, response, and reset result in the run record. A pass or hold is bounded by what the rig can inject and observe; the gap note and retest trigger keep unrepresented field conditions visible.

31.7 Fault Injection and Negative Testing

Production IoT systems fail in ways that happy-path tests never see. Simulation and HIL are useful because they make faults repeatable.

31.8 Automation and CI Evidence

Automated validation should produce evidence, not just a green badge. The release reviewer needs enough context to understand what ran and what was outside the run.

release-evidence/
  build-record.txt
  requirements-trace.csv
  unit-test-report.xml
  emulator-run-log.txt
  virtual-circuit/
  hil-fixture-setup.md
  packet-captures/
  bench-measurements/
  residual-risk-register.md
  release-decision.md

A reliable Automation and CI Evidence review needs the figure Figure 31.2. Its Automation record and candidate evidence in markers reveal where Automated validation should produce reviewable artifacts, not only a pass/fail badge enters the running decision.

Three labelled stops organise Figure 31.2. Automation record uses Automation record to hold review evidence; moving to candidate evidence in shows where it uses candidate evidence in to hold review evidence; ending at Scope shows where it highlights Scope. Together they explain why Automated validation should produce reviewable artifacts, not only a pass/fail badge matters to Automation and CI Evidence.

31.9 Release Evidence Gate

Challenge Release Evidence Gate with the visual in Figure 31.3: Requirements names one checkpoint and Physical Evidence names another. Their relationship frames The release gate combines model evidence, physical evidence, and residual-risk review before a deployment decision.

For Release Evidence Gate, the visual sequence in Figure 31.3 opens with Requirements, where it uses Requirements to state a required condition. Physical Evidence follows to show how it uses Physical Evidence to hold review evidence; Residual Risk then uses Residual Risk to expose invalidating risk. That progression connects The release gate combines model evidence, physical evidence, and residual-risk review before a deployment decision to the next Release Evidence Gate check.

The release gate should make disagreement visible. If simulation passes but HIL fails, the disagreement is useful evidence. It points to a model gap, hardware issue, timing problem, fixture issue, or incomplete requirement.

31.10 Incremental Examples

31.11 Gateway Reconnect Validation

A sensor gateway sometimes stops forwarding readings after a broker outage. The team has a simulator, a gateway development board, a test broker, and a packet capture point.

The validation summary does not say “simulation passed, so release.” It says which claim passed at each level and which physical or deployment assumptions remain.

31.12 Try It Now

Rewrite this weak validation result into a release-ready statement:

“All simulator tests passed, so the device can ship.”

A stronger answer should name the requirement, validation level, model boundary, automated artifacts, HIL or bench evidence, field evidence still needed, residual risk, and the release decision or owner.

31.13 Choose the Evidence Level

For each claim, choose the lowest useful evidence level and the physical evidence needed later:

First: The queue never exceeds its configured limit when the broker is offline.

Next: The device stays under the required current draw during sleep.

Then: The alert reaches the dashboard after a real cellular outage clears.

Use unit test, SIL/emulator, virtual integration, HIL bench, bench measurement, system test, or field pilot.

31.14 Practice Checks

31.15 Common Pitfalls

31.16 Summary

Simulation-driven testing and validation turns model runs into release evidence. A strong process starts with requirements, chooses the evidence level that matches each claim, states model boundaries, runs negative as well as normal scenarios, records automation and HIL artifacts, adds physical and field evidence where needed, and ends with a release decision that names residual risks. The goal is not to maximize the number of tests. The goal is to make each important release claim defensible.

31.17 References

First: Renode Testing Documentation - official Renode documentation for repeatable emulation-based testing.

Next: QEMU System Emulation Documentation - official QEMU documentation for system emulation.

Then: GitHub Actions Documentation - official CI workflow documentation.

After that: PlatformIO Unit Testing - official PlatformIO unit-testing documentation.

Also inspect: pytest Documentation - official Python test automation documentation.

Finally: Robot Framework User Guide - official keyword-driven automation documentation.

31.18 See Also

First: Simulating Hardware Programming: keep virtual hardware claims scoped before validation expands to HIL and bench evidence.

Next: End-to-End Test Strategy: connect release gates to requirement evidence and residual risk.

Then: Network Traffic Analysis: use packet captures and logs when simulation disagrees with field behavior.

After that: Specification Sheet Fundamentals: turn datasheet limits into validation conditions.

31.19 What’s Next

31.20 Key Takeaway

Simulation, testing, and validation form a chain: simulate to explore, test to measure, and validate to prove requirements. Each stage should produce evidence that supports the next design decision.

31.21 Continue Your Route

This final part closes the route from Build the Validation Matrix through Key Takeaway. Return to Simulation and Validation: Evidence Strategy or continue from the design-methodology module index.