Chapters

30 Simulation and Validation: Evidence Strategy

design-methodology
simulating
testing
validation

30.1 Start With the Decision

A simulator can prove an algorithm and still miss a broken cable or clock. Each claim needs the cheapest evidence that can truly test it.

30.2 Route Overview

This is part 1 of 2. Continue with Simulation and Validation: Test Matrices and Release Gates.

30.3 Part Objectives

  • Choose simulation, bench, or field evidence for a claim.
  • Separate coverage counts from confidence in system behaviour.

30.4 Chapter Roadmap

  • Start With the Evidence Ladder
  • Validation as Evidence Chain
  • Matrix Claims and Failure Modes
  • Why Evidence Levels Fail
  • In 60 Seconds
  • Prerequisites
  • What This Chapter Adds
  • Build Claim-to-Release Evidence
  • Choose the Right Evidence Level
  • Coverage Is Not the Same as Confidence

30.5 Start With the Evidence Ladder

Firmware is the software stored on a device. A protocol is a shared set of message rules. Picture a team testing a freezer alarm first on a laptop, then on a bench device, and finally inside a real cold room.

Name the claim at each step. A small code test can prove a calculation. A model can replay a lost message. A bench can show real timing and power. A field trial can show walls, weather, people, and service gaps.

Do not let one step claim more than it saw. A clean model can hide a weak antenna, worn storage, bad wiring, or a hard repair job. A field pass can also miss rare faults unless the team adds them on purpose.

This freezer ladder cannot choose every test or prove every release. It does not set sample size, fault range, or pass limits. Those choices follow from risk and the exact product claim.

Use the Practitioner sections to build the claim-to-release record. Use Under the Hood for model limits, repeat tests, injected faults, and remaining risk. The deeper work joins the rungs; it does not make a model equal to the field.

Walk one claim up the ladder. Write the alarm rule. Test the math alone. Test bad input. Test lost input. Run the device code. Replay a link break. Add the real board. Measure the clock. Measure the power. Warm the sensor. Cool the sensor. Fill the storage. Cut the link. Restore the link. Move into the cold room. Try the far wall. Try a busy shift. Ask staff to reset it. Record each result. Mark what still lacks proof. Set the release gate. Keep the failed case. Run it after each change. A higher rung adds evidence. It does not erase lower tests.

Picture a team moving from simulated firmware to a bench rig, then to a field pilot, without losing the reason each step exists. Simulation-to-validation work builds an evidence ladder: what the model can prove, what emulation or hardware-in-the-loop must add, what bench tests still need to confirm, and what only installed field behavior can settle.

30.6 Learning Objectives

By the end of this chapter, you will be able to:

  • Build a simulation-driven validation plan that connects requirements to unit, simulation, emulation, hardware-in-the-loop, system, and field evidence.
  • Choose the right validation level for firmware logic, protocol behavior, physical timing, power, environmental, and release claims.
  • Define model boundaries so simulation results do not overclaim what was proven.
  • Package automated test results, hardware runs, logs, traces, and residual risks into a release evidence record.
  • Plan regression testing and fault injection so future changes do not silently break known behaviors.

30.7 Validation as Evidence Chain

Simulation-driven validation works when each evidence level answers a different part of the release claim. Unit tests can prove parser, state-machine, and calculation behavior. Software-in-the-loop and emulators can replay deterministic firmware scenarios. Virtual integration can exercise boards, sensors, displays, protocol partners, and fault inputs. Hardware-in-the-loop, bench tests, and field pilots prove the physical and deployment behavior that models cannot.

The validation plan should say which claim is being checked, which evidence level can support that claim, and which claims remain open. A gateway-reconnect simulator can support retry and buffering logic. It does not prove flash wear, cellular outage behavior, power draw, RF range, broker load, installer workflow, or field recovery until those are checked with the right physical or operational evidence.

The Validation as Evidence Chain argument uses Figure 30.1 to compare Simulation-to-Validation Evidence Ladder. Look next for Requirement claim before accepting Simulation-driven validation is defensible when model evidence, physical checks, and the final release record stay connected on one evidence ladder as a design claim.

Six-rung simulation validation evidence ladder linking a requirement claim to unit and SIL checks, virtual integration, HIL bench evidence, system and field checks, and a release record.
Figure 30.1: Simulation-driven validation is defensible when model evidence, physical checks, and the final release record stay connected on one evidence ladder.

Look across Figure 30.1 in the order Simulation-to-Validation Evidence Ladder, Requirement claim, Release record. Simulation-to-Validation Evidence Ladder uses Simulation-to-Validation Evidence Ladder to hold review evidence; the transition to Requirement claim uses Requirement claim to state a required condition; the final Release record marker uses Release record to hold review evidence. That order ties Simulation-driven validation is defensible when model evidence, physical checks, and the final release record stay connected on one evidence ladder back to Validation as Evidence Chain.

  • Model evidence: Unit tests, SIL, QEMU, Renode, virtual circuits, simulated brokers, and scripted fault inputs.
  • Physical evidence: HIL fixtures, serial logs, packet captures, logic-analyzer traces, current profiles, RF surveys, and environmental tests.
  • Release evidence: Requirement trace, failures, mitigations, residual risks, monitoring actions, owners, and rollback conditions.

30.8 Matrix Claims and Failure Modes

Start the matrix with claim language that can fail. “Gateway reconnects safely after broker outage” is testable. “Connectivity works” is too vague. Then assign evidence levels: unit tests for queue bounds and retry state, SIL for protocol state, virtual integration for simulated broker refusal and recovery, HIL for the gateway board under controlled network conditions, and field evidence for real backhaul, broker, cloud, and operator behavior.

Use concrete automation artifacts. PlatformIO, pytest, Robot Framework, GitHub Actions, Renode tests, QEMU scripts, MQTT brokers, packet captures, and HIL controller scripts should all preserve versioned inputs and outputs. Each run needs firmware commit, board target, dependency versions, model or fixture version, input scenario, expected result, observed result, skipped cases, and who owns any failure.

For a practical gateway release, write the matrix row before choosing the tool. If the claim is “buffered readings survive a 20-minute MQTT broker outage,” the fast checks cover queue limits, timestamp ordering, retry backoff, and serialization. The virtual run can drive CONNACK delay, DNS failure, credential rejection, and broker recovery. The HIL bench then runs the real board with the production firmware, a test broker, serial capture, packet capture, and controlled power input. The field pilot finally checks cellular behavior, operator response, cloud dashboards, and alerting under the actual deployment path.

Keep every artifact reviewable by a second engineer. A useful evidence packet includes the requirement ID, test case name, firmware image hash, board revision, model version, fixture calibration note, input script, observed log, failed assertion, skipped test reason, and residual-risk owner. If a simulated broker failure passes but the HIL run resets during reconnect, the matrix should preserve both results. The next action may be firmware backoff, flash-write reduction, power-rail measurement, watchdog tuning, or a narrower release claim.

  1. Pick the weakest claim first. Validate the failure mode most likely to break the release: low supply, broker outage, bad sensor data, full queue, interrupted update, or credential rejection.
  2. Assign the lowest useful level. Use fast unit or SIL checks for pure logic, then move to virtual, HIL, bench, or field only when the claim needs that evidence.
  3. Keep disagreements visible. If simulation passes and HIL fails, record the model gap, hardware issue, fixture issue, requirement gap, or release decision instead of smoothing it away.

30.9 Why Evidence Levels Fail

A failed unit test usually points to code logic. A failed emulator run may point to boot flow, configuration, scheduling, or protocol handling. A failed virtual integration run may point to pin mapping, peripheral assumptions, command sequence, or model setup. A failed HIL run may point to physical timing, voltage, sensor behavior, watchdog behavior, fixture calibration, or real firmware build differences. A failed field pilot may point to installation, backhaul, cloud latency, user workflow, support process, or environmental variation.

That difference matters because release gates need the right owner and next action. A low-supply reset failure may belong to firmware, hardware design, power supply, battery sizing, or test-fixture calibration. A reconnect failure may belong to MQTT session handling, TLS certificate state, DNS, gateway queue policy, cellular backoff, cloud broker limits, or application stale-data rules.

Root-cause evidence also has different clocks. CI results are usually tied to a commit and can rerun in minutes. HIL results are tied to fixture wiring, calibration state, firmware image, device serial number, board revision, and instrument setup. Field evidence is tied to installation date, firmware rollout cohort, radio environment, SIM profile, cloud region, broker limits, user action, and support process. A defensible release record keeps those clocks separate, so a later regression can identify whether the changed variable was code, model, fixture, hardware, site condition, or operations policy.

When model and physical evidence disagree, avoid treating one as automatically authoritative. A model may be too ideal, but the bench can also be wrong because of a stale firmware image, a bad fixture cable, a misconfigured power supply, or an uncalibrated sensor source. The technical review should ask what each evidence level actually observed: firmware state, packet sequence, boot reason, queue depth, current profile, bus trace, broker log, cloud event, support ticket, or rollback action. That observation-first approach turns disagreement into a repair path rather than a debate about which tool to trust.

  • State evidence: Firmware state, queue depth, retry timer, watchdog status, update version, and boot reason.
  • Interface evidence: MQTT/CoAP exchange, TCP reset, TLS handshake, DNS failure, UART/SPI/I2C trace, and HIL input script.
  • Operational evidence: Field logs, incident timeline, support action, monitoring alert, rollback trigger, and residual-risk owner.

In 60 Seconds

Simulation-driven validation is not “test everything in a simulator.” It is a layered evidence system. Fast tests check logic, simulation and emulation check repeatable scenarios, hardware-in-the-loop checks real firmware with controlled inputs, bench tests measure physical behavior, and field pilots reveal deployment conditions. The quality of the process comes from tying each test to a requirement, recording what the model can and cannot prove, and making release decisions from the combined evidence.

30.10 Prerequisites

You should already be comfortable with:

30.11 What This Chapter Adds

The previous chapter focused on using virtual hardware to develop and exercise firmware. This chapter focuses on validation strategy: how to combine simulation, emulation, HIL, bench hardware, and field evidence without confusing one kind of evidence for another.

Trace

Requirement first

Every test should point to a requirement, risk, interface, or release claim. Untied tests become noise.

Layer

Use the right evidence level

Fast software tests, simulated peripherals, HIL rigs, bench measurements, and field pilots answer different questions.

Scope

Document model limits

A simulated pass may prove logic while leaving timing, power, RF, analog, or environmental claims open.

Gate

Release from evidence

A release gate should show passing tests, failed tests, mitigations, residual risk, and owners for unresolved work.

30.12 Build Claim-to-Release Evidence

Simulation-driven validation starts with a claim that could fail. The team chooses the fastest evidence level that can check that claim, records the model boundary, adds physical or field evidence where the model stops, and only then makes a release decision.

Use the evidence ladder introduced in the overview as a review pattern. A project may move up and down as risks are discovered. The key is to state what each layer proves.

1. RequirementName the behavior, interface, safety condition, service expectation, or environmental claim being validated.
2. Unit and SILCheck pure logic, state machines, parsers, scheduling, and error handling with fast automated tests.
3. Virtual integrationRun firmware against virtual boards, simulated peripherals, protocol partners, and fault inputs.
4. HIL benchRun production-like firmware on physical hardware while controlled equipment injects inputs and observes outputs.
5. System and fieldValidate power, RF, timing, thermal, enclosure, cloud, user, and deployment behavior in real conditions.
6. Release recordPackage evidence, residual risks, open defects, waivers, and signoff criteria for the release gate.

30.13 Choose the Right Evidence Level

Use simulation where it is strong. Use hardware where physics matters. Use field trials where deployment assumptions matter.

Validation Level
Best For
Weak For
Evidence Artifact
Unit test
Pure functions, parsers, calculations, state transitions, serialization, and boundary conditions.
Pin timing, real peripherals, power, RF behavior, and deployment conditions.
Test report, coverage summary, failing assertion, and requirement link.
SIL or emulator
Firmware services, boot flow, protocol stacks, deterministic scenarios, and repeatable automation.
Exact electrical behavior, physical timing margins, analog sensor accuracy, and board assembly issues.
Emulator script, firmware image, logs, model boundary, and CI artifact.
Virtual integration
Firmware with virtual boards, sensors, displays, buttons, serial logs, and protocol partners.
Real noise, loading, current draw, RF range, thermal behavior, and manufacturing variability.
Project files, pin map, virtual wiring, screenshots, serial logs, and run notes.
HIL bench
Production firmware on physical hardware with controlled sensor, network, power, or actuator inputs.
Large-scale field conditions, user behavior, installation effects, and long deployment variation.
Test controller script, equipment setup, calibration note, logs, traces, and pass/fail report.
System and field
End-to-end device, gateway, cloud, mobile, installer, environment, and operations behavior.
Fast debugging and tightly controlled reproduction of every root cause.
Field logs, incident timeline, trace packet, survey notes, firmware version, and release decision.

High line coverage can still miss the wrong behavior. A meaningful test has a requirement, input condition, expected result, assertion, and evidence that would fail if the behavior were wrong.

30.14 Continue to the Next Part

Carry this evidence into Simulation and Validation: Test Matrices and Release Gates, which begins with Build the Validation Matrix.