Rehearse a fleet rollout and rollback

Steps

Screens captured against JupyterLite Python 3.12.7 / Pyodide 0.27.6; Chromium 148.0.7778.96 on 2026-09-10; the tool may have moved on — the text steps are the contract.

1. 1 Step 1

   Do

   Run `python3 main.py --step 1` to create the six-device fleet from seed 731.

   You will see

   A table lists pump-01 through pump-06 with region, reported firmware, battery, temperature, and heartbeat age.

   Why it matters

   A rollout decision needs an explicit reported-state snapshot rather than only a fleet total.

2. 2 Step 2

   Do

   Run `python3 main.py --step 2` to evaluate the fleet's health snapshot against three thresholds.

   You will see

   The rules are printed above six rows: four devices are OK, while pump-04 and pump-06 show ALERT for their measured values.

   Why it matters

   Putting readings beside thresholds makes cohort exclusion and later alerts auditable.

3. 3 Step 3

   Do

   Run `python3 main.py --step 3` to fire alerts and exclude unhealthy devices from the rollout.

   You will see

   Timestamped alert lines name pump-04's temperature breach and pump-06's heartbeat and battery breaches before the operator exclusion event.

   Why it matters

   The individual rule evidence explains why those devices are excluded while still remaining under monitoring.

4. 4 Step 4

   Do

   Run `python3 main.py --step 4` to inspect the rolling firmware plan and safety gate.

   You will see

   The eligible, canary, and next-batch cohorts are named beside the success signal and `pause when cohort error rate > 20%; restore 2.3.1` rule.

   Why it matters

   A staged change is safe only when scope, success evidence, breach threshold, and rollback target are decided before execution.

5. 5 Step 5

   Do

   Run `python3 main.py --step 5` to update and observe the two canary devices.

   You will see

   Commands and acknowledgements for pump-01 and pump-03 report firmware 2.4.0 with no errors; `0.0% <= 20.0% -> PROCEED` opens the next batch.

   Why it matters

   Reported version and error signals prove the canary result before rollout scope widens.

6. 6 Step 6

   Do

   Run `python3 main.py --step 6` to update the next batch and inject pump-05's boot failure.

   You will see

   One of two batch devices fails, making `50.0% > 20.0% -> BREACH`; the timeline immediately pauses rollout and requests rollback to 2.3.1 for four devices.

   Why it matters

   The error-rate arithmetic beside the automatic actions proves why containment occurred and which state recovery requests.

7. 7 Step 7

   Do

   Run `python3 main.py --step 7` to verify rollback acknowledgements and compare requested with reported firmware.

   You will see

   Four rollback acknowledgements precede a six-row table in which requested and reported versions are 2.3.1 and every drift value is NO.

   Why it matters

   Recovery is complete only after observed device state matches the rollback request, including unchanged excluded devices.

8. 8 Step 8

   Do

   Run `python3 main.py --step 8` to print the incident record in event order.

   You will see

   The timeline moves from ALERT through DECISION, ERROR, CONTAIN, RECOVER, and VERIFY, then assigns the release-manager and a quarantine follow-up.

   Why it matters

   A chronological incident record connects signals to operator decisions, recovery evidence, ownership, and the next investigation.