Rehearse a fleet rollout and rollback

Operate a deterministic fictional device fleet, evaluate health thresholds, stage firmware through a canary, and preserve the timeline of an automatic rollback.

Cloud Clara: I want you to connect every fleet signal to the gate decision, recovery action, and final reported state., your practice guide

Cloud Clara: I want you to connect every fleet signal to the gate decision, recovery action, and final reported state.
Predict the reading, then compare it with the measurement.

Python 3 in your browser (JupyterLite)

Python · no install

Tier 2 · Web · paste-in setup · No account

Version tested: Python 3.12.7 / Pyodide 0.27.6 in JupyterLite 0.6.4; Chromium 148.0.7778.96; captureSource playwright:jupyterlite. Date: 2026-09-10.

Open the notebook in your browser and run each Python cell; no install or account is needed.

Three ways to run: use JupyterLite here with no install; run main.py locally from the downloadable lab folder; or open the same notebook in Google Colab.

Open in your browser (new tab)

Steps

Screens captured against JupyterLite Python 3.12.7 / Pyodide 0.27.6; Chromium 148.0.7778.96 on 2026-09-10; the tool may have moved on — the text steps are the contract.

  1. Step 1

    Do
    In the JupyterLite notebook, run the Step 1 cell to create the six-device fleet from seed 731.
    You will see
    Input: execute notebook step 1 with the record’s unchanged fixture, seed, and parameters. Observed: Deterministic fleet inventory (seed=731) Observed: pump-01 north 2.3.1 77% 46C 48s Observed: pump-02 north 2.3.1 65% 48C 4s
    Why it matters
    Expected: STEP 1 fleet-inventory devices=6 regions=east,north,west target=2.4.0 from this deterministic notebook step. Interpretation: A rollout decision needs an explicit reported-state snapshot rather than only a fleet total. Action: Run python3 main.py --step 1 to create the six-device fleet from seed 731. Boundary: this local fixture is not a live device, network, or site measurement.

  2. Step 2

    Do
    In the JupyterLite notebook, run the Step 2 cell to evaluate the fleet's health snapshot against three thresholds.
    You will see
    Input: execute notebook step 2 with the record’s unchanged fixture, seed, and parameters. Observed: Rules: temperature >= 80C; heartbeat age > 90s; battery < 20%. Observed: pump-01 46C 48s 77% OK Observed: pump-02 48C 4s 65% OK
    Why it matters
    Expected: STEP 2 health-snapshot healthy=4 alerting=2 from this deterministic notebook step. Interpretation: Putting readings beside thresholds makes cohort exclusion and later alerts auditable. Action: Run python3 main.py --step 2 to evaluate the fleet's health snapshot against three thresholds. Boundary: this local fixture is not a live device, network, or site measurement.

  3. Step 3

    Do
    In the JupyterLite notebook, run the Step 3 cell to fire alerts and exclude unhealthy devices from the rollout.
    You will see
    Input: execute notebook step 3 with the record’s unchanged fixture, seed, and parameters. Observed: 09:00:01 monitor snapshot received devices=6 Observed: 09:00:02 ALERT pump-04 rule=temperature 84C >= 80C Observed: 09:00:02 ALERT pump-06 rule=heartbeat age 125s > 90s
    Why it matters
    Expected: STEP 3 alerts-fired rules=3 excluded=2 devices=pump-04,pump-06 from this deterministic notebook step. Interpretation: The individual rule evidence explains why those devices are excluded while still remaining under monitoring. Action: Run python3 main.py --step 3 to fire alerts and exclude unhealthy devices from the rollout. Boundary: this local fixture is not a live device, network, or site measurement.

  4. Step 4

    Do
    In the JupyterLite notebook, run the Step 4 cell to inspect the rolling firmware plan and safety gate.
    You will see
    Input: execute notebook step 4 with the record’s unchanged fixture, seed, and parameters. Observed: Eligible: pump-01 pump-02 pump-03 pump-05 Observed: Canary cohort: pump-01 pump-03 (one device from north and east) Observed: Next batch: pump-02 pump-05
    Why it matters
    Expected: STEP 4 rollout-plan eligible=4 canaries=2 breach-threshold=20% from this deterministic notebook step. Interpretation: A staged change is safe only when scope, success evidence, breach threshold, and rollback target are decided before execution. Action: Run python3 main.py --step 4 to inspect the rolling firmware plan and safety gate. Boundary: this local fixture is not a live device, network, or site measurement.

  5. Step 5

    Do
    In the JupyterLite notebook, run the Step 5 cell to update and observe the two canary devices.
    You will see
    Input: execute notebook step 5 with the record’s unchanged fixture, seed, and parameters. Observed: 09:01:10 COMMAND pump-01 install=2.4.0 Observed: 09:01:11 ACK pump-01 firmware=2.4.0 error=NO Observed: 09:01:12 COMMAND pump-03 install=2.4.0
    Why it matters
    Expected: STEP 5 canary-result installed=2 errors=0 rate=0.0% decision=PROCEED from this deterministic notebook step. Interpretation: Reported version and error signals prove the canary result before rollout scope widens. Action: Run python3 main.py --step 5 to update and observe the two canary devices. Boundary: this local fixture is not a live device, network, or site measurement.

  6. Step 6

    Do
    In the JupyterLite notebook, run the Step 6 cell to update the next batch and inject pump-05's boot failure.
    You will see
    Input: execute notebook step 6 with the record’s unchanged fixture, seed, and parameters. Observed: 09:02:00 COMMAND pump-02 install=2.4.0 Observed: 09:02:01 ACK pump-02 firmware=2.4.0 error=NO Observed: 09:02:02 COMMAND pump-05 install=2.4.0
    Why it matters
    Expected: STEP 6 rollout-breach errors=1 rate=50.0% decision=ROLLBACK from this deterministic notebook step. Interpretation: The error-rate arithmetic beside the automatic actions proves why containment occurred and which state recovery requests. Action: Run python3 main.py --step 6 to update the next batch and inject pump-05's boot failure. Boundary: this local fixture is not a live device, network, or site measurement.

  7. Step 7

    Do
    In the JupyterLite notebook, run the Step 7 cell to verify rollback acknowledgements and compare requested with reported firmware.
    You will see
    Input: execute notebook step 7 with the record’s unchanged fixture, seed, and parameters. Observed: Rollback verification Observed: 09:02:10 ACK pump-01 firmware=2.3.1 rollback=PASS Observed: 09:02:11 ACK pump-02 firmware=2.3.1 rollback=PASS
    Why it matters
    Expected: STEP 7 rollback-check restored=4 fleet-drift=0 excluded-unchanged=2 from this deterministic notebook step. Interpretation: Recovery is complete only after observed device state matches the rollback request, including unchanged excluded devices. Action: Run python3 main.py --step 7 to verify rollback acknowledgements and compare requested with reported firmware. Boundary: this local fixture is not a live device, network, or site measurement.

  8. Step 8

    Do
    In the JupyterLite notebook, run the Step 8 cell to print the incident record in event order.
    You will see
    Input: execute notebook step 8 with the record’s unchanged fixture, seed, and parameters. Observed: 09:00:02 ALERT pump-04 temperature threshold breached Observed: 09:00:02 ALERT pump-06 heartbeat and battery thresholds breached Observed: 09:01:14 DECISION canary passed; open next batch
    Why it matters
    Expected: STEP 8 incident-closed containment=PAUSE recovery=ROLLBACK evidence=timeline from this deterministic notebook step. Interpretation: A chronological incident record connects signals to operator decisions, recovery evidence, ownership, and the next investigation. Action: Run python3 main.py --step 8 to print the incident record in event order. Boundary: this local fixture is not a live device, network, or site measurement.

Chapter checks

These questions refer to the chapter’s examples. Use the return links to review their answers.

  1. Why do production fleet problems differ from single-device testing?

    Return to the chapter’s knowledge check

  2. After an outage, why is a fixed 60-second reconnect delay (no jitter) still a problem?

    Return to the chapter’s knowledge check

  3. Why can identical X.509 certificate expiry dates across a fleet cause an outage?

    Return to the chapter’s knowledge check

Return to Production Architecture Case Studies · Browse Labs