Cloud, SDN & Production Architectures · Study deck
Production Fleet Management
Imagine an operator who asks for an update on one hundred pumps.
Cloud Clara is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: When requested and reported state diverge, the system needs a timeout, owner, fallback behavior, escalation rule, and closure proof rather than a silent assumption that the fleet will eventually converge.
- Explain: A retry storm can overload gateways, a rollback can target devices that never applied the update, and an operator can close an incident while the local fallback is still active.
- Explain: For a firmware rollout, include the exact cohort query, excluded devices, requested version, reported version, delivery result, post-change health window, pause threshold, rollback package, and owner.
- Explain: Production management starts with the operating boundary.
Major section
Start With the Fleet State Question
The control screen says the job was sent.
- A fleet needs both the state we want and the state each device reports.
- The first panel shows the fleet to be updated.
- The second records a sent job, which cannot prove what every pump runs.
- The status board then shows differing per-pump reports.
Major section
Start With the Fleet State Question (continued)
The final panel groups the rollout so the operator can review one group before continuing.
- The record below requires fresh device evidence and a safe pause or rollback plan.
- Firmware is the software stored on a device.
- Admit a device only when its identity is known.
- The fleet will change after release.
Major section
Start With the Fleet State Question (continued)
If an update fails, keep the old working state or move the device to a known safe state.
- The record must survive those changes.
- Practitioner covers entry, update, support, retirement, and proof.
- Under the Hood explains state drift, staged rollout, and degraded modes.
- Those deeper controls refine the simple group record.
Major section
Start With the Fleet State Question (continued)
A fleet may be updated, degraded, isolated, misconfigured, or healthy, and the operator needs proof of which state is real.
- They do not turn a sent request into proof of a completed change.
- The owner asks for a change.
- Each device reports its own state.
- The rollout can pause.
Major section
Production Management Operating Model
Production management starts with the operating boundary.
- The boundary names the devices, gateways, remote services, operators, users, and support teams that share responsibility for the live system.
- The example requests an update for 500 devices, but 80 sleeping devices have not reported the new version.
- @fig-production-management-operating-model separates a requested update from a confirmed fleet state.
- It makes the decision concrete by showing production architecture management operating model showing operating boundary, lifecycle control, identity control, update control, observability, fallback behavior, support ownership, so the relationships can be checked before they are turned into a design claim.
Major section
Management Domains
A production architecture has multiple management domains.
- The fleet has an update request, but sleeping devices have not reported the new version and some gateways still buffer messages.
- Observability must compare requested state, reported state, command delivery, buffer state, and support notes before recovery is chosen.
- The update case in @fig-production-management-domains separates a requested action from a finished rollout.
- Names who acts on alerts, what proof closes an issue, and how lessons from incidents update the architecture record.
Major section
Fleet Lifecycle Control
A production system should know which state each asset is in and which transitions require proof.
- At fleet lifecycle control, Fleet Lifecycle Control makes the architecture testable.
- Between them, the fleet lifecycle control relationship becomes visible: Fleet lifecycle control.
- Lifecycle control prevents devices from becoming invisible after installation.
Major section
Update and Configuration Control
Remote change is part of the production architecture.
- The management model should support small changes, large changes, pause decisions, rollback decisions, and operator communication without relying on a single all-or-nothing release.
- This recovery proof, rollback path sequence supports the claim: Update and configuration control.
- Why these devices are included, which devices are excluded, and what dependency or environment facts affect the change.
- What post-change signals prove that the device, gateway, service, and operator workflow still behave as expected.
Major section
Decision Record
Production architecture decisions should leave a record that can be checked after a rollout, update, incident, or ownership change.
- The question names which behavior is being approved or rejected; the operating boundary names devices, gateways, services, users, and support teams in scope.
- Proof considered includes lifecycle, identity, update, observability, fallback, and support results.
- The fields in @fig-production-management-decision-record make a production choice checkable after an incident or rollout.
- A signal, incident, update, or changed dependency becomes the review condition.
Major section
Common Pitfalls
An Edge-Fog-Cloud diagram is not enough.
- The architecture must name who owns health, updates, identity, fallback, and support action.
- A change route without validation, pause, and recovery proof can turn a small defect into a fleet issue.
- Metrics and logs help only when they route to a support owner with a clear action and closure signal.
Major section
Production as Living Boundary
A production architecture is not finished when the first device sends data.
- The prototype question is whether readings arrive.
- The production question is harder: whether each sensor has a known asset record, credential, firmware version, policy, gateway route, support owner, and retirement path.
- Good production evidence is mundane but decisive.
Major section
Production as Living Boundary (continued)
If a gateway buffers reports during a cellular outage, the boundary must show whether the local flood-safe rule continues, which remote commands are paused, and who reviews the delayed data when the link returns.
- The same architecture must survive cohort updates, credential rotation, stale health reports, service dependency failures, support handoffs, and physical replacement without losing proof of ownership.
- Together those artifacts show whether the live fleet is still inside the promised boundary.
- If one artifact is missing, the architecture should say which owner must restore it before the fleet change proceeds.
Major section
Fleet Change Record
Every rollout, update, policy change, or retirement should leave a record that joins the technical state to the operating owner.
- Without that record, production work depends on memory during the exact moments when memory is least reliable.
- For a firmware rollout, include the exact cohort query, excluded devices, requested version, reported version, delivery result, post-change health window, pause threshold, rollback package, and owner.
- The record should separate the platform state from the field state.
Major section
Requested vs Actual State Drift
Production systems carry several kinds of state at once.
- Command state says whether an action was accepted, expired, rejected, or still pending.
- These states drift for ordinary reasons: devices sleep, links fail, gateways buffer, update cohorts pause, clocks skew, identities rotate, and support tickets cross team boundaries.
- A production architecture should make state mismatch visible.
Major section
Requested vs Actual State Drift (continued)
When requested and reported state diverge, the system needs a timeout, owner, fallback behavior, escalation rule, and closure proof rather than a silent assumption that the fleet will eventually converge.
- The implementation detail can vary, but the invariant is the same.
- A device-shadow service, MQTT retained state, gateway queue, update orchestrator, and ticket system may each hold a different piece of truth.
- Without that separation, recovery actions can make incidents worse.
Major section
Requested vs Actual State Drift (continued)
The architecture should define which source is authoritative for each decision and which mismatches are allowed during sleep, outage, or rollout windows.
- A retry storm can overload gateways, a rollback can target devices that never applied the update, and an operator can close an incident while the local fallback is still active.
- Production reviews should therefore ask for invariants rather than screenshots.
- Those invariants make different platforms comparable without pretending every vendor exposes the same controls.
Deck summary
Key takeaways
The control screen says the job was sent.
- The final panel groups the rollout so the operator can review one group before continuing.
- If an update fails, keep the old working state or move the device to a known safe state.
- A fleet may be updated, degraded, isolated, misconfigured, or healthy, and the operator needs proof of which state is real.
- Production management starts with the operating boundary.
Retrieval practice
Recall check 1 of 3
Cloud Clara says: answer from memory, then check your reasoning.
Q1A staged firmware rollout dashboard says 96% complete. The requested version is 2.3, some devices still report 2.2, several gateways have queued commands, and one region has stale health reports. What should production management check first?
Show answer
Answer: A Fleet management distinguishes requested state, reported state, command state, gateway queue state, health freshness, and pause or rollback criteria before accepting a rollout.
Retrieval practice
Recall check 2 of 3
Cloud Clara says: answer from memory, then check your reasoning.
Q2A regional stormwater pilot reports pump current and sump level from three stations through local gateways. The next rollout adds 40 stations, shared maintenance crews, staged firmware updates, and local flood-safe behavior during backhaul outages. Before approval, which missing proof matters most?
Show answer
Answer: B A technically successful pilot is not enough for production sign-off.
Retrieval practice
Recall check 3 of 3
Cloud Clara says: answer from memory, then check your reasoning.
Q3A fleet dashboard shows that an update was requested for 500 devices, but 80 sleeping devices have not reported the new version and some gateways are buffering. What is the production-management response?
Show answer
Answer: A Production management separates requested, reported, command, buffer, and support state so rollout drift is visible and tied to recovery ownership.
Print reference
Answers
Answer key.
- A · Fleet management distinguishes requested state, reported state, command state, gateway queue state, health freshness, and pause or rollback criteria before accepting a rollout.
- B · A technically successful pilot is not enough for production sign-off.
- A · Production management separates requested, reported, command, buffer, and support state so rollout drift is visible and tied to recovery ownership.