9 Transport Error Handling
9.1 Start With One Error State
Name the Failed State Before Choosing Recovery
Picture a remote pump that receives a start command just as its link drops. The control screen retries, but the operator cannot tell whether the first command ran. A blind retry could start the pump twice, while silence could leave it stopped.
An actuator is a part that turns a command into physical action. A gateway is the device that joins the pump network to another network. Trace the command across both, and give every attempt an identity, send time, result, and final physical state. Decide which evidence allows a retry and which evidence requires a safe stop or a human check.
Force one error at a time. Lose the reply, repeat the command, damage a record, restart the gateway, and reconnect after the command has expired. Check that the system labels the state instead of guessing. Record the bounded action, the owner, and the event that triggers another test.
This first exercise does not remove every source of failure. Devices and links can fail in combinations. The deeper sections show how detection, backoff, duplicate control, checks, and recovery records support a defensible error-handling plan.
Reliability work begins when a message path lands in an unwanted state. Name the state first: timeout, duplicate, checksum failure, reconnect, partial command, or stale acknowledgement. Once the state is named, recovery can be reviewed as a bounded contract with evidence, owner, and retest trigger.
9.2 Overview: Reliable Error Handling Leaves a Bounded State
Transport reliability is not the absence of errors. Links drop frames, acknowledgements are missed, duplicate messages arrive, checksums fail, sessions expire, and gateways restart. Reliable handling means the system detects the condition, classifies it, takes a bounded action, and leaves evidence of the final state.
A useful error record does not stop at "timeout" or "retry." It names the affected boundary, explains whether the condition is transient or persistent, records the decision, shows the retry or timeout limit, and states what must be retested when transport behavior changes.
For example, gateway-west sends close command C-841 to actuator-17 and expects an acknowledgement before the command window closes. If the first response is missing, the record should show whether the gateway retried, how long it waited, and what happened after the final attempt. A bounded outcome might be: two retries with backoff, command marked unresolved, valve state left at last-confirmed open, operator alarm raised, and later duplicate acknowledgements ignored unless they match the active command id. That evidence is much stronger than a retry counter alone because it proves the system did not silently claim closure, loop forever, or apply stale messages after recovery.
The same record should also state the non-claim. It does not prove that the radio path is healthy forever, that the actuator moved, or that the root cause was known. It proves a narrower handling behavior: when this boundary saw this abnormal condition, the implementation moved to a named state and exposed enough evidence for an operator or reviewer to decide the next step. That boundary keeps recovery evidence useful without over-selling it.
The error state becomes actionable only when the review follows it all the way to a bounded outcome. Figure 9.1 gives that route before the chapter separates detection, decision, and stopping rules.
Follow Figure 9.1 from the observed condition into classification, then through the chosen recovery action and its evidence. Finish at the named final state and retest trigger. This order matters because a retry or reset is not itself success; the loop closes only when the system exposes what state remains and what change would invalidate the conclusion.
Detect
Record the observable condition: timeout, missing acknowledgement, duplicate sequence, corrupt input, stale session, or rejected command.
Decide
Classify the condition before acting. The system may accept, reject, retry, hold, reset, reconnect, report, or fail safe.
Bound
Name the stopping condition, final state, owner, and retest trigger so recovery behavior does not become an unbounded loop.
9.3 Practitioner: Build a Decision Record for Each Error Path
Start with one transport boundary: device to gateway, gateway to cloud, command response, connection lifecycle, queue handoff, or session resume. Then walk the abnormal condition through one review record. The goal is to show why the system did what it did and what state remains afterward.
Keep the evidence at the same grain as the action. A packet capture, gateway log, queue event, device state sample, and operator alarm may each answer a different part of the same recovery decision.
The loop needs a durable record when different logs and state observations contribute to one decision. Inspect Figure 9.2 to see how those fragments are kept at the same boundary as the recovery action.
Read Figure 9.2 from the observed condition and affected boundary into classification and recovery evidence. Then check the action limit, final state, owner, and retest trigger. Keeping those fields together prevents a timeout counter or reset log from masquerading as complete recovery proof and connects the practitioner record to the chapter’s state-machine contract.
Review habit: do not approve an error path until the retry or recovery action has a limit, a final state, and a retest trigger.
Duplicate Message Example
A gateway receives a message with a sequence value that was already accepted. A weak record only says "duplicate." A stronger record shows the previous accepted state, whether the duplicate changed state, whether an acknowledgement was sent, and what condition would reopen sequence-handling review.
9.4 Under the Hood: Error Handling Is a State-Machine Contract
Under the hood, transport error handling is a state transition contract. A message may move from pending to acknowledged, timed out, retried, rejected, unresolved, or failed safe. A connection may move from open to suspect, reconnecting, resumed, or closed. The reliability question is whether those transitions are explicit, bounded, and observable.
TCP, UDP, CoAP, MQTT, DTLS, and application protocols expose different signals. TCP may report connection failure or reset. UDP may expose no delivery signal unless the application adds one. CoAP, MQTT, and application command paths often add acknowledgements, message IDs, sequence values, retry policies, and session state. The review record should match the actual signals available at that boundary.
The state-machine contract should name guards and side effects, not just states. A timeout guard might move a command from pending to retrying only while the retry budget remains. The final timeout might move it to unresolved, emit an alarm, preserve last-confirmed state, and reject late acknowledgements that no longer match the active command. Without those guards, a diagram can look tidy while the implementation still accepts stale, duplicated, or out-of-order recovery events.
Idempotency
Repeated delivery must not create repeated physical action unless the command was designed for that behavior.
Ordering
Late or reordered messages need sequence, timestamp, or state evidence before they are accepted.
Backoff
Retry timing should reduce contention and power cost instead of synchronizing many devices into another burst.
Fail Safe
When evidence is missing, the system needs a defined hold, alert, last-known-good, or safe-state decision.
Failure mode: treating a log entry as a recovery action. A useful record shows what state changed, what state did not change, and when the same path must be retested.
9.5 Summary
Reliable handling means more than noticing an error or attempting a retry. Follow each abnormal condition from detection through classification and bounded action to the final exposed state, then record the evidence, owner, and change that reopens review. The summary below connects timeout, duplicate, corrupt-input, reconnect, and fail-safe behavior through that same state-machine contract.
Keep the conclusion deliberately narrow. One observed recovery proves the configured path under the recorded condition; it does not establish a permanent root cause or guarantee every future link. That discipline is why limits, final state, operator visibility, and retest ownership belong beside the technical mechanism in the closing record.
- Reliability error handling starts from an observable condition, not a guessed cause.
- A reviewable record names the affected boundary, classification, decision, limit, recovery evidence, final state, owner, and retest trigger.
- Retry, reset, reject, hold, reconnect, and fail-safe actions need explicit stopping conditions.
- Duplicate, stale, reordered, corrupt, and timed-out inputs must not change state silently.
- Transport protocol choice affects which signals are available, but application state handling still needs proof.
9.6 Key Takeaway
Error handling is part of protocol design. Decide how the system retries, rejects, holds, alerts, resets, or fails safe before deployment, and keep enough evidence to prove the resulting state.