8 Service Resilience: Breakers, Retries, and Timeouts
8.1 Start With the Decision
A blind retry can run one actuator command twice. Breakers, backoff, and deadlines must preserve the command’s safety rule.
8.2 Route Overview
This is part 2 of 2. Review Service Resilience: Operating Context and Control Map for the preceding evidence.
8.3 Learning Objectives
- Configure circuit-breaker states and recovery tests.
- Design safe retries, backoff, jitter, and deadlines.
8.4 Chapter Roadmap
- Failure Modes in IoT Services
- Circuit Breaker Pattern
- Knowledge Check: Circuit Breaker Behavior
- Retry, Backoff, and Jitter
- Retry Rule for IoT Commands
- Knowledge Check: Retry Safety
- Timeouts and Deadlines
- Checkpoint: Retry and Deadline Budget
- Bulkhead Pattern
- Knowledge Check: Bulkhead Isolation
- Fallback and Graceful Degradation
- Try It: Degraded-Mode Test Card
- Combining the Patterns
- Knowledge Check: Pattern Ordering
- Checkpoint: Ordered Resilience Controls
- Architecture Review Checklist
- Common Pitfalls
- Label the Diagram
- Code Challenge
- Summary
- Key Takeaway
- Knowledge Check
- Quiz: SOA Resilience Patterns
- Interactive Quiz: Match Resilience Pattern Concepts
- Interactive Quiz: Sequence the Steps
- Try It Yourself: Resilience Review Note
- References
- What’s Next
- Navigation
8.5 Failure Modes in IoT Services
IoT platforms fail differently from ordinary web applications because they combine remote devices, lossy networks, cloud APIs, queues, databases, and sometimes physical actuators.
8.5.1 Slow Dependency
A database query, external API, model inference endpoint, or message broker takes much longer than expected. The service appears alive, but callers spend their resources waiting.
8.5.2 Retry Storm
Devices, gateways, or services retry at the same schedule after an outage. A recovering dependency receives synchronized traffic and fails again.
8.5.3 Unsafe Duplicate Command
A lost acknowledgement makes a caller retry a command that already succeeded. This is dangerous for actuators, locks, alarms, payments, or provisioning operations.
8.5.5 Partial Cloud Loss
Cloud services are unreachable but the edge gateway, local device network, or cached configuration still works. The system needs a local or degraded operating mode.
8.5.6 Poison Message
One malformed message fails repeatedly and blocks a queue consumer. The pipeline needs retry limits and a dead-letter route.
8.6 Circuit Breaker Pattern
A circuit breaker is a state machine around a dependency call. It is not a retry mechanism. It decides whether the caller should attempt the dependency now or fail fast and use fallback behavior.
Inspect Figure 8.1 as three explicit transitions. Closed admits normal calls and counts slow or failed outcomes; crossing the threshold opens the circuit and returns fallback without calling the dependency; after cooldown, half-open admits only limited probes whose results close or reopen it.
The state diagram in Figure 8.1 makes recovery bounded. A timer alone does not declare the dependency healthy: probe outcomes do. Operators therefore need the failure window, open time, probe count, fallback rate, and state transitions in telemetry before they can distinguish protection from an over-sensitive breaker.
8.6.1 Closed
Calls pass through. The policy counts failures and slow responses over a rolling window or consecutive sequence.
8.6.2 Open
Calls do not reach the dependency. The caller returns fallback behavior quickly and protects its own resource pool.
8.6.3 Half-Open
After a cooldown, the policy allows a small number of probes. Successful probes close the circuit; failed probes reopen it.
Do not copy someone else’s breaker thresholds. Choose the failure window, threshold, cooldown, and half-open probe count from your own request rate, downstream SLO, timeout budget, and false-positive tolerance.
8.6.4 Circuit Breaker Review Questions
What counts as a failure? Include hard errors, timeouts, rejected calls, and responses that are technically successful but too slow for the caller’s deadline.
What is the fallback? A breaker without fallback often just changes a slow failure into a fast error. That can still be useful, but the user experience should be intentional.
How does recovery get tested? Half-open probes should be limited so a recovering dependency is not immediately flooded.
Who sees the state? Expose open circuits, failure rates, fallback counts, and probe outcomes in service dashboards and alerts.
8.7 Retry, Backoff, and Jitter
Retries are useful only when the failure is likely transient and the operation is safe to repeat. A retry policy should answer four questions before it is enabled.
Read Figure 8.2 along its timeline. Normal device traffic reaches capacity before the dependency failure; synchronized clients then retry on the same schedule, creating a second spike above server capacity precisely while the service is trying to recover.
The retry timeline in Figure 8.2 shows overload caused by coordination, not one unusually busy client. Exponential backoff lowers attempt frequency, jitter spreads attempts in time, a retry budget caps amplification, and idempotency prevents a delayed acknowledgement from turning the recovery wave into duplicate physical commands.
8.7.1 Is the operation safe to retry?
Reads are usually safe. Writes need idempotency keys, command sequence numbers, or server-side deduplication. Physical commands deserve extra caution.
8.7.2 Is there enough deadline left?
Retries must fit inside the caller’s total deadline. A retry that starts after the user-facing budget is gone only creates extra load.
8.7.3 Does the error look transient?
Connection resets, rate limits, and temporary unavailability can be retry candidates. Invalid input, authorization failure, and missing resources usually are not.
8.7.4 Are retries desynchronized?
Exponential backoff reduces pressure. Jitter spreads devices and services so they do not all retry at the same instant.
def retry_delay(base_seconds, attempt, cap_seconds, jitter_fraction, random_value):
"""Return a capped exponential backoff delay with caller-provided randomness."""
backoff = min(base_seconds * (2 ** attempt), cap_seconds)
jitter = backoff * jitter_fraction * random_value
return backoff + jitterDo not blindly retry commands that may change the physical world. If an actuator command might already have succeeded, retry only when the command has an idempotency key or device-side command sequence number that prevents duplicate execution.
The retry questions establish the danger of repeating work. The next control is the clock: without a deadline, even a “safe” retry can arrive too late to help.
8.8 Timeouts and Deadlines
A timeout caps one operation. A deadline caps the whole user-visible or workflow-visible budget. Resilient services usually need both.
1. Caller budget Start with the user, device, or workflow SLO. A status request may have a small budget; a firmware transfer may have a larger one.
2. Per-hop budget Allocate time to each dependency call. Leave room for serialization, network delay, fallback, and response handling.
3. Retry budget Only retry while enough budget remains for another useful attempt and fallback.
4. Cancel work Propagate cancellation so downstream services stop doing work after the caller no longer needs the result.
8.8.1 Timeout Too Short
Normal network variance looks like failure. The service may open circuits or use fallback unnecessarily.
8.8.2 Timeout Too Long
Workers, sockets, queue slots, and memory are held by work that the user has already stopped waiting for.
Checkpoint: Retry and Deadline Budget
You now know:
- Retries need three gates: the operation is safe, the error is transient, and the deadline still has room for another useful attempt.
- Exponential backoff and jitter prevent device fleets from retrying at the same instant during recovery.
- Connect timeout, read timeout, and overall deadline should be separate signals so operators can see where time was lost.
8.9 Bulkhead Pattern
Bulkheads limit the blast radius of a dependency, tenant, device fleet, or workload class. The goal is simple: one failing dependency should not consume all resources that healthy work needs.
Inspect Figure 8.3 across the telemetry, analytics, and notification pools. The exhausted analytics partition rejects or queues analytics work inside its own budget, while the telemetry and notification partitions retain workers and connections for their separate service obligations.
The bulkhead diagram in Figure 8.3 is useful only when its limits are enforced before shared capacity disappears. Review pool size, queue bound, rejection behavior, tenant or workload key, and fallback separately; then prove that saturating analytics leaves telemetry ingestion and notification delivery within their accepted budgets.
8.9.1 Worker Pools
Separate workers for telemetry ingestion, command dispatch, alerts, and analytics. A blocked analytics dependency should not stop command dispatch.
8.9.2 Connection Pools
Give each downstream database, API, or broker its own connection budget. A slow dependency cannot take every socket.
8.9.3 Queues
Separate high-priority command queues from bulk telemetry or batch analytics. Queue limits should shed low-value work first.
8.9.4 Rate Budgets
Apply per-tenant, per-device-class, or per-service limits so one noisy fleet does not starve everyone else.
8.9.5 Service Partitions
Separate life-safety or control-plane capabilities from convenience or analytics features where the risk justifies it.
8.9.6 Dead Letter Routes
Move poison messages aside after bounded attempts so they do not block the main processing stream.
Bulkheads keep one queue, pool, or workload from exhausting the caller. Once the blast radius is contained, the design still needs to say what the user or device receives while degraded.
8.10 Fallback and Graceful Degradation
Fallback is not the same as hiding the problem. A good fallback is honest, useful, and bounded.
8.10.1 Last Known State
Show cached telemetry with a visible freshness indicator when live reads are unavailable.
8.10.2 Queue for Later
Accept a safe idempotent request, place it in a durable queue, and report that execution is pending.
8.10.3 Local Control
Let gateways or devices execute critical local rules when cloud coordination is unavailable.
8.10.4 Read-Only Mode
Disable writes while allowing status, history, and diagnostics to remain available.
8.10.5 Manual Confirmation
For high-risk physical actions, require an operator to confirm state rather than silently retrying.
Before a resilience pattern is accepted, write one test card for the degraded path it creates:
- Trigger: name the exact condition, such as cloud API down, dependency timeout, full queue, or open circuit.
- Allowed behavior: state what the user, device, or service may still do while degraded.
- Blocked behavior: state which write, command, or automation must stop instead of retrying silently.
- Freshness signal: decide how the interface or log shows cached data, queued work, read-only mode, or explicit unavailability.
- Recovery check: define the first healthy signal that lets the service leave degraded mode without a retry surge.
The fallback is ready only when this card can be exercised in a test environment and the result is visible in metrics, logs, and the user-facing workflow.
8.11 Combining the Patterns
These controls should be ordered deliberately. One practical service-call flow is:
1. Start deadline Attach a total budget and cancellation signal.
2. Check bulkhead Reserve from the dependency-specific pool or reject quickly if that pool is full.
3. Ask breaker If open, skip the dependency and use fallback.
4. Attempt call Use per-attempt timeout and collect latency, status, and error signals.
5. Retry if safe Retry transient failures only while deadline, retry budget, and idempotency rules allow it.
6. Record outcome Update breaker state, metrics, traces, logs, and user-facing degraded-mode counters.
Checkpoint: Ordered Resilience Controls
You now know:
- Reserve bulkhead capacity before sending dependency traffic, then ask the circuit breaker whether the call should be admitted.
- A first attempt that fails after 1.8 seconds inside a two-second deadline usually leaves no useful retry budget.
- Fallbacks must be honest: cached state needs freshness, queued work needs pending status, unsafe commands need manual confirmation or explicit unavailability.
8.12 Architecture Review Checklist
Use this checklist during design review or incident follow-up.
Dependency inventory List every synchronous dependency the service calls during the workflow.
Deadline budget Document the total deadline, per-attempt timeout, retry limit, and cancellation behavior.
Retry eligibility Mark each operation as safe, idempotent with key, unsafe to retry, or asynchronous retry only.
Breaker policy Define the failure signal, opening rule, cooldown, half-open probe limit, and fallback behavior.
Bulkhead boundary Identify the pool, queue, connection budget, or service partition that contains each dependency failure.
Fallback contract Decide what users, devices, and downstream systems see during degraded mode.
Recovery test Run controlled dependency failures and verify that the system degrades, alerts, and recovers without synchronized retry storms.
8.13 Common Pitfalls
8.13.1 Retrying Everything
Blind retries duplicate unsafe commands and amplify load. Retry only transient failures with idempotency and a deadline budget.
8.13.2 Timeout Inflation
Increasing timeouts may hide symptoms while making resource exhaustion worse. Investigate the dependency and the caller budget.
8.13.3 Circuit Breaker Without Fallback
Failing fast protects resources, but the product still needs a user-visible or workflow-visible degraded result.
8.13.5 Missing Jitter
Backoff without jitter can still synchronize clients. Jitter is essential for device fleets.
8.13.6 No Degraded-Mode Tests
Untested fallback paths often fail during the first real outage. Practice cloud-down, dependency-down, and queue-backlog scenarios.
8.14 Summary
- Resilience patterns prevent one failed dependency from consuming the whole service.
- Deadlines and timeouts bound waiting.
- Retries need idempotency, backoff, jitter, and a retry budget.
- Circuit breakers fail fast after repeated failure signals and use half-open probes for recovery.
- Bulkheads isolate resource pools so healthy features keep working.
- Fallback behavior should be explicit, tested, observable, and honest about degraded mode.
8.15 Key Takeaway
Resilience is a coordinated contract, not a single pattern. Bound each call with deadlines, retry only safe operations with backoff and jitter, isolate shared resources, and test fallback behavior before a dependency outage reaches users or devices.
8.16 Knowledge Check
8.17 Try It Yourself: Resilience Review Note
Choose one synchronous dependency in an IoT service and write a short review note.
service_call: command-api -> authorization-service
user_visible_deadline: 2s
per_attempt_timeout: 500ms
retry_policy:
eligible_errors: [connection-reset, 503, 429]
max_attempts: 2
backoff: exponential
jitter: required
idempotency:
required: true
key: command_id
circuit_breaker:
opens_when: failure-rate-exceeds-policy-window
half_open: limited-probes
bulkhead:
resource: authorization-client-pool
limit_basis: measured-load-test
fallback:
behavior: reject unsafe commands, queue safe idempotent commands
observability:
metrics: [deadline-exhausted, retry-attempts, open-circuit, fallback-count]Then run a dependency-down test and verify that healthy requests still have capacity, unsafe commands are not duplicated, and retry traffic ramps back gradually.
8.18 References
- Microsoft Azure Architecture Center: Circuit Breaker pattern
- Microsoft Azure Architecture Center: Retry pattern
- Microsoft Azure Architecture Center: Bulkhead pattern
- AWS Builders Library: Timeouts, retries, and backoff with jitter
- Google SRE Book: Addressing Cascading Failures
- IETF RFC 9110: HTTP Semantics
8.19 What’s Next
8.19.1 State Machine Patterns
Use explicit state models for device behavior, workflow recovery, and circuit-breaker-like transitions.
8.19.2 SOA Container Orchestration
Deploy resilient services with health checks, rollout controls, autoscaling, and service-level resource policy.
8.19.3 SOA API Design and Service Discovery
Design API contracts with idempotency, versioning, rate limits, and discovery rules that make resilience possible.
8.19.4 Cloud Computing for IoT
Review cloud and edge placement choices that affect fallback, latency, and recovery behavior.
8.21 Continue Your Route
This final part closes the route from Failure Modes in IoT Services through Navigation. Return to Service Resilience: Operating Context and Control Map or continue from the design-patterns module index.