Queue Reliability Practice

Queue Reliability Practice

animation
integration-gateways
messaging

Challenge 1: Expiry and Stale Work

A command can be safe when sent but harmful after a long wait. A time-to-live (TTL) limits its useful life in the queue. Record each expiry and choose what happens next: send it for review, count and drop it, or take a compensating action. A consumer must also check the source time before it acts. Silent expiry can hide a stuck worker.

A dead-letter exchange (DLX) is the broker routing point that sends expired or rejected messages to a configured destination. Now test this rule: trace an expired command. Use the steps below and record what you observe.

Challenge 2: Dead Letters and Head-of-Line Blocking

A dead-letter queue (DLQ) holds work the consumer cannot handle. In an ordered flow, one bad record can block all those behind it. Retry a brief timeout, but move a bad schema or denied request aside. Keep the payload, source, time, error, retry count, and consumer version. Repair the fault and check whether replay is safe before sending it back.

The broker can add an x-death header to describe why and where a message was dead-lettered, along with a count. Now test this rule: isolate a poison message before replay. Use the steps below and record what you observe.

Six steps from captured failure through retry, isolation, diagnosis, repair, and a recorded choice of replay, compensation, or discard.
A failed message passes a retry gate, then moves through isolation, diagnosis, repair, and a recorded disposition.

Follow the six stages from left to right. First capture the fault and test whether it is transient and within the retry budget. If not, isolate the message with its ID and headers so the rest of the flow can proceed. Diagnose the cause and record the fix. At the last stage an owner chooses safe replay, compensation, or discard with a reason. The lower evidence rail keeps the facts needed to justify that choice. Use the workbench below to explore how a broker routes a dead letter; it does not prove that replay is safe in your own system.

Challenge 3: Deduplication and Idempotency

Retries can deliver the same message twice. Deduplication checks a stable message ID and rejects a repeat. Keep the source and message ID for the longest allowed retry or manual replay delay. If the ID expires first, a late repeat looks new; reject replay beyond that window. Idempotent handling provides a second defense: the same command has the same effect even if it runs twice. For example, set a relay to off instead of toggling it.

Now test this rule: review one replayed ID and its side effect. Use the steps below and record what you observe.

Problem
Evidence
Decision
Why It Works
Duplicate delivery
The same message id appears more than once after a reconnect, retry, or replay.
Keep a deduplication window at least as long as the retry and replay horizon for that flow.
A repeated id is recognized while it can still arrive, so it is rejected before it is processed.
Duplicate side effect
A repeated command changes state twice, such as incrementing a counter or toggling a relay.
Use idempotency keys or absolute state-setting commands instead of relative toggles.
Processing the same intent twice converges to one result instead of doubling it.
Cache too small
The deduplication cache evicts ids before delayed duplicates arrive.
Size the cache from burst and replay evidence, not from average message rate.
Late duplicates still fall inside the window, so detection does not silently lapse.
Missing identity
Messages arrive without a stable source id and sequence or message id.
Add stable identity for deduplication; separately test whether repeated operations are inherently safe.
Without a stable ID, ID-based deduplication cannot recognize a repeat. An inherently idempotent operation may still make repeats harmless.

Challenge 4: Consumer Groups and Ordering

A group of workers can share the load, but parallel work can change the order of events. One global order would force a single worker. Instead, split the stream into partitions, each processed in sequence. Pick a key that groups the events whose order matters, such as the device ID, and send every event with that key to the same partition. Other devices use other partitions. If an item fails, hold later work for that key until the retry finishes. Then test the order under failure.

Now test this rule: check per-device ordering across workers. Use the steps below and record what you observe.

Challenge 5: Backpressure and Circuit Breaking

Backpressure tells the sender to slow down when the consumer cannot keep up. A circuit breaker can also pause calls to a failing consumer. Watch oldest age, depth, in-flight work, and errors together. Set a clear limit and a safe recovery test before allowing traffic to resume.

Now test this rule: distinguish overflow from recovery. Use the steps below and record what you observe.

Queue Reliability Review Record

Close the lab by recording the design as a release artifact so the behavior is reproducible and operable, not just functional.

Area
What to Record
Evidence
Failure If Missing
Ingress contract
Id fields, timestamps, priority, TTL, schema, and source identity.
Sample envelopes and a schema reference.
No basis to detect stale, duplicate, or replay-unsafe work.
Dead-letter rules
Retriable vs non-retriable classes, captured context, and replay approval.
A dead-letter sample with full headers and error class.
Poison messages block healthy traffic with no audit trail.
Deduplication
Id format, window horizon, idempotency key, and side-effect check.
A duplicate-injection test result.
Retries and replays double side effects.
Backpressure
Trigger metrics, producer and consumer actions, and breaker behavior.
A slow-consumer and recovery drill result.
Downstream collapses instead of degrading gracefully.

Read the Failure Route Before Replaying

Use a concrete outage. A sensor publishes one 240-byte record every 10 s, or six records each minute. During a 20-minute uplink loss, the gateway receives (6×20=120)(6 \times 20=120) records. Their payloads occupy (120×240=28,800)(120 \times 240=28{,}800) bytes before queue metadata. If the queue limit is 100 records, 20 records cannot fit. The lab must state whether the oldest 20 are dropped, the newest 20 are rejected, or acquisition is slowed; an unspecified overflow policy is already a failure.

Now place a malformed record at position 37. With ordered single-message processing, that record can block positions 38 through 120 if the consumer retries forever. Set a retry ceiling of three attempts. After the third failure, route position 37 to the dead-letter queue and allow position 38 to proceed. Record ordering has changed around one exception, so the consumer must know whether strict order or continued service matters more for this stream.

Predict the lab evidence before running it. First, expect 120 arrivals and a 20-record overflow under the stated limit; compare queue counters with the calculation. Second, expect record 37 to show three failed attempts and one dead-letter entry; check that positions after it continue. Third, repair the decoder and replay only that record, then confirm one accepted result and no duplicate side effect. If a counter differs, stop and explain the queue transition rather than adjusting the expected answer after the run.

The lab is complete only when normal traffic, overflow, poison-message isolation, and replay each have an observed result. A screenshot of an empty queue proves none of those states by itself.

Add one consumer crash after it performs a side effect but before acknowledgement. On restart, the queue may redeliver that record. Use its stable identifier to prove that the target stores one result, then acknowledge the repeated delivery. This challenge separates transport completion from application completion and reveals whether at-least-once behavior is safe for the chosen message.

End by reconciling counts: produced records equal acknowledged records plus queued, dead-lettered, and explicitly dropped records. Any unexplained remainder is a message whose fate the queue evidence cannot show.

Run one final burst after recovery. New live records and replayed old records now compete for the consumer. State whether live alarms take priority, whether original order is restored, and how old data is labelled. Verify that the oldest queued observation keeps its original time and that the queue returns below its warning level without hiding dead-letter items.

Inspect the dead-letter entry after the lab. It should show the original message unchanged, the decoder error, three attempt times, and replay outcome. Delete or retain it according to the written evidence policy only after the repaired result is linked.

Save the queue configuration with the counter evidence and the exact poison-message fixture used during the challenge.

Technical boundaries. The outage calculation uses one 240-byte record every 10 s, a 20-minute uplink loss, and a 100-record queue limit. Its 28,800 bytes cover payloads before queue metadata; browser routing decisions do not prove replay is safe.

Back to the chapter