Queue Reliability Practice
Queue Reliability Practice
Challenge 1: Expiry and Stale Work
A command can be safe when sent but harmful after a long wait. A time-to-live (TTL) limits its useful life in the queue. Record each expiry and choose what happens next: send it for review, count and drop it, or take a compensating action. A consumer must also check the source time before it acts. Silent expiry can hide a stuck worker.
A dead-letter exchange (DLX) is the broker routing point that sends expired or rejected messages to a configured destination. Now test this rule: trace an expired command. Use the steps below and record what you observe.
- In the AMQP workbench below, select Stale command and press Step until the death trigger appears.
- The browser run here displayed
DEAD-LETTERED | TTL expiry | expired: route through DLX; it is a broker-behavior model, not an AMQP server receipt. - Record the original message ID, produced time, TTL, expiry time, and disposition before considering a replay.
- At the consumer, compare the command timestamp with its safety deadline; an expired command must not actuate simply because it reached the front of the queue.
- Choose a measured expiry path: dead letter for investigation, metric-only drop for replaceable telemetry, or compensation when a stale command has consequences.
Challenge 2: Dead Letters and Head-of-Line Blocking
A dead-letter queue (DLQ) holds work the consumer cannot handle. In an ordered flow, one bad record can block all those behind it. Retry a brief timeout, but move a bad schema or denied request aside. Keep the payload, source, time, error, retry count, and consumer version. Repair the fault and check whether replay is safe before sending it back.
The broker can add an x-death header to describe why and where a message was dead-lettered, along with a count. Now test this rule: isolate a poison message before replay. Use the steps below and record what you observe.
- Select Bad payload in the AMQP workbench and step to DLX routing; the browser run displayed
DEAD-LETTERED | Consumer rejection | Inspect then retry. - Switch to Retry limit; the run displayed
DEAD-LETTERED | Quorum delivery limit | delivery_limit: route through DLX. - Copy the original message ID, payload, routing key, error class, retry count, and
x-deathreason into the triage record. - Repair the schema or authorization failure first, then check freshness, identity, ordering, and idempotency before approving a single replay.
- These outputs came from the browser workbench; verify the same path against your own AMQP broker metrics and consumer logs before release.
Follow the six stages from left to right. First capture the fault and test whether it is transient and within the retry budget. If not, isolate the message with its ID and headers so the rest of the flow can proceed. Diagnose the cause and record the fix. At the last stage an owner chooses safe replay, compensation, or discard with a reason. The lower evidence rail keeps the facts needed to justify that choice. Use the workbench below to explore how a broker routes a dead letter; it does not prove that replay is safe in your own system.
Challenge 3: Deduplication and Idempotency
Retries can deliver the same message twice. Deduplication checks a stable message ID and rejects a repeat. Keep the source and message ID for the longest allowed retry or manual replay delay. If the ID expires first, a late repeat looks new; reject replay beyond that window. Idempotent handling provides a second defense: the same command has the same effect even if it runs twice. For example, set a relay to off instead of toggling it.
Now test this rule: review one replayed ID and its side effect. Use the steps below and record what you observe.
- Write down a stable source ID and message ID for a test command, plus the latest possible retry or operator replay time.
- Send the same ID twice through your own consumer and inspect the deduplication log: the second delivery should be recognized within the replay window.
- Test an absolute state-setting command twice and confirm the protected state has one effect; a relative toggle would produce a different result.
- Compare cache retention with the longest retry-and-replay delay, not average arrival rate; evicting the ID early makes late duplicates appear new.
- No production consumer was run here. The browser workbench models routing decisions; the deduplication and side-effect proof requires your own consumer log.
Challenge 4: Consumer Groups and Ordering
A group of workers can share the load, but parallel work can change the order of events. One global order would force a single worker. Instead, split the stream into partitions, each processed in sequence. Pick a key that groups the events whose order matters, such as the device ID, and send every event with that key to the same partition. Other devices use other partitions. If an item fails, hold later work for that key until the retry finishes. Then test the order under failure.
Now test this rule: check per-device ordering across workers. Use the steps below and record what you observe.
- Publish numbered events for two device IDs and record the partition or consumer assigned to each ID.
- Pause one worker before acknowledging an event, then note whether the next event for that same device overtakes it.
- Route unrelated device IDs independently so a slow device does not stop all useful work.
- Record the retry path for a failed ordered event; reinserting it behind newer events would break the per-device sequence.
- This requires a real queue and multiple consumers; no ordering result was run in this lane, so the expected invariant remains to be verified locally.
Challenge 5: Backpressure and Circuit Breaking
Backpressure tells the sender to slow down when the consumer cannot keep up. A circuit breaker can also pause calls to a failing consumer. Watch oldest age, depth, in-flight work, and errors together. Set a clear limit and a safe recovery test before allowing traffic to resume.
Now test this rule: distinguish overflow from recovery. Use the steps below and record what you observe.
- In the AMQP workbench, select Burst overflow and step to the decision trace; the browser run displayed
DEAD-LETTERED | Queue overflow | maxlen: route through DLX. - Switch to Valid telemetry; it displayed
ACKED | Consumer ACK | Successful processing: no DLX publish. - On your own broker, graph depth together with oldest age, in-flight count, retries, consumer errors, and downstream latency before changing intake.
- Open the breaker on repeated downstream failures, then allow a small half-open trial and resume full delivery only after trial messages succeed.
- The browser result demonstrates routing policy only; broker pressure, breaker transitions, and safe resume need production-like consumer metrics.
Queue Reliability Review Record
Close the lab by recording the design as a release artifact so the behavior is reproducible and operable, not just functional.
Read the Failure Route Before Replaying
Use a concrete outage. A sensor publishes one 240-byte record every 10 s, or six records each minute. During a 20-minute uplink loss, the gateway receives records. Their payloads occupy bytes before queue metadata. If the queue limit is 100 records, 20 records cannot fit. The lab must state whether the oldest 20 are dropped, the newest 20 are rejected, or acquisition is slowed; an unspecified overflow policy is already a failure.
Now place a malformed record at position 37. With ordered single-message processing, that record can block positions 38 through 120 if the consumer retries forever. Set a retry ceiling of three attempts. After the third failure, route position 37 to the dead-letter queue and allow position 38 to proceed. Record ordering has changed around one exception, so the consumer must know whether strict order or continued service matters more for this stream.
Predict the lab evidence before running it. First, expect 120 arrivals and a 20-record overflow under the stated limit; compare queue counters with the calculation. Second, expect record 37 to show three failed attempts and one dead-letter entry; check that positions after it continue. Third, repair the decoder and replay only that record, then confirm one accepted result and no duplicate side effect. If a counter differs, stop and explain the queue transition rather than adjusting the expected answer after the run.
The lab is complete only when normal traffic, overflow, poison-message isolation, and replay each have an observed result. A screenshot of an empty queue proves none of those states by itself.
Add one consumer crash after it performs a side effect but before acknowledgement. On restart, the queue may redeliver that record. Use its stable identifier to prove that the target stores one result, then acknowledge the repeated delivery. This challenge separates transport completion from application completion and reveals whether at-least-once behavior is safe for the chosen message.
End by reconciling counts: produced records equal acknowledged records plus queued, dead-lettered, and explicitly dropped records. Any unexplained remainder is a message whose fate the queue evidence cannot show.
Run one final burst after recovery. New live records and replayed old records now compete for the consumer. State whether live alarms take priority, whether original order is restored, and how old data is labelled. Verify that the oldest queued observation keeps its original time and that the queue returns below its warning level without hiding dead-letter items.
Inspect the dead-letter entry after the lab. It should show the original message unchanged, the decoder error, three attempt times, and replay outcome. Delete or retain it according to the written evidence policy only after the repaired result is linked.
Save the queue configuration with the counter evidence and the exact poison-message fixture used during the challenge.
Technical boundaries. The outage calculation uses one 240-byte record every 10 s, a 20-minute uplink loss, and a 100-record queue limit. Its 28,800 bytes cover payloads before queue metadata; browser routing decisions do not prove replay is safe.