9  AMQP Delivery Reliability Chain Contracts

Start with the story: Reliability in AMQP is not one magic checkbox. It is a chain of receipts: the producer needs a broker confirm, the broker needs durable storage, the message needs persistence, and the consumer must acknowledge only after the real work is done.

9.1 Learning Objectives

After this page, you should be able to:

• Explain why publisher confirms, queue durability, message persistence, and consumer acknowledgments protect different loss windows.
• Diagnose which reliability link is missing from an AMQP delivery path.
• Explain why at-least-once delivery requires idempotent consumers.
• Identify the exact broker triggers that dead-letter a message.
• Design a DLX retry and parking flow that preserves investigation evidence.

9.2 Why This Follows AMQP Message Delivery

AMQP Message Delivery teaches message structure, delivery modes, publisher confirms, consumer acknowledgments, prefetch, dead-letter queues, and delivery sizing. This page tightens the contract that connects those pieces: no single flag makes delivery reliable, and each acknowledgment only means something inside its scope.

Use it when a deployment needs a zero-loss audit, when a durable queue still loses messages after restart, when repeated deliveries must be made safe with idempotency, or when a DLQ runbook needs to distinguish retryable failures from messages that should move to parking.

9.3 Overview: “Reliable” Is a Chain, Not a Switch

A message survives from producer to processed only if every link in a chain holds. There is no single “reliable” flag. At-least-once delivery end to end needs four independent mechanisms lined up:

1. Publisher confirms so the producer knows the broker actually took the message.
2. A durable queue so the queue definition survives a broker restart.
3. A persistent message (delivery-mode = 2) so the message body is written to disk, not just held in RAM.
4. Consumer acknowledgements so the broker only removes a message after it has been processed, not merely delivered.

Break any one and you open a silent loss window: no confirm means the producer never learns a publish was dropped; a non-durable queue vanishes on restart; a transient message in a durable queue is still lost on restart; auto-ack deletes a message the instant it is sent, before the consumer has done anything with it.

Concrete chain example: a gateway sends a valve-state alarm and keeps its local copy until the broker sends a publisher confirm. If the broker crashes before the confirm, the gateway resends from local storage. If the broker confirms and then restarts, the durable queue and persistent message are what make the alarm still appear after recovery. If the consumer receives the alarm, writes it to an incident table, and crashes before basic.ack, the broker requeues the unacknowledged delivery on channel close. That is why the downstream incident write needs an alarm id: the next consumer may see the same delivery again, and the database should treat it as the same alarm, not a second incident.

The practical audit question is therefore always, “where can the message be acknowledged but not yet recoverable?” Publisher confirms answer the producer side, persistence answers the broker restart side, and manual acknowledgements answer the consumer side. Treating those as one chain keeps operators from over-trusting a single durable setting.

9.4 Practitioner: What Each Link Protects Against

Mechanism	Protects against	If you omit it
Publisher confirm (confirm.select)	Broker rejected, unroutable, or crashed mid-publish	Producer believes a lost message was delivered
Durable queue (declare durable=true)	Broker restart wiping the queue definition	Queue and all its messages disappear on restart
Persistent message (delivery-mode=2)	Broker restart wiping in-memory message bodies	Durable queue comes back empty
Manual consumer ack (basic.ack)	Consumer crashing after delivery, before processing	Message removed on delivery; work is lost

The two disk settings only work together: a persistent message in a transient queue is not saved, and a transient message in a durable queue is not saved either. You need both. Publisher confirms even close the fsync race – the broker only confirms a persistent message routed to a durable queue after it is on disk.

Worked example – a meter reading that must not be lost. The producer opens confirm mode, publishes with delivery-mode=2 to a durable queue, and waits for the broker’s confirm before deleting its local copy. The consumer reads, writes the reading to the database, and only then sends basic.ack. If the consumer dies between read and ack, the broker sees the channel drop, requeues the unacked message, and another consumer reprocesses it.

Now add sizing. If four consumers each use basic.qos(prefetch_count=20), as many as 80 messages can be delivered but unacknowledged at once. A worker crash with 17 unacked messages does not lose those messages; they return to the queue when the channel closes. A producer using confirms in batches of 50 should keep at least the current unconfirmed batch locally, because a connection drop after publishing message 1,237 but before its confirm leaves the producer unsure whether the broker stored it. Retrying from a message id makes the duplicate detectable.

That requeue is why at-least-once can duplicate. AMQP 0-9-1 does not give true exactly-once on its own; you reach “effectively once” by making the consumer idempotent – dedupe on the message-id or a natural key so a reprocessed message is harmless.

9.5 Under the Hood: When a Message Is Dead-Lettered

A queue configured with x-dead-letter-exchange re-publishes a message to that exchange on exactly three triggers – nothing else:

• the consumer rejects or nacks it with requeue=false (a poison message it cannot process),
• the message’s TTL expires (a per-message expiration or the queue’s x-message-ttl), or
• the queue overflows a length limit (x-max-length or x-max-length-bytes), which by default drops the oldest message.

On dead-lettering, the broker routes the message to the dead-letter exchange using x-dead-letter-routing-key if set, otherwise the message’s original routing key, and prepends an x-death header recording the reason, the source queue, a count, and a timestamp. That count is what lets a retry loop give up after N attempts instead of cycling a poison message forever. A common design is: main queue -> DLX -> a “retry” queue with a short TTL that dead-letters back to the main exchange, plus a “parking” queue for messages whose x-death count has exceeded the limit.

For example, a decoder that cannot parse a payload should send basic.nack(requeue=false), not keep requeueing the same poison message at the head of the main queue. The DLX can route it to a retry queue with x-message-ttl=30000; after 30 seconds that retry queue dead-letters it back to the main exchange. When the message fails again, the next x-death entry or count tells the retry policy it has already been through the loop. After three failed attempts, the handler can route it to a parking queue with the original body, headers, failure reason, and source queue preserved for investigation. The DLQ is therefore an evidence path as much as a reliability path: it stops one bad message from blocking good traffic while keeping enough context to repair the producer or consumer.

Figure 9.1: A dead-letter exchange is a controlled branch: retryable failures loop through a TTL queue, while messages over the retry limit move to a parking queue with their evidence intact.