10 AMQP Delivery: Sizing and Reliability Contracts

10.8.1 Learning Objectives

Prove Every Link in One Alarm Chain

Picture a site unit sending a valve alarm to an incident worker. The message reaches a queue, the worker writes the incident, and then the worker crashes before it says the job is done. The next worker may see the same alarm, so the team must prevent a second incident without losing the first.

Trace one alarm from local copy to accepted store, waiting queue, worker, final record, and done signal. Name what each signal proves and what it does not. Keep one alarm id through every link so a resend or redelivery can be joined to the same physical event.

Test a failed send, a store restart, a message held only in memory, a worker crash before and after the write, a rejected item, and a full retry path. Check both loss and duplicate effects. One durable setting cannot protect a different gap in the chain.

Keep any urgent safe valve action local when delivery is late. The chain can notify and retain evidence, but it must not be the sole route to stop immediate harm.

This opening does not promise exactly-once physical action. Practitioner maps each loss window and owner. Under the Hood examines confirms, durable definitions, stored bodies, worker replies, repeated delivery, dead-letter paths, and safe replay.

After this page, you should be able to:

• Explain why publisher confirms, queue durability, message persistence, and consumer acknowledgments protect different loss windows.
• Diagnose which reliability link is missing from an AMQP delivery path.
• Explain why at-least-once delivery requires idempotent consumers.
• Identify the exact broker triggers that dead-letter a message.
• Design a DLX retry and parking flow that preserves investigation evidence.

10.8.2 Why This Follows AMQP Message Delivery

AMQP Message Delivery teaches message structure, delivery modes, publisher confirms, consumer acknowledgments, prefetch, dead-letter queues, and delivery sizing. This page tightens the contract that connects those pieces: no single flag makes delivery reliable, and each acknowledgment only means something inside its scope.

Use it when a deployment needs a zero-loss audit, when a durable queue still loses messages after restart, when repeated deliveries must be made safe with idempotency, or when a DLQ runbook needs to distinguish retryable failures from messages that should move to parking.

10.8.3 Overview: “Reliable” Is a Chain, Not a Switch

A message survives from producer to processed only if every link in a chain holds. There is no single “reliable” flag. At-least-once delivery end to end needs four independent mechanisms lined up:

1. Publisher confirms so the producer knows the broker actually took the message.
2. A durable queue so the queue definition survives a broker restart.
3. A persistent message (delivery-mode = 2) so the message body is written to disk, not just held in RAM.
4. Consumer acknowledgements so the broker only removes a message after it has been processed, not merely delivered.

Break any one and you open a silent loss window: no confirm means the producer never learns a publish was dropped; a non-durable queue vanishes on restart; a transient message in a durable queue is still lost on restart; auto-ack deletes a message the instant it is sent, before the consumer has done anything with it.

Concrete chain example: a gateway sends a valve-state alarm and keeps its local copy until the broker sends a publisher confirm. If the broker crashes before the confirm, the gateway resends from local storage. If the broker confirms and then restarts, the durable queue and persistent message are what make the alarm still appear after recovery. If the consumer receives the alarm, writes it to an incident table, and crashes before basic.ack, the broker requeues the unacknowledged delivery on channel close. That is why the downstream incident write needs an alarm id: the next consumer may see the same delivery again, and the database should treat it as the same alarm, not a second incident.

The practical audit question is therefore always, “where can the message be acknowledged but not yet recoverable?” Publisher confirms answer the producer side, persistence answers the broker restart side, and manual acknowledgements answer the consumer side. Treating those as one chain keeps operators from over-trusting a single durable setting.

10.8.4 Practitioner: What Each Link Protects Against

The two disk settings only work together: a persistent message in a transient queue is not saved, and a transient message in a durable queue is not saved either. You need both. Publisher confirms even close the fsync race — the broker only confirms a persistent message routed to a durable queue after it is on disk.

Worked example — a meter reading that must not be lost. The producer opens confirm mode, publishes with delivery-mode=2 to a durable queue, and waits for the broker’s confirm before deleting its local copy. The consumer reads, writes the reading to the database, and only then sends basic.ack. If the consumer dies between read and ack, the broker sees the channel drop, requeues the unacked message, and another consumer reprocesses it.

Now add sizing. If four consumers each use basic.qos(prefetch_count=20), as many as 80 messages can be delivered but unacknowledged at once. A worker crash with 17 unacked messages does not lose those messages; they return to the queue when the channel closes. A producer using confirms in batches of 50 should keep at least the current unconfirmed batch locally, because a connection drop after publishing message 1,237 but before its confirm leaves the producer unsure whether the broker stored it. Retrying from a message id makes the duplicate detectable.

That requeue is why at-least-once can duplicate. AMQP 0-9-1 does not give true exactly-once on its own; you reach “effectively once” by making the consumer idempotent — dedupe on the message-id or a natural key so a reprocessed message is harmless.

10.8.5 Under the Hood: When a Message Is Dead-Lettered

A queue configured with x-dead-letter-exchange re-publishes a message to that exchange on exactly three triggers — nothing else:

• the consumer rejects or nacks it with requeue=false (a poison message it cannot process),
• the message’s TTL expires (a per-message expiration or the queue’s x-message-ttl), or
• the queue overflows a length limit (x-max-length or x-max-length-bytes), which by default drops the oldest message.

On dead-lettering, the broker routes the message to the dead-letter exchange using x-dead-letter-routing-key if set, otherwise the message’s original routing key, and prepends an x-death header recording the reason, the source queue, a count, and a timestamp. That count is what lets a retry loop give up after N attempts instead of cycling a poison message forever. A common design is: main queue -> DLX -> a “retry” queue with a short TTL that dead-letters back to the main exchange, plus a “parking” queue for messages whose x-death count has exceeded the limit.

For example, a decoder that cannot parse a payload should send basic.nack(requeue=false), not keep requeueing the same poison message at the head of the main queue. The DLX can route it to a retry queue with x-message-ttl=30000; after 30 seconds that retry queue dead-letters it back to the main exchange. When the message fails again, the next x-death entry or count tells the retry policy it has already been through the loop. After three failed attempts, the handler can route it to a parking queue with the original body, headers, failure reason, and source queue preserved for investigation. The DLQ is therefore an evidence path as much as a reliability path: it stops one bad message from blocking good traffic while keeping enough context to repair the producer or consumer.

Inspect Figure 10.1 to separate a bounded retry loop from the terminal evidence path for a poison message.

Trace Figure 10.1 from the main queue through rejection, expiry, or overflow into the dead-letter exchange. Retryable work waits in the TTL queue and returns; excessive failures move to parking with context intact. This route turns x-death history into a retry decision instead of an infinite cycle.