14 AMQP Reliability: Acknowledgements and Crash Recovery

14.9.1 Learning Objectives

After this page, you should be able to:

• Explain why a publisher confirm does not prove that a message reached a queue.
• Detect unroutable publishes with mandatory=true, basic.return, and alternate exchanges.
• Size a catch-all alternate-exchange queue for a realistic misrouting incident.
• Distinguish AMQP topic * and # word matching from shell glob and MQTT wildcard behavior.
• Build positive, negative, and boundary routing tests that prevent silent AMQP drops.

14.9.2 Why This Follows AMQP Implementation Pitfalls

AMQP Implementation Pitfalls reviews the major failure modes: durability gaps, unsafe acknowledgments, wildcard mistakes, protocol-selection errors, and missing idempotency. This page tightens one operational contract inside those pitfalls: a publish can be accepted by the broker and still reach no queue unless unroutable handling and routing-pattern tests are deliberate.

Use it when producers rely on publisher confirms, when a topic hierarchy has variable depth, when AMQP routing is being migrated from MQTT topics, or when a production runbook needs proof that bad bindings become alerts rather than missing telemetry.

14.9.3 Overview: The Dangerous Failures Are the Silent Ones

Many AMQP surprises are not crashes — they are messages that vanish with no error at all. Two common sources of "the broker is eating my messages" incidents are a message that reaches no queue and a topic binding pattern that does not match the routing key the developer assumed it would. In both cases the publish can appear successful while data never becomes available to a consumer.

A confirm and a routing verdict answer different questions. The figure below follows one AMQP 0-9-1 publish through independent binding evaluation and then shows what becomes observable when the final route count is zero.

Inspect Figure 14.1 to distinguish publisher acceptance from the later routing verdict that decides whether any queue receives a copy.

Trace Figure 14.1 through binding evaluation: zero matches require return or alternate-exchange policy, one match creates one queue copy, and several distinct matches create several copies. The confirm branch remains separate. This is the diagnostic model for silent wildcard failures developed in the chapter.

Test one key that should match several queues, one that should match exactly one, and one that should match none. For each case, record the expected queue set, alternate or return outcome, and the separate confirm result. A clean confirm without routing evidence is not proof that a consumer can receive the message.

The cure is to know exactly when AMQP discards a message quietly, and to turn those quiet discards into visible signals you can alert on. A durable queue protects the queue definition; a persistent message protects the payload; a publisher confirm says the broker accepted responsibility for the publish at its configured boundary; a consumer ACK says downstream delivery processing completed; an idempotency key makes a redelivery safe to run twice.

For example, suppose a packaging line publishes 2,400 quality alerts per hour with routing keys such as sensor.floor1.temperature.critical. If the analytics queue is bound to sensor.*, every four-word alert misses the queue even though the producer receives a confirm. At 2,400 alerts/hour, that is 19,200 missing records across one shift. With no alternate exchange, mandatory=true returns each finally unroutable publish. With a catch-all alternate exchange, the broker extends the routing chain and captures the records instead; a message successfully routed by that alternate counts as routed for the mandatory flag.

That is also why a useful review asks "which guarantee detects this failure?" for every critical path. Restart loss needs durable plus persistent storage; crash loss needs manual ACK after the side effect; duplicate execution needs an idempotency key; routing loss needs a matching binding plus return or alternate-exchange handling. If the answer is only "publisher confirms are on", the design is still missing part of the failure story.

14.9.4 Practitioner: An Unroutable Message Is Dropped by Default

When an exchange receives a message that matches no binding, its default behavior is to discard it silently. There are three ways to change that, and one dangerous misconception:

The misconception that bites hardest: a publisher confirm does not mean "routed to a queue". The broker confirms an unroutable message as successfully handled — it did its job, there was just nowhere to send it. To detect unroutable messages you must combine confirms with the mandatory flag (or an alternate exchange). Confirms alone will happily ack data straight into the void.

A practical production check uses two intentionally bad routes before launch. First publish with mandatory=true and no alternate exchange; the publisher should receive basic.return with NO_ROUTE. Then publish through the production exchange with its catch-all alternate configured; the catch-all queue should receive the message and the publisher should not receive a mandatory return because the routing chain found a queue. If the alternate chain also reaches no queue, mandatory=true makes that final zero-route result visible as basic.return. Publisher confirm remains a separate acceptance signal; for a finally unroutable mandatory publish, the return is delivered before the confirm acknowledgement.

Size the catch-all queue as an incident buffer, not as the normal data path. If the plant publishes 40 messages/s and a bad release misroutes 10% for 15 minutes, the alternate queue needs room for 40 x 0.10 x 900 = 3,600 messages plus margin. Alert on any non-zero return rate for critical streams, and alert separately when alternate-exchange depth grows faster than the operator can drain it. That turns a routing typo into a visible rollback trigger instead of a missing-data investigation tomorrow.

14.9.5 Under the Hood: Topic Wildcards Are Word-Matching, Not Glob

Topic routing keys are dot-separated words, and the two wildcards mean precise, different things: * matches exactly one word; # matches zero or more words. The most common bug is expecting * to span multiple levels the way a shell glob would.

Note that sensor.* matches only keys of the form sensor.<one-word> — so it matches sensor.floor1 but not sensor.floor1.temp (too many words) and not sensor (too few). If you want "everything under sensor" you need sensor.#. This also differs from MQTT, whose wildcards are + (one level) and # (rest of the tree, and only as the final token). Same idea, different characters and different edge behavior — a frequent source of migration bugs.

The exchange evaluates each binding independently. A single publish can match zero queues, one queue, or several queues; it is not a first-match router. With bindings sensor.#, sensor.*.temp, and sensor.floor1.#, the key sensor.floor1.temp routes to all three queues. The key sensor.floor1.temp.critical routes to sensor.# and sensor.floor1.#, but not sensor.*.temp, because that pattern has exactly three words and the routing key has four.

This is why routing tests should include negative and boundary cases. For a binding intended to capture "all temperature under any floor", test sensor.floor1.temp, sensor.floor1.room7.temp, sensor.temp, and sensor.floor1.temp.critical. If only the first passes, the binding is fixed-depth; if all relevant depths pass and unrelated keys fail, it is a safe hierarchy binding. Keep these tests next to publisher-confirm tests so the team does not confuse "broker accepted the message" with "a consumer can receive the message".