15  Availability and Confirms

Start with the story: A broker setup that is correct in a lab can still fail in production if one node, one confirm loop, or one growing queue becomes the hidden weak point. This page turns those weak points into explicit availability and monitoring contracts.

15.1 Learning Objectives

After this page, you should be able to:

• Explain why a durable single-node queue is still a production availability risk.
• Choose quorum queues for critical RabbitMQ workloads and describe majority-failure behavior.
• Separate publishing and consuming connections while reusing channels deliberately.
• Track asynchronous publisher confirms with delivery tags, multiple acknowledgments, nacks, and mandatory returns.
• Tie confirm-window, queue-depth, unacked-message, and dead-letter metrics to alert thresholds.

15.2 Why This Follows AMQP Production Operations

AMQP Production Operations teaches durable declarations, client-library implementations, dead-letter queues, monitoring, and capacity calculators. This page tightens the production availability contract underneath that code: a queue must tolerate broker node loss, a publisher must sustain throughput without losing confirm state, and operators must know before backlog age violates the product SLA.

Use it when a RabbitMQ deployment is moving from pilot to production, when publishers block on synchronous confirms, when a single broker node is still a failure domain, or when a runbook needs concrete alert thresholds for publish pressure, queue backlog, consumer health, and dead-letter flow.

15.3 Overview: Production Adds Availability and Scale

Everything so far makes a single broker correct. Production adds two harder requirements: the queue must survive a broker node failing, and the publisher must sustain thousands of messages per second without blocking on each confirm. The two answers are replicated queues and asynchronous publisher confirms. A durable queue on one node is still a single point of failure; a synchronous “publish, wait for confirm, repeat” loop is safe but slow.

Production monitoring follows the broker path: producers create load, bindings choose queues, queue depth stores backlog, and consumers determine drain rate.

The goal of this layer is to get both safety and throughput at once, plus keep publishing healthy when consumers or the broker fall behind. Read the flow from left to right: producer rate controls ingress, exchange bindings decide fan-out, each queue becomes a separate backlog, and consumers provide the drain. Production operations is mostly the discipline of keeping those four numbers visible and bounded.

For example, three factories publish 1,200 telemetry messages per second into one topic exchange. If the alert queue receives 5% of traffic and has two consumers that each process 40 messages/s, its capacity is 80 messages/s against 60 messages/s of ingress, so it drains. If a new binding doubles alert traffic to 120 messages/s without adding consumers, backlog grows by 40 messages/s: a 50,000-message queue fills in about 21 minutes. That is an alerting problem before it is a data-loss problem.

Define the alert before the release: maximum queue depth, maximum age of the oldest message, minimum active consumers, and maximum unconfirmed publishes. Those thresholds make the broker observable as a production service. If the oldest message is 90 seconds old but the SLA is 30 seconds, the queue is already failing the product promise even if disk is still available and no messages have been dropped.

15.4 Practitioner: Quorum Queues and Connection Hygiene

For data-safety-critical queues, the modern choice in RabbitMQ is the quorum queue. It replicates its contents across an odd number of nodes using the Raft consensus algorithm, so it keeps working as long as a majority (a quorum) of replicas is available and a confirmed message is one a quorum has accepted. It replaces the older classic mirrored queues, which are deprecated for high availability.

Property	Classic queue	Quorum queue
Replication	Single node by default	Raft across an odd set of nodes
Survives a node loss	No (queue unavailable)	Yes, while a majority survives
Best for	Transient, high-churn, non-critical	Orders, commands, anything you cannot lose

Two connection habits prevent self-inflicted outages. First, use separate connections for publishing and consuming: when the broker hits a memory or disk alarm it issues connection.blocked and pauses publishers, and heavy consuming can apply TCP back-pressure — sharing one connection lets consumer flow stall your publishers. Second, pool connections and channels rather than opening one per message; a channel is cheap but not free, and per-message churn exhausts the broker.

Turn those settings into a deployment rule. A three-node quorum queue can tolerate one broker node failing; if two nodes are unavailable, writes stop because no majority can confirm them. For critical commands, set the producer timeout lower than the operator escalation time: if confirms stop for 10 seconds, the publisher should fail fast, surface the queue name and last delivery tag, and switch the device gateway into a degraded mode instead of buffering indefinitely in RAM.

Connection hygiene has the same measurable shape. If 500 edge gateways each open one publishing connection and reuse channels per stream, the broker handles hundreds of stable sockets. If each gateway opens a connection per message at 2 msg/s, the broker sees 1,000 TCP handshakes per second before it processes payloads. That is wasted capacity and a common reason a “small” pilot collapses when it becomes a production fleet.

15.5 Under the Hood: Asynchronous Confirms at Throughput

Synchronous confirms (publish one, block for its ack) are simple but cap you at one message per round trip. The production pattern is asynchronous confirms: call confirm.select once to put the channel in confirm mode, then keep publishing while tracking each message’s delivery tag (a monotonically increasing sequence number) in an outstanding map. The broker acks tags asynchronously and out of band:

• basic.ack with a delivery tag clears that message; with multiple=true it clears every tag up to and including it, so acks arrive in efficient ranges.
• basic.nack signals the broker could not take responsibility for the message — the publisher should resend it.
• a basic.return (only if you published mandatory) arrives first for an unroutable message, because a confirm alone would ack it as handled.

So a robust publisher keeps a map of unconfirmed tags, removes them as acks arrive, resends on nack, and treats a returned message as a routing failure. That gives quorum-level durability and high throughput, because the publisher never blocks waiting for an individual ack. This is the state you want before flipping a fleet from pilot to production.

The throughput difference is large. With a 25 ms broker round trip, synchronous confirms cap one channel near 1000 / 25 = 40 msg/s. If the same channel keeps a window of 1,000 outstanding confirms, the theoretical pipe becomes roughly 1000 / 0.025 = 40,000 msg/s before broker, disk, and network limits. The outstanding map is the safety valve: cap it, expose its size as a metric, and pause publishing when it approaches the window instead of letting memory grow without bound.

Pair that publisher metric with queue and consumer signals. Alert when queue depth grows for more than one drain interval, when unacknowledged messages exceed consumers x prefetch_count, or when dead-letter rate rises above the expected poison-message baseline. Those thresholds connect protocol mechanics to operations: confirms protect accepted writes, queues reveal backlog, ACKs reveal consumer health, and DLQs reveal data the normal path could not process.