11 AMQP Prefetch Reliability Contracts
Start with the story: Prefetch is the rule that stops one slow worker from grabbing the whole pile of jobs. It sets how many unacknowledged messages a consumer may hold before the broker gives other workers a chance.
11.1 Learning Objectives
After this page, you should be able to:
- Explain how
basic.qos(prefetch_count=N)bounds the unacknowledged work assigned to a consumer. - Size prefetch from handler time, round-trip latency, fairness, and crash-redelivery cost.
- Diagnose prefetch starvation from ready-message and unacked-message metrics.
- Explain why unlimited prefetch can make a broker look idle while work is pinned to a stalled consumer.
- Choose monitoring thresholds that match the intended prefetch policy.
11.2 Why This Follows AMQP Reliability Patterns
AMQP Reliability Patterns teaches acknowledgments, persistence, dead-lettering, queue limits, and competing consumers. This page tightens one operational contract inside that reliability model: manual acknowledgments need a deliberate prefetch window, or one stalled worker can hoard unacked messages and cause large redelivery bursts.
Use it when a queue appears to drain but throughput stalls, when competing consumers process at different speeds, when crash redelivery must be bounded, or when a runbook needs a concrete maximum unacked-message count.
11.3 Overview: Prefetch Is the Consumer’s Reliability Dial
Durable queues and persistent messages protect the broker side, but the consumer side has its own decisive knob: prefetch, set with basic.qos(prefetch_count=N). Prefetch is the number of messages the broker will hand a consumer before it must acknowledge any of them. It quietly controls three things at once — throughput, fair distribution across workers, and how much work is at risk if a consumer crashes.
The trap is treating it as “bigger is faster”. A prefetch of 1 is the fairest and safest setting but can starve throughput; a very high prefetch can make one worker hoard a burst while others sit idle, and turns a single crash into a large redelivery. Reliability tuning is finding the prefetch that keeps a consumer busy without hoarding.
Concrete example: a queue has 3,000 telemetry enrichment jobs and three consumers. With unlimited prefetch, the first fast TCP connection can receive hundreds or thousands of deliveries before the broker notices other consumers are available. If that consumer then blocks on a database call, those deliveries are invisible to the other workers because they are already assigned and unacknowledged. With prefetch_count=20, the stalled worker can hold only 20 at a time, leaving the broker free to keep feeding the other two consumers.
This is why prefetch belongs in the reliability conversation, not just the performance conversation. It defines the largest batch that can be delayed or redelivered by one consumer failure.
A good first setting is usually based on the worst acceptable redelivery burst. If a handler writes to an idempotent database table, a redelivery window of 20 may be harmless. If each message triggers a physical device action, even five unacked commands may be too many to replay at once. The broker cannot infer that risk from the queue name; the application owner has to set the window deliberately.
11.4 Practitioner: Sizing Prefetch to the Round Trip
With prefetch=1, a consumer processes a message, sends basic.ack, and only then receives the next one. During the network round trip it is idle, so throughput is capped at roughly one message per (processing time + round trip). To keep the pipe full you need enough messages in flight to cover the round trip while one is processing:
useful prefetch ≈ ceil((round_trip + processing_time) / processing_time)
| Prefetch | Effect | Best for |
|---|---|---|
| 1 | Perfectly fair; throughput limited by the ack round trip | Slow, uneven, or long tasks where fairness matters most |
| Small (e.g. 10-50) | Hides round-trip latency, still fairly even | Typical fast handlers — the usual sweet spot |
| Very high / unlimited | One consumer hoards bursts; large redelivery on crash; high memory | Almost never; a common misconfiguration |
Worked example. Handlers take 5 ms each; the broker round trip is 20 ms. At prefetch 1, each consumer does about one message per 25 ms (40/s). Raising prefetch to ceil((20+5)/5) = 5 lets five messages overlap the round trip, so the consumer stays busy and approaches 200/s — a 5x gain with no change to durability or acks. Pushing prefetch to 10,000 gains nothing more and risks one worker grabbing the whole backlog.
The same arithmetic flips for slow handlers. If image analysis takes 500 ms and the broker round trip is 20 ms, ceil((20+500)/500) = 2 is already enough to hide network latency. A prefetch of 100 would let one worker reserve 50 seconds of work before anyone else can see it. Choose prefetch from processing time and failure cost, not from a habit of using large round numbers.
Use the measured p95 handler time, not the happy-path average, when sizing production queues. A handler that is normally 5 ms but becomes 100 ms during database compaction should not be given a window that lets it reserve minutes of work. Start with a small value, watch per-consumer unacked counts, and raise it only when consumers are idle because of round-trip latency.
11.5 Under the Hood: Unacked Messages and Prefetch Starvation
The broker tracks every delivered-but-unacknowledged message per consumer and counts it against that consumer’s prefetch. Two consequences follow. First, if a consumer reaches its prefetch limit and stops acking — because it deadlocked, is waiting on a slow database, or simply forgot to ack — the broker sends it nothing more. Its share of the queue is frozen even though the queue is full and other consumers may be idle. This is prefetch starvation, and it looks exactly like a broker problem when it is really a missing or delayed ack.
Second, unacked messages are the recovery unit. If the consumer’s channel or connection drops, the broker requeues everything still unacked for that consumer and marks it redelivered=true so the next consumer knows it may be a repeat. A large prefetch therefore means a large redelivery burst on failure. The reliable pattern is manual ack with a modest prefetch: enough in flight to stay fast, few enough that a crash redelivers a small, quickly-reprocessed batch.
Operationally, a prefetch problem shows up as a mismatch between queue depth and worker activity. The queue may appear to drain because messages were delivered, but useful throughput stalls because they are sitting unacked inside one process. Check per-consumer unacked counts, not only ready-message counts. A healthy bounded setup has ready messages falling, unacked messages spread across consumers, and acknowledgements continuing at roughly the handler rate.
The monitoring threshold should match the prefetch policy. With three consumers and prefetch 20, more than 60 unacked messages means the configuration is not what you think, or consumers are using multiple channels. With unlimited prefetch, the same count tells you very little until a stall has already trapped a large batch. This is why reliability runbooks should record both the intended prefetch and the expected maximum unacked total.