1243  AMQP Message Structure and Delivery Guarantees

1243.1 Learning Objectives

By the end of this chapter, you will be able to:

• Analyze Message Structure: Understand AMQP message format including headers, properties, and body sections
• Configure Delivery Modes: Set message persistence, priority, and time-to-live (TTL) properties
• Implement Delivery Guarantees: Choose and implement at-most-once, at-least-once, and exactly-once semantics
• Apply Acknowledgments: Use publisher confirms and consumer acknowledgments for reliable messaging
• Handle Message Failures: Configure dead-letter queues and negative acknowledgments

1243.2 Prerequisites

Before diving into this chapter, you should be familiar with:

• AMQP Core Architecture: This chapter builds on understanding of exchanges, queues, bindings, and the producer-broker-consumer model
• AMQP Fundamentals: Core AMQP concepts and protocol basics
• Networking Basics: TCP connections and reliable delivery concepts

NoteRelated Chapters

Deep Dives: - AMQP Core Architecture - Exchanges, queues, and bindings - AMQP Features and Frames - Advanced features and protocol frames - AMQP Implementations and Labs - Hands-on broker setup

Related Concepts: - MQTT QoS Levels - Compare with MQTT delivery guarantees - CoAP Reliability - RESTful reliability patterns

TipFor Beginners: Message Delivery

Think of AMQP message delivery like sending registered mail:

• At-Most-Once: Regular mail - might get lost, but fastest and cheapest
• At-Least-Once: Tracked mail with retry - guaranteed to arrive, might arrive twice if tracking fails
• Exactly-Once: Registered mail with signature - guaranteed exactly one delivery, most expensive

Key terms:

Term	Simple Explanation
Persistent	Message saved to disk, survives broker restart
ACK	Consumer says “I processed this message successfully”
NACK	Consumer says “I failed to process this, please retry”
Dead Letter Queue	Special queue for messages that can’t be processed
Publisher Confirm	Broker tells producer “I received your message”

---

1243.3 Message Structure

An AMQP message consists of multiple sections that provide metadata and payload.

1243.3.1 AMQP Message Format

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#16A085', 'secondaryColor': '#E67E22', 'tertiaryColor': '#fff', 'noteTextColor': '#2C3E50', 'noteBkgColor': '#fff', 'textColor': '#2C3E50', 'fontSize': '14px'}}}%%
graph TB
    M[AMQP Message]
    M --> H[Header<br/>Durable, Priority, TTL]
    M --> P[Properties<br/>Content-Type, Message-ID,<br/>Correlation-ID, Reply-To]
    M --> B[Body<br/>Message Payload<br/>Binary or Text]

    style M fill:#E67E22,stroke:#16A085,stroke-width:3px,color:#fff
    style H fill:#2C3E50,stroke:#16A085,stroke-width:2px,color:#fff
    style P fill:#16A085,stroke:#2C3E50,stroke-width:2px,color:#fff
    style B fill:#2C3E50,stroke:#16A085,stroke-width:2px,color:#fff

Figure 1243.1: AMQP message structure with header, properties, and body sections

1243.3.2 Header Section

The header contains delivery-related metadata:

Field	Description	Common Values
Durable	Message survives broker restart	True (persistent), False (transient)
Priority	Message priority level	0-9 (0 lowest, 9 highest)
TTL	Time-To-Live before expiration	Milliseconds (e.g., 60000 for 1 minute)
First-Acquirer	First consumer to receive	Boolean
Delivery-Count	Number of delivery attempts	Integer

Example: Setting persistent, high-priority message:

import pika

properties = pika.BasicProperties(
    delivery_mode=2,  # Persistent (survives restart)
    priority=8,       # High priority (0-9 scale)
    expiration='60000'  # TTL: 60 seconds
)

channel.basic_publish(
    exchange='orders',
    routing_key='order.priority',
    body='{"order_id": 12345}',
    properties=properties
)

1243.3.3 Properties Section

Properties provide message metadata for routing and processing:

Property	Description	Example
Content-Type	MIME type of body	application/json, text/plain
Content-Encoding	Encoding applied	gzip, utf-8
Correlation-ID	Links related messages	UUID for request-reply matching
Reply-To	Return address queue	reply_queue_abc123
Message-ID	Unique identifier	UUID for deduplication
Timestamp	Creation time	Unix timestamp
Type	Application-specific type	sensor.reading, order.created
App-ID	Producing application	temperature-sensor-01

Example: Complete message with properties:

properties = pika.BasicProperties(
    content_type='application/json',
    content_encoding='utf-8',
    message_id=str(uuid.uuid4()),
    correlation_id=request_correlation_id,
    reply_to='response_queue',
    timestamp=int(time.time()),
    type='sensor.temperature',
    app_id='weather-station-01',
    delivery_mode=2
)

1243.3.4 Body Section

The body contains the actual message payload:

• Can be binary or text
• Encoding specified in properties
• No size limit (but broker may impose limits)
• Common formats: JSON, Protocol Buffers, MessagePack

---

1243.4 Delivery Guarantees

AMQP provides configurable delivery semantics to match application requirements.

1243.4.1 Three Delivery Modes

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#16A085', 'secondaryColor': '#E67E22', 'tertiaryColor': '#fff', 'noteTextColor': '#2C3E50', 'noteBkgColor': '#fff', 'textColor': '#2C3E50', 'fontSize': '14px'}}}%%
graph LR
    A1[At-Most-Once<br/>0 or 1 delivery] -->|No ACK, Fast| U1[Best effort<br/>May lose messages]

    A2[At-Least-Once<br/>1+ deliveries] -->|ACK required, Retries| U2[Guaranteed delivery<br/>Possible duplicates]

    A3[Exactly-Once<br/>Exactly 1 delivery] -->|Transactions, Deduplication| U3[No loss<br/>No duplicates]

    style A1 fill:#2C3E50,stroke:#16A085,stroke-width:2px,color:#fff
    style A2 fill:#16A085,stroke:#2C3E50,stroke-width:2px,color:#fff
    style A3 fill:#E67E22,stroke:#16A085,stroke-width:2px,color:#fff
    style U1 fill:#7F8C8D,stroke:#2C3E50,stroke-width:2px,color:#fff
    style U2 fill:#7F8C8D,stroke:#2C3E50,stroke-width:2px,color:#fff
    style U3 fill:#7F8C8D,stroke:#2C3E50,stroke-width:2px,color:#fff

Figure 1243.2: AMQP delivery guarantee levels: at-most-once, at-least-once, exactly-once

1243.4.2 1. At-Most-Once (0 or 1)

Message delivered once or not at all. Fastest but may lose messages.

Characteristics:

• No acknowledgment required
• Fastest, lowest overhead
• Fire-and-forget pattern

Implementation:

# Producer: No confirms
channel.basic_publish(
    exchange='sensors',
    routing_key='temperature',
    body='22.5',
    properties=pika.BasicProperties(delivery_mode=1)  # Transient
)

# Consumer: Auto-acknowledge
channel.basic_consume(
    queue='sensor_data',
    on_message_callback=callback,
    auto_ack=True  # Message removed immediately on delivery
)

Use cases:

• Non-critical telemetry
• Sensor readings where occasional loss is acceptable
• High-throughput streaming where speed matters more than completeness

1243.4.3 2. At-Least-Once (1 or more)

Message guaranteed to be delivered, but may arrive multiple times.

Characteristics:

• Requires acknowledgment
• Retries on failure
• Possible duplicates (consumer must handle)

Implementation:

# Producer: With confirms
channel.confirm_delivery()

try:
    channel.basic_publish(
        exchange='orders',
        routing_key='order.created',
        body=json.dumps(order),
        properties=pika.BasicProperties(
            delivery_mode=2,  # Persistent
            message_id=str(uuid.uuid4())
        )
    )
except pika.exceptions.UnroutableError:
    # Handle unroutable message
    retry_or_log(order)

# Consumer: Manual acknowledgment
def callback(ch, method, properties, body):
    try:
        process_order(body)
        ch.basic_ack(delivery_tag=method.delivery_tag)
    except Exception:
        ch.basic_nack(delivery_tag=method.delivery_tag, requeue=True)

channel.basic_consume(
    queue='orders',
    on_message_callback=callback,
    auto_ack=False  # Manual ACK required
)

Use cases:

• Important events and commands
• Order processing (with idempotent handlers)
• Notifications that must be delivered

1243.4.4 3. Exactly-Once (exactly 1)

Message delivered exactly once with no loss or duplicates.

Characteristics:

• Uses transactions or deduplication
• Highest reliability, highest overhead
• Most complex to implement

Implementation (transaction-based):

# Producer: Transactional publishing
channel.tx_select()
try:
    channel.basic_publish(
        exchange='payments',
        routing_key='payment.process',
        body=json.dumps(payment),
        properties=pika.BasicProperties(delivery_mode=2)
    )
    channel.tx_commit()
except Exception:
    channel.tx_rollback()
    raise

# Consumer: With deduplication
def callback(ch, method, properties, body):
    message_id = properties.message_id

    # Check if already processed
    if redis.exists(f'processed:{message_id}'):
        ch.basic_ack(delivery_tag=method.delivery_tag)
        return

    try:
        result = process_payment(body)
        redis.setex(f'processed:{message_id}', 86400, '1')  # 24h TTL
        ch.basic_ack(delivery_tag=method.delivery_tag)
    except Exception:
        ch.basic_nack(delivery_tag=method.delivery_tag, requeue=True)

Use cases:

• Financial transactions
• Critical commands that cannot be duplicated
• Billing and accounting events

---

1243.5 Publisher Confirms

Publisher confirms allow producers to know when messages are safely received by the broker.

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#16A085', 'secondaryColor': '#E67E22', 'tertiaryColor': '#fff', 'noteTextColor': '#2C3E50', 'noteBkgColor': '#fff', 'textColor': '#2C3E50', 'fontSize': '14px'}}}%%
sequenceDiagram
    participant P as Publisher
    participant B as Broker

    P->>B: 1. Publish message
    B->>B: 2. Persist to disk
    B->>P: 3. Confirm (ACK)
    Note over P: Message safely<br/>received

Figure 1243.3: Publisher confirm sequence: publish, persist, and broker acknowledgment

Implementation:

# Enable publisher confirms
channel.confirm_delivery()

# Synchronous confirm (blocking)
channel.basic_publish(
    exchange='orders',
    routing_key='order.new',
    body=order_json,
    mandatory=True
)
# Raises exception if not confirmed

# Asynchronous confirms (non-blocking)
def on_confirm(frame):
    if frame.method.NAME == 'Basic.Ack':
        print(f"Message {frame.method.delivery_tag} confirmed")
    else:
        print(f"Message {frame.method.delivery_tag} rejected")

channel.add_on_return_callback(on_confirm)

---

1243.6 Consumer Acknowledgments

Consumer acknowledgments ensure messages are processed successfully before removal from queues.

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#16A085', 'secondaryColor': '#E67E22', 'tertiaryColor': '#fff', 'noteTextColor': '#2C3E50', 'noteBkgColor': '#fff', 'textColor': '#2C3E50', 'fontSize': '14px'}}}%%
sequenceDiagram
    participant B as Broker
    participant C as Consumer

    B->>C: 1. Deliver message
    C->>C: 2. Process message
    alt Success
        C->>B: 3a. ACK (remove from queue)
    else Failure
        C->>B: 3b. NACK (requeue or DLQ)
    end

Figure 1243.4: Consumer acknowledgment flow with success ACK and failure NACK paths

1243.6.1 ACK Types

Type	Method	Effect
Positive ACK	basic_ack()	Message removed from queue
Negative ACK	basic_nack()	Message requeued or sent to DLQ
Reject	basic_reject()	Single message reject (legacy)

1243.6.2 Prefetch Count

Prefetch controls how many unacknowledged messages a consumer can hold.

# Limit to 1 unacknowledged message at a time
channel.basic_qos(prefetch_count=1)

# Best for:
# - Long-running tasks (prevents one consumer hoarding all work)
# - Fair distribution across multiple consumers

Prefetch recommendations:

Task Duration	Recommended Prefetch	Reason
< 100ms	50-100	Reduce round-trip overhead
100ms - 1s	10-20	Balance throughput and fairness
1s - 10s	1-5	Prevent consumer overload
> 10s	1	One task at a time

---

1243.7 Dead Letter Queues

Dead letter queues (DLQ) capture messages that cannot be processed successfully.

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#16A085', 'secondaryColor': '#E67E22', 'tertiaryColor': '#fff', 'noteTextColor': '#2C3E50', 'noteBkgColor': '#fff', 'textColor': '#2C3E50', 'fontSize': '14px'}}}%%
graph LR
    E[Exchange] -->|route| Q1[Main Queue]
    Q1 -->|consume| C[Consumer]
    C -.->|NACK requeue=false| Q1
    Q1 -->|failed messages| DLX[Dead Letter<br/>Exchange]
    DLX -->|route| DLQ[Dead Letter<br/>Queue]
    DLQ -->|manual investigation| A[Admin]

    style E fill:#E67E22,stroke:#16A085,stroke-width:2px,color:#fff
    style Q1 fill:#16A085,stroke:#2C3E50,stroke-width:2px,color:#fff
    style C fill:#2C3E50,stroke:#16A085,stroke-width:2px,color:#fff
    style DLX fill:#E67E22,stroke:#2C3E50,stroke-width:2px,color:#fff
    style DLQ fill:#7F8C8D,stroke:#2C3E50,stroke-width:2px,color:#fff
    style A fill:#2C3E50,stroke:#16A085,stroke-width:2px,color:#fff

Figure 1243.5: Dead letter queue pattern for handling failed messages

1243.7.1 DLQ Triggers

Messages are sent to DLQ when:

1. Consumer rejects with requeue=false: basic_nack(requeue=False)
2. Message TTL expires: Message exceeded time-to-live
3. Queue length exceeded: Queue reached maximum length
4. Message rejected after max retries: Application-level retry exhaustion

1243.7.2 DLQ Configuration

# Declare dead letter exchange
channel.exchange_declare(
    exchange='dlx',
    exchange_type='direct'
)

# Declare dead letter queue
channel.queue_declare(queue='dead_letters')
channel.queue_bind(queue='dead_letters', exchange='dlx', routing_key='failed')

# Declare main queue with DLQ configuration
channel.queue_declare(
    queue='orders',
    arguments={
        'x-dead-letter-exchange': 'dlx',
        'x-dead-letter-routing-key': 'failed',
        'x-message-ttl': 300000  # Optional: 5 min TTL
    }
)

---

1243.8 Knowledge Check

Test your understanding of message structure and delivery guarantees.

NoteQuiz: Delivery Guarantees

Question 1: A smart home system uses AMQP to route commands from a mobile app to IoT devices. Temperature sensor readings are sent every 10 seconds (non-critical), while door lock commands are security-critical and must be guaranteed. Which delivery guarantee strategies should be used?

Explanation: Different data types require different delivery guarantees based on criticality and overhead tolerance:

Temperature readings (At-Most-Once): - Fire-and-forget delivery (no ACK overhead) - If one reading lost, next comes in 10 seconds - Minimal network and broker overhead

Lock commands (At-Least-Once with Idempotency): - Guaranteed delivery with manual acknowledgment - Duplicates handled safely (lock/unlock are idempotent) - Consumer checks message_id before processing

Why not exactly-once for both? - Exactly-once requires transactions (3x overhead) - Temperature readings don’t need transactional guarantees - Wastes resources on high-frequency, non-critical data

Why at-most-once is wrong for locks: - Lock commands could be lost on network glitch - User stuck outside if unlock command never arrives - Security breach potential

Question 2: An industrial IoT system processes machine diagnostics via AMQP. A consumer receives a diagnostic message, processes it for 15 seconds, but crashes before sending acknowledgment. The broker has been configured with auto_ack=False and prefetch_count=1. What happens to the message?

Explanation: With manual acknowledgment (auto_ack=False), messages remain “in-flight” until explicitly acknowledged.

Timeline: 1. Broker delivers message to Consumer A 2. Message status: “unacknowledged” (awaiting ACK) 3. Consumer A crashes without sending ACK 4. Broker detects connection loss 5. Broker requeues message (back to front of queue) 6. Another consumer (or same after restart) receives message

Key points: - auto_ack=False = message NOT removed until ACK received - prefetch_count=1 = only 1 unacked message at a time - Connection loss = all unacked messages requeued

Why other options are wrong: - B: Message not lost because ACK never sent - C: DLQ only for explicit NACK(requeue=false) or TTL expiry - D: Broker doesn’t wait - immediately requeues for other consumers

---

1243.9 Summary

This chapter covered AMQP message structure and delivery guarantees:

• Message Format: Analyzed header (durable, priority, TTL), properties (content-type, correlation-ID, message-ID), and body sections
• Delivery Modes: Compared at-most-once (fast, may lose), at-least-once (guaranteed, may duplicate), and exactly-once (no loss, no duplicates)
• Publisher Confirms: Implemented broker acknowledgment to producers for reliable publishing
• Consumer Acknowledgments: Applied manual ACK/NACK with prefetch control for processing guarantees
• Dead Letter Queues: Configured DLQ for capturing failed messages for investigation
• Idempotency: Used message-ID for deduplication in at-least-once scenarios

1243.10 What’s Next

The next chapter AMQP Features and Frames explores advanced AMQP capabilities including security (SASL, TLS), interoperability, and the AMQP 1.0 frame types for protocol-level understanding.