1248  AMQP Routing Patterns and Exercises

1248.1 Learning Objectives

By the end of this chapter, you will be able to:

• Design Topic Exchange Patterns: Create routing keys and binding patterns for complex IoT message distribution
• Calculate Queue Throughput: Estimate message rates, storage requirements, and bandwidth for capacity planning
• Implement Offline Consumer Support: Configure durable queues to buffer messages during maintenance windows
• Apply Delivery Guarantees: Choose and implement at-most-once, at-least-once, or exactly-once semantics based on requirements
• Compare AMQP Routing with MQTT: Evaluate trade-offs between server-side routing (AMQP) and client-side filtering (MQTT)
• Handle Consumer Failures Gracefully: Design systems that recover from offline consumers without data loss

1248.2 Prerequisites

Before diving into this chapter, you should be familiar with:

• AMQP Fundamentals: Understanding of exchanges, queues, and bindings
• AMQP Implementation Misconceptions: Common pitfalls to avoid in routing patterns
• AMQP Architecture and Frames: Exchange types and message structure

TipFor Beginners: How to Use This Chapter

This chapter provides hands-on exercises to practice AMQP routing patterns. Work through the scenarios in order:

1. Knowledge Check Questions: Test your understanding of routing concepts
2. Message Routing Design Exercise: Design a complete AMQP topology for a manufacturing system
3. Throughput Calculations: Learn to estimate system capacity requirements

Each section builds on the previous one. If you get stuck on a Knowledge Check question, review the detailed explanation before moving on.

1248.3 Knowledge Check: Routing Patterns

Test your understanding of AMQP routing, delivery guarantees, and consumer behavior with these practical scenario questions.

1248.3.1 Scenario 1: Offline Consumer Buffering

Question: An analytics service processes sensor data in batches and has a scheduled 30-minute maintenance window every night (2:00 AM - 2:30 AM). During this window, the consumer is offline. Sensors continue publishing 10 messages/second. The analytics queue is configured as durable with persistent messages. What happens to the 18,000 messages published during the maintenance window?

💡 Explanation: This demonstrates AMQP’s queue persistence and offline consumer support:

Complete Message Flow During Maintenance:

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#E67E22', 'secondaryColor': '#16A085', 'tertiaryColor': '#7F8C8D'}}}%%
sequenceDiagram
    participant S as Sensors
    participant B as AMQP Broker
    participant C as Analytics Consumer

    Note over S,C: Before Maintenance (1:59 AM)
    S->>B: Publish (10 msg/sec)
    B->>C: Deliver messages
    Note over C: Processing real-time

    Note over S,C: Maintenance Window (2:00-2:30 AM)
    C-xB: Disconnect (maintenance)
    S->>B: Publish continues (10 msg/sec)
    Note over B: Queue depth: 0→18,000<br/>Messages persist to disk<br/>Durable queue

    Note over S,C: After Maintenance (2:30 AM)
    C->>B: Reconnect
    B->>C: Deliver 18,000 queued messages
    Note over C: Process backlog<br/>30 min to clear

Figure 1248.1: AMQP message queuing during consumer maintenance window

Detailed Breakdown:

During Maintenance (Consumer Offline):

Time: 2:00 AM - 2:30 AM (30 minutes)
Messages published: 10 msg/sec × 1800 sec = 18,000 messages
Queue behavior:
  ├─ Accepts all incoming messages
  ├─ Writes to disk (messages are PERSISTENT)
  ├─ Queue depth grows: 0 → 18,000
  └─ No messages lost

Disk storage calculation:
  Average message: 200 bytes (sensor ID, timestamp, value, metadata)
  Total storage: 18,000 × 200 bytes = 3.6 MB

After Maintenance (Consumer Returns):

Time: 2:30 AM onwards
Consumer reconnects: Subscribes to analytics queue
Queue delivers: FIFO order (oldest first)
Processing rate: 10 msg/sec (consumer's normal rate)
Time to clear backlog: 18,000 / 10 = 1800 sec = 30 minutes
Complete by: 3:00 AM

Key AMQP Configuration for Offline Support:

# Queue declaration (enables persistence)
channel.queue_declare(
    queue='analytics-queue',
    durable=True,          # ← Queue survives broker restart
    exclusive=False,       # Multiple consumers allowed
    auto_delete=False      # Don't delete when consumer disconnects
)

# Message publishing (persistent messages)
channel.basic_publish(
    exchange='sensor-data',
    routing_key='sensor.temperature.line1.machine3',
    body='{"temp": 75.3, "timestamp": 1698765432}',
    properties=pika.BasicProperties(
        delivery_mode=2,   # ← PERSISTENT (survives broker restart)
        content_type='application/json'
    )
)

Why Other Options Are Wrong:

• A (Messages lost): Only true for non-durable queues or transient messages
• C (Automatic backup): AMQP doesn’t automatically spawn backup consumers
• D (Broker rejects): Broker accepts until configured limits (max-length, max-length-bytes)

1248.3.2 Scenario 2: Exactly-Once Delivery for Critical Commands

Question: A critical industrial control system uses AMQP for command delivery to actuators controlling a chemical mixing process. Commands must be executed exactly once (no duplicates, no omissions) because duplicate execution could cause dangerous overfilling. The system experiences a network glitch causing brief disconnection. Which AMQP delivery mode and consumer acknowledgment strategy ensures exactly-once execution?

💡 Explanation: This demonstrates AMQP delivery guarantees for critical systems:

Exactly-Once with Idempotency Keys:

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#E67E22', 'secondaryColor': '#16A085', 'tertiaryColor': '#7F8C8D'}}}%%
sequenceDiagram
    participant P as Publisher
    participant B as Broker
    participant C as Consumer
    participant D as Dedup DB

    P->>B: Publish (id=cmd-001)
    B->>C: Deliver
    C->>D: Check executed(cmd-001)?
    D-->>C: No
    C->>C: Execute command
    C->>D: Mark executed(cmd-001)
    C->>B: ACK
    alt ACK lost
        Note over B: Timeout
        B->>C: Redeliver
        C->>D: Check executed(cmd-001)?
        D-->>C: Yes! Skip
        C->>B: ACK (skip execution)
    end

Figure 1248.2: AMQP exactly-once delivery using idempotency key deduplication

Implementation with Idempotency:

# ✅ EXACTLY-ONCE - Idempotency prevents duplicates
executed_ids = set()  # Or use Redis/database for persistence

def on_message(channel, method, properties, body):
    command_id = properties.message_id  # Idempotency key

    if command_id in executed_ids:
        print(f"Duplicate {command_id} detected, skipping execution")
    else:
        execute_command(body)
        executed_ids.add(command_id)

    channel.basic_ack(delivery_tag=method.delivery_tag)

Why At-Least-Once with Manual ACK (Option C) is Insufficient:

# CLOSE, but still allows duplicates
def on_message(channel, method, properties, body):
    execute_command(body)  # Execute first
    channel.basic_ack(method.delivery_tag)  # ACK after

# If ACK lost after execution:
# 1. Broker redelivers (correct behavior)
# 2. Consumer executes AGAIN (wrong - duplicate!)
# 3. Dangerous for critical systems

Delivery Mode Summary:

Mode	Guarantees	Use Case
At-Most-Once	0 or 1 delivery	Non-critical telemetry
At-Least-Once	1+ deliveries (may duplicate)	Important events (consumer handles duplicates)
Exactly-Once	Exactly 1 execution	Financial transactions, critical commands

1248.3.3 Scenario 3: Topic Exchange Wildcard Patterns

Question: A manufacturing system publishes quality control readings with routing keys: “sensor.visual.line1.station3”, “sensor.pressure.line1.station5”, “sensor.visual.line2.station1”. The defect analysis system needs to subscribe to ALL visual inspection data from ALL lines and stations, but NOT pressure or other sensor types. Using AMQP topic exchange, what binding pattern achieves this?

💡 Explanation: Understanding AMQP topic exchange wildcard routing:

Routing Key Structure:

sensor.visual.line1.station3
  │      │     │       │
  │      │     │       └─ word 4: station ID
  │      │     └───────── word 3: line ID
  │      └─────────────── word 2: sensor type
  └────────────────────── word 1: category

Wildcard Rules: - * matches exactly one word - # matches zero or more words

Pattern Analysis:

Pattern	Matches sensor.visual.line1.station3?	Why
sensor.visual.*	❌ No	Expects 3 words, has 4
sensor.visual.#	✅ Yes	# matches “line1.station3”
sensor.visual.*.station.*	❌ No	Wrong format (expects 5 words)
#.visual.#	✅ Yes, but too broad	Matches any key containing “visual”

Correct Pattern: sensor.visual.#

# Bind queue with correct pattern
channel.queue_bind(
    exchange='sensor-data',
    queue='defect-analysis',
    routing_key='sensor.visual.#'  # ← Matches all visual sensors
)

# Test routing:
# ✓ sensor.visual.line1.station3 → MATCH
# ✓ sensor.visual.line2.station1 → MATCH
# ✓ sensor.visual.line3.station7.camera2 → MATCH (future-proof)
# ✗ sensor.pressure.line1.station5 → NO MATCH (correct rejection)

Scalability Benefits: - Single binding rule handles infinite line/station combinations - New equipment automatically included without config changes - Efficient broker routing: O(1) per message

1248.3.4 Scenario 4: Routing Key Pattern Matching

Question: A manufacturing system publishes equipment sensor data to AMQP. Engineers need to bind queues to a topic exchange using these routing patterns:

• Pattern A: “sensor.temperature.*”
• Pattern B: “sensor.*.line1”
• Pattern C: “sensor.#”
• Pattern D: “sensor.temperature.line1.machine3”

Given a message with routing key “sensor.temperature.line1.machine3”, which binding patterns will route this message to their respective queues?

💡 Explanation: Let’s evaluate each pattern against “sensor.temperature.line1.machine3” (4 words):

**Pattern A: “sensor.temperature.*“ - Expects 3 words: sensor, temperature, (one word) - Routing key has 4 words - Does NOT match** ✗

**Pattern B: “sensor.*.line1” - Expects 3 words: sensor, (one word), line1 - Routing key has 4 words - Does NOT match** ✗

Pattern C: “sensor.#” - Matches “sensor” followed by zero or more words - # matches “temperature.line1.machine3” (3 words) - Matches ✓

Pattern D: “sensor.temperature.line1.machine3” - Exact match with routing key - Matches ✓

Critical Insight: * matches exactly one word (not one-or-more), so patterns A and B fail because they expect fewer levels than the routing key provides.

Pattern	Expected Words	Actual Words	Match?
sensor.temperature.*	3	4	✗
sensor.*.line1	3	4	✗
sensor.#	1+ (any)	4	✓
sensor.temperature.line1.machine3	4	4	✓

1248.3.5 Scenario 5: Message Persistence Configuration

Question: A data processing pipeline uses AMQP with the following Python code for publishing sensor readings:

# Option A
channel.basic_publish(
    exchange='sensors',
    routing_key='temp.warehouse.zone1',
    body=json.dumps({'temp': 22.5, 'timestamp': 1609459200})
)

# Option B
channel.basic_publish(
    exchange='sensors',
    routing_key='temp.warehouse.zone1',
    body=json.dumps({'temp': 22.5, 'timestamp': 1609459200}),
    properties=pika.BasicProperties(
        delivery_mode=2,
        content_type='application/json'
    )
)

# Option C
channel.basic_publish(
    exchange='sensors',
    routing_key='temp.warehouse.zone1',
    body=json.dumps({'temp': 22.5, 'timestamp': 1609459200}),
    mandatory=True
)

The system requires messages to survive broker restarts. Which code option ensures message persistence?

💡 Explanation: In AMQP, message persistence requires explicit configuration:

Option B is correct because delivery_mode=2 explicitly marks messages as persistent, causing the broker to write them to disk before acknowledging receipt.

Option A sends transient messages (delivery_mode=1 by default), stored only in memory and lost on restart.

Option C uses mandatory=True, which is unrelated to persistence - it causes the broker to return messages that cannot be routed to any queue.

Option D is incorrect because both queue durability AND message persistence are required:

Configuration	Result
Durable queue + transient messages	Queue survives restart (empty), messages lost
Non-durable queue + persistent messages	Queue gone, messages gone
Durable queue + persistent messages	✓ Complete reliability

Complete persistence configuration:

# 1. Durable queue
channel.queue_declare(queue='sensor-queue', durable=True)

# 2. Persistent messages
properties = pika.BasicProperties(delivery_mode=2)

# 3. Durable exchange (recommended)
channel.exchange_declare(exchange='sensors', durable=True)

1248.4 Hands-On Exercise: Message Routing Design

1248.4.1 Exercise Objective

Design a complete AMQP routing topology for a smart manufacturing system, including exchange configuration, queue bindings, throughput calculations, and failure handling.

1248.4.2 Scenario

Manufacturing plant with: - 100 sensors (temperature, pressure, vibration) - 20 machines (status updates every 10 seconds) - 5 production lines - 3 consumer applications: 1. Real-time monitoring dashboard 2. Predictive maintenance system 3. Production analytics (batch processing)

1248.4.3 Task 1: Design Exchange and Queue Topology

Requirements: - Dashboard needs ALL sensor data in real-time - Maintenance needs only vibration and temperature anomalies - Analytics needs all data but can process in batches

Solution:

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#E67E22', 'secondaryColor': '#16A085', 'tertiaryColor': '#7F8C8D'}}}%%
graph TD
    subgraph Publishers
        S1[100 Sensors]
        M1[20 Machines]
    end

    subgraph Exchange
        TE[Topic Exchange<br/>sensors]
    end

    subgraph Queues
        Q1[Dashboard Queue<br/>All messages]
        Q2[Maintenance Queue<br/>Temp + Vibration]
        Q3[Analytics Queue<br/>All messages batched]
    end

    S1 & M1 -->|Routing keys:<br/>sensor.temp.line1.m3<br/>sensor.vibr.line2.m7| TE
    TE -->|Pattern: #| Q1
    TE -->|Pattern: sensor.temp.#<br/>sensor.vibr.#| Q2
    TE -->|Pattern: #| Q3

    Q1 --> D[Dashboard<br/>Real-time]
    Q2 --> MA[Maintenance<br/>Anomaly detection]
    Q3 --> AN[Analytics<br/>Batch processing]

    style TE fill:#E67E22,stroke:#2C3E50,color:#fff
    style Q1 fill:#16A085,stroke:#2C3E50,color:#fff
    style Q2 fill:#16A085,stroke:#2C3E50,color:#fff
    style Q3 fill:#16A085,stroke:#2C3E50,color:#fff

Figure 1248.3: AMQP topic exchange routing sensor data to multiple queues

Routing key design: - sensor.temperature.line1.machine3 - sensor.vibration.line2.machine7 - sensor.pressure.line1.machine3

Binding configuration:

# Dashboard: receives everything
channel.queue_bind(exchange='sensor-data', queue='dashboard',
                   routing_key='#')

# Maintenance: temperature and vibration only
channel.queue_bind(exchange='sensor-data', queue='maintenance',
                   routing_key='sensor.temperature.#')
channel.queue_bind(exchange='sensor-data', queue='maintenance',
                   routing_key='sensor.vibration.#')

# Analytics: receives everything (batch processing)
channel.queue_bind(exchange='sensor-data', queue='analytics',
                   routing_key='#')

1248.4.4 Task 2: Calculate Message Throughput

Data rates: - 100 sensors × 10 seconds = 10 messages/second from sensors - 20 machines × 10 seconds = 2 messages/second from machines - Total: 12 messages/second

Queue ingress rates:

Queue	Pattern	Rate	Calculation
Dashboard	#	12 msg/s	All messages
Maintenance	sensor.temp.# + sensor.vibr.#	~3.3 msg/s	33% of sensors × 10 msg/s
Analytics	#	12 msg/s	All messages

Storage requirements (Analytics queue for 24 hours): - Messages per day: 12 msg/s × 86,400 s = 1,036,800 messages - Average message size: 200 bytes (sensor ID, timestamp, value, metadata) - Storage: 1,036,800 × 200 bytes ≈ 200 MB/day

Bandwidth: - Outbound: 12 msg/s × 200 bytes = 2.4 KB/s = 19.2 kbps - Very modest bandwidth requirements ✓

1248.4.5 Task 3: Design Failure Handling

What if Analytics consumer is offline for maintenance?

Without AMQP (direct streaming): - Messages lost - Must re-query sensors (if possible) - Data gaps in analysis

With AMQP (durable queue): 1. Messages accumulate in Analytics queue 2. Queue persisted to disk (survives broker restart) 3. When Analytics consumer returns, processes backlog 4. No data loss ✓

Queue size management configuration:

channel.queue_declare(
    queue='analytics',
    durable=True,
    arguments={
        'x-max-length': 100000,           # Max 100K messages
        'x-overflow': 'reject-publish',    # Reject when full
        'x-dead-letter-exchange': 'dlx'    # Route rejected to DLX
    }
)

Monitoring alerts: - Queue depth > 50,000: Warning (consumer lagging) - Queue depth > 80,000: Critical (near capacity) - Consumer disconnected > 1 hour: Alert operations

1248.4.6 Task 4: Compare with MQTT

If we used MQTT instead:

Aspect	AMQP	MQTT
Protocol overhead	8-20 bytes/message	2 bytes/message
Offline buffering	Server-side (durable queues)	Application must implement
Routing complexity	Topic exchange, headers, etc.	Flat topic hierarchy only
Dead letter queues	Built-in	Not available
Memory footprint	100-500 KB	10-50 KB

Verdict: AMQP is better for this enterprise integration scenario: - Complex routing requirements (multiple consumers with different filters) - Offline consumer support (Analytics maintenance windows) - Enterprise backend integration (not constrained devices)

MQTT would be better if: - Sensors were battery-powered - Network bandwidth was severely limited - Simple pub/sub was sufficient

1248.5 Summary

This chapter covered practical AMQP routing patterns through hands-on exercises:

• Topic Exchange Wildcards: * matches exactly one word, # matches zero or more - critical for correct message routing
• Offline Consumer Support: Durable queues with persistent messages buffer messages during maintenance windows
• Delivery Guarantees: At-most-once, at-least-once, and exactly-once semantics with implementation patterns
• Throughput Calculations: Estimating message rates, storage requirements, and bandwidth for capacity planning
• Failure Handling: Designing resilient systems that recover from consumer outages without data loss
• AMQP vs MQTT: Choosing the right protocol based on routing complexity, reliability requirements, and device constraints

1248.6 What’s Next

Continue to AMQP Production Implementation to learn production configuration best practices, or return to the AMQP Implementations Overview for the complete implementation guide.