47  Pub/Sub and Topic Routing

In 60 Seconds

Publish-subscribe decouples producers from consumers through a broker that handles all routing. Topic hierarchies use “/” separators (e.g., sensors/temperature/room1) with wildcards: “+” matches one level, “#” matches all remaining levels. QoS 0 is fire-and-forget; QoS 1 guarantees at-least-once delivery; QoS 2 provides exactly-once at highest overhead.

Key Concepts
  • Publish-Subscribe (Pub/Sub): Messaging pattern where publishers send messages to topics without knowledge of subscribers; brokers route messages to all active subscribers of each topic
  • Message Broker: Central infrastructure component (Mosquitto, RabbitMQ, Kafka) routing messages from publishers to subscribers based on topic matching
  • Topic Hierarchy: MQTT topic tree structure using / separators (e.g., building/floor2/sensor/temperature) enabling flexible wildcard subscriptions
  • Wildcard Subscription: MQTT + (single level) and # (multi-level) wildcards enabling subscribers to receive messages from multiple topics with one subscription
  • Routing Table: Broker data structure mapping active topics to their subscriber lists, updated dynamically as clients subscribe and unsubscribe
  • Quality of Service (QoS): MQTT delivery guarantee levels — QoS 0 (fire-and-forget), QoS 1 (at least once), QoS 2 (exactly once) — traded against throughput
  • Topic Fan-Out: Broker behavior delivering one published message to N subscribers simultaneously; a single message to a topic with 1000 subscribers creates 1000 deliveries
  • Dead Letter Queue: Storage for messages that cannot be delivered to any subscriber, enabling recovery and debugging of routing failures
Minimum Viable Understanding
  • Pub/sub decouples producers from consumers: devices publish to topics and subscribe to topics of interest, with a broker handling all routing – publishers never need to know who is listening.
  • Topic hierarchies use / separators (e.g., sensors/temperature/room1) with wildcards: + matches one level, # matches all remaining levels.
  • QoS levels trade reliability for overhead: QoS 0 (fire-and-forget), QoS 1 (at-least-once, may duplicate), QoS 2 (exactly-once, highest overhead).

47.1 Learning Objectives

By the end of this chapter, you will be able to:

  • Construct topic-based publish-subscribe messaging pipelines for IoT sensor data
  • Differentiate MQTT-style + and # wildcards and apply them to filter hierarchical topics
  • Evaluate QoS 0, 1, and 2 trade-offs and select appropriate levels for given reliability requirements
  • Implement subscriber registration, routing, and per-subscriber inbox management
  • Design retained message strategies that deliver last-known values to new subscribers

Pub/sub is like a magical bulletin board where you only see the notices you care about!

Sammy the Sensor had a problem. Every time he measured the temperature, he had to run around and tell EVERY device individually. “Hey Lila, it’s 25 degrees! Hey Max, it’s 25 degrees! Hey Bella, it’s 25 degrees!” He was exhausted!

Then the Sensor Squad discovered the Magic Bulletin Board (that’s the broker!). Now Sammy just pins his temperature reading to the board under “sensors/temperature/room1”. Lila the LED only watches the “alerts” section. Max the Microcontroller watches “sensors/temperature” for ALL rooms. Bella the Battery watches everything with the magic # symbol!

The best part? When new friend Danny the Display joined the squad, he just started watching the bulletin board too – Sammy didn’t have to change anything! That is the power of decoupling – the sender and receiver don’t need to know about each other.

47.2 Introduction

The publish-subscribe (pub/sub) pattern is the foundation of scalable IoT messaging. Instead of devices communicating directly, they publish to topics and subscribe to topics of interest. A central broker handles routing, enabling loose coupling between producers and consumers.

Publish-subscribe messaging pattern showing publishers sending messages to a central broker, which routes them to subscribers based on topic matching

Publish-subscribe messaging pattern with broker routing messages from publishers to subscribers via topic matching
Why Pub/Sub for IoT?
  1. Decoupling: Publishers don’t need to know who is listening
  2. Scalability: Add new subscribers without changing publishers
  3. Filtering: Subscribers receive only relevant messages
  4. Reliability: Broker can buffer during disconnections

47.3 Topic Hierarchies

IoT topics follow a hierarchical structure using / as a separator. This creates a tree that enables efficient filtering:

sensors/
├── temperature/
│   ├── room1     <- sensors/temperature/room1
│   ├── room2     <- sensors/temperature/room2
│   └── lobby     <- sensors/temperature/lobby
├── humidity/
│   └── room1     <- sensors/humidity/room1
└── pressure/
    └── outdoor   <- sensors/pressure/outdoor

alerts/
├── motion/
│   └── entrance  <- alerts/motion/entrance
└── fire/
    └── building1 <- alerts/fire/building1

47.3.1 Topic Design Best Practices

Pattern Example Use Case
{type}/{subtype}/{location} sensors/temperature/room1 Sensor telemetry
{device}/{action} gateway01/command Device commands
{building}/{floor}/{room} hq/3/conference Location-based
{version}/{type}/{id} v2/sensors/abc123 API versioning
Topic Design Anti-Patterns

Avoid these common mistakes:

  • Too flat: sensor_temp_room1 - loses hierarchical filtering
  • Too deep: building/floor/wing/room/wall/sensor/type - hard to manage
  • Sensitive data in topic: user/john.doe/password - topics may be logged
  • Leading/trailing slashes: /sensors/ - creates empty segments

47.4 Topic Matching with Wildcards

MQTT-style wildcards enable flexible subscriptions:

47.4.1 Single-Level Wildcard (+)

Matches exactly one level in the topic hierarchy:

// sensors/+/room1 matches:
//   sensors/temperature/room1 ✓
//   sensors/humidity/room1    ✓
//   sensors/temperature/room2 ✗ (different location)
//   sensors/a/b/room1         ✗ (two levels, not one)

47.4.2 Multi-Level Wildcard (#)

Matches any remaining levels (must be last character):

// sensors/# matches:
//   sensors/temperature         ✓
//   sensors/temperature/room1   ✓
//   sensors/humidity/room1/raw  ✓
//   alerts/motion               ✗ (doesn't start with sensors/)

47.4.3 Topic Matching Implementation

/**
 * Match a topic against a subscription filter
 * Supports MQTT wildcards:
 *   + matches exactly one level (sensors/+/temp matches sensors/room1/temp)
 *   # matches any remaining levels (sensors/# matches sensors/room1/temp/celsius)
 */
bool matchTopic(const char* topic, const char* filter) {
  const char* t = topic;
  const char* f = filter;

  while (*t && *f) {
    // Multi-level wildcard matches everything remaining
    if (*f == '#') {
      return true;
    }

    // Single-level wildcard
    if (*f == '+') {
      // Skip to next separator in topic
      while (*t && *t != '/') t++;
      f++;
      // Skip separator in filter if present
      if (*f == '/') f++;
      // Skip separator in topic if present
      if (*t == '/') t++;
      continue;
    }

    // Exact character match
    if (*t != *f) {
      return false;
    }

    t++;
    f++;
  }

  // Both should be at end for exact match
  // Or filter ends with # for wildcard match
  return (*t == '\0' && *f == '\0') || (*f == '#');
}

47.4.4 Topic Matching Examples

// Demonstration of topic matching
const char* topics[] = {
  "sensors/temperature/room1",
  "sensors/humidity/room1",
  "alerts/motion/entrance",
  "system/status/gateway"
};

const char* filters[] = {"sensors/#", "sensors/temperature/+", "alerts/#"};

// Results:
// sensors/temperature/room1 vs sensors/#              -> MATCH
// sensors/temperature/room1 vs sensors/temperature/+  -> MATCH
// sensors/temperature/room1 vs alerts/#               -> no match

// sensors/humidity/room1    vs sensors/#              -> MATCH
// sensors/humidity/room1    vs sensors/temperature/+  -> no match
// sensors/humidity/room1    vs alerts/#               -> no match

// alerts/motion/entrance    vs sensors/#              -> no match
// alerts/motion/entrance    vs sensors/temperature/+  -> no match
// alerts/motion/entrance    vs alerts/#               -> MATCH
Try It: Topic Wildcard Matcher

Test MQTT-style topic matching interactively. Enter a topic and a subscription filter to see whether they match — using the same + (single-level) and # (multi-level) wildcard rules implemented in the matchTopic() function above.

47.5 Subscriber Management

47.5.1 Subscriber Data Structure

Each subscriber has an identity, topic filter, QoS preference, and personal inbox:

// Subscriber information
struct Subscriber {
  char clientId[32];            // Unique client identifier
  char topicFilter[MAX_TOPIC_LENGTH]; // Subscription topic (may include wildcards)
  QoSLevel maxQos;              // Maximum QoS this subscriber accepts
  bool active;                  // Is subscription active
  MessageQueue inbox;           // Personal message queue
  uint32_t lastActivity;        // Timestamp of last message delivery
};

// Publisher information
struct Publisher {
  char clientId[32];            // Unique client identifier
  char sensorType[32];          // Type of data published
  uint32_t messagesSent;        // Statistics
  uint32_t lastPublish;         // Last publish timestamp
  bool active;                  // Is publisher active
};

47.5.2 Register Subscriber

/**
 * Register a new subscriber
 */
int registerSubscriber(const char* clientId, const char* topicFilter, QoSLevel maxQos) {
  if (subscriberCount >= MAX_SUBSCRIBERS) {
    Serial.printf("[BROKER] ERROR: Max subscribers reached\n");
    return -1;
  }

  Subscriber* sub = &subscribers[subscriberCount];
  strncpy(sub->clientId, clientId, 31);
  strncpy(sub->topicFilter, topicFilter, MAX_TOPIC_LENGTH - 1);
  sub->maxQos = maxQos;
  sub->active = true;
  sub->lastActivity = millis();
  initQueue(&sub->inbox, 10);  // Each subscriber gets a 10-message inbox

  Serial.printf("[BROKER] Subscriber registered: %s -> %s (QoS %d)\n",
                clientId, topicFilter, maxQos);

  // Deliver retained messages matching this subscription
  for (int i = 0; i < MAX_RETAINED_MESSAGES; i++) {
    if (retainedMessages[i].valid &&
        matchTopic(retainedMessages[i].topic, topicFilter)) {
      Serial.printf("[BROKER] Delivering retained message to new subscriber: %s\n",
                    retainedMessages[i].topic);
      enqueue(&sub->inbox, &retainedMessages[i].message);
    }
  }

  return subscriberCount++;
}

47.5.3 Publish to Subscribers

The core routing function matches topics and delivers to all matching subscribers:

/**
 * Publish a message to all matching subscribers
 * Core routing function of the broker
 */
int publishMessage(Message* msg) {
  int deliveryCount = 0;

  // Store retained message if flagged
  if (msg->retained) {
    storeRetainedMessage(msg);
  }

  // Route to all matching subscribers
  for (int i = 0; i < subscriberCount; i++) {
    if (!subscribers[i].active) continue;

    if (matchTopic(msg->topic, subscribers[i].topicFilter)) {
      totalTopicMatches++;

      // Downgrade QoS to subscriber's maximum
      QoSLevel effectiveQos = min(msg->qos, subscribers[i].maxQos);
      Message deliveryMsg = *msg;
      deliveryMsg.qos = effectiveQos;

      // Enqueue to subscriber's inbox
      if (enqueue(&subscribers[i].inbox, &deliveryMsg)) {
        deliveryCount++;
        subscribers[i].lastActivity = millis();
      }
    }
  }

  totalMessagesRouted++;
  return deliveryCount;
}

47.6 QoS Levels Explained

Quality of Service (QoS) defines delivery guarantees. Higher QoS means more reliability but also more overhead.

Comparison of MQTT QoS levels showing QoS 0 fire-and-forget with one packet, QoS 1 at-least-once with PUBACK acknowledgment, and QoS 2 exactly-once with four-step PUBREC PUBREL PUBCOMP handshake

QoS level comparison showing the packet exchange sequences for QoS 0, QoS 1, and QoS 2 delivery guarantees

47.6.1 QoS 0: At-Most-Once

Fire and forget - fastest but no guarantee:

/**
 * Simulate QoS 0: At-most-once delivery
 * Message is sent once with no acknowledgment
 */
void handleQoS0(Message* msg, Subscriber* sub) {
  Serial.printf("[QoS 0] Fire-and-forget delivery to %s\n", sub->clientId);
  // Message is already in queue, no further action needed
  msg->acknowledged = true;
}

47.6.2 QoS 1: At-Least-Once

Acknowledged delivery - reliable but may duplicate:

/**
 * Simulate QoS 1: At-least-once delivery
 * Message is resent until acknowledged (may cause duplicates)
 */
void handleQoS1(Message* msg, Subscriber* sub) {
  Serial.printf("[QoS 1] At-least-once delivery to %s\n", sub->clientId);

  // Simulate acknowledgment with random success
  bool ackReceived = (random(100) > 20);  // 80% success rate

  if (ackReceived) {
    msg->acknowledged = true;
    Serial.printf("[QoS 1] PUBACK received for message %lu\n", msg->messageId);
  } else {
    msg->deliveryCount++;
    qosRetries++;
    Serial.printf("[QoS 1] No PUBACK - will retry (attempt %d)\n", msg->deliveryCount);
  }
}

47.6.3 QoS 2: Exactly-Once

Four-step handshake - guaranteed single delivery but highest overhead:

/**
 * Simulate QoS 2: Exactly-once delivery
 * Four-step handshake ensures no duplicates
 */
void handleQoS2(Message* msg, Subscriber* sub) {
  Serial.printf("[QoS 2] Exactly-once delivery to %s\n", sub->clientId);

  // Simulate 4-step handshake: PUBLISH -> PUBREC -> PUBREL -> PUBCOMP
  Serial.printf("[QoS 2] Step 1: PUBLISH sent (ID=%lu)\n", msg->messageId);

  bool pubrecReceived = (random(100) > 10);  // 90% success
  if (!pubrecReceived) {
    Serial.printf("[QoS 2] PUBREC not received - retrying\n");
    qosRetries++;
    return;
  }
  Serial.printf("[QoS 2] Step 2: PUBREC received\n");

  Serial.printf("[QoS 2] Step 3: PUBREL sent\n");

  bool pubcompReceived = (random(100) > 10);  // 90% success
  if (!pubcompReceived) {
    Serial.printf("[QoS 2] PUBCOMP not received - retrying PUBREL\n");
    qosRetries++;
    return;
  }
  Serial.printf("[QoS 2] Step 4: PUBCOMP received - transaction complete\n");

  msg->acknowledged = true;
  exactlyOnceTransactions++;
}

47.6.4 When to Use Each QoS Level

QoS Use Case Example Trade-off
0 High-frequency telemetry Temperature every second Fast, may lose some
1 Important sensor data Hourly energy readings Reliable, may duplicate
2 Critical transactions Firmware updates, billing Guaranteed, slower
  • QoS 0 is like dropping a postcard in a mailbox - quick but no confirmation
  • QoS 1 is like certified mail - you get a delivery confirmation, but sometimes duplicates arrive
  • QoS 2 is like registered mail with signature - guaranteed exactly once but requires multiple steps
Try It: QoS Overhead Comparator

Compare the packet overhead and delivery guarantees of QoS 0, 1, and 2 for a given message rate. See how reliability costs scale with volume.

47.7 Retained Messages

Retained messages are stored by the broker and delivered to new subscribers. This ensures they receive the last known value immediately:

// Retained message storage
struct RetainedMessage {
  char topic[MAX_TOPIC_LENGTH];
  Message message;
  bool valid;
};

RetainedMessage retainedMessages[MAX_RETAINED_MESSAGES];

/**
 * Store a retained message
 */
void storeRetainedMessage(Message* msg) {
  // Find existing slot for this topic or empty slot
  int slot = -1;
  for (int i = 0; i < MAX_RETAINED_MESSAGES; i++) {
    if (!retainedMessages[i].valid) {
      if (slot < 0) slot = i;
    } else if (strcmp(retainedMessages[i].topic, msg->topic) == 0) {
      slot = i;
      break;
    }
  }

  if (slot >= 0) {
    strncpy(retainedMessages[slot].topic, msg->topic, MAX_TOPIC_LENGTH - 1);
    memcpy(&retainedMessages[slot].message, msg, sizeof(Message));
    retainedMessages[slot].valid = true;
    Serial.printf("[BROKER] Retained message stored: %s\n", msg->topic);
  }
}

47.7.1 Retained Message Use Cases

Topic Retained Value Purpose
sensors/temperature/room1 Latest reading New dashboard shows current temp
device/gateway01/status Online/offline New monitoring app sees device state
config/firmware/version Current version New device checks for updates

47.8 Message Persistence

During network outages, messages must be persisted locally and replayed when connectivity returns:

/**
 * Persist messages during network outage
 * In real systems, this would write to flash/EEPROM
 */
void persistMessage(Message* msg) {
  Serial.printf("[PERSIST] Storing message %lu for offline delivery\n", msg->messageId);
  Serial.printf("[PERSIST] Topic: %s, QoS: %d, Payload: %.50s...\n",
                msg->topic, msg->qos, msg->payload);
}

/**
 * Replay persisted messages when network returns
 */
void replayPersistedMessages() {
  Serial.printf("[PERSIST] Network restored - replaying buffered messages\n");

  // In real implementation, would read from flash storage
  for (int i = 0; i < subscriberCount; i++) {
    int pendingCount = getQueueSize(&subscribers[i].inbox);
    if (pendingCount > 0) {
      Serial.printf("[PERSIST] Subscriber %s has %d pending messages\n",
                    subscribers[i].clientId, pendingCount);
    }
  }
}

47.9 Worked Example: MQTT Topic Hierarchy Design for a 50-Building Smart Campus

Worked Example: Designing a Scalable Topic Hierarchy for 15,000 Sensors

Scenario: A university campus has 50 buildings, 15,000 sensors (temperature, occupancy, energy meters, door locks), and 5 consuming applications (facilities dashboard, energy billing, security system, HVAC optimizer, research data lake). Design the MQTT topic hierarchy and calculate broker throughput requirements.

Step 1: Topic Hierarchy Design

campus/{building_id}/{floor}/{room}/{sensor_type}/{metric}

Examples: - campus/eng-hall/3/305/temperature/current – Room 305 temperature - campus/eng-hall/3/305/occupancy/count – Room 305 people count - campus/library/1/lobby/energy/kwh – Library lobby energy

Step 2: Subscription Patterns for Each Consumer

Consumer Subscription Pattern Matches Why This Pattern
Facilities dashboard campus/# All 15,000 sensors Needs full campus view
Energy billing campus/+/+/+/energy/# 500 energy meters Only energy data, any building/floor/room
Security system campus/+/+/+/door/# 1,000 door sensors All door events campus-wide
HVAC optimizer (Eng Hall only) campus/eng-hall/+/+/temperature/# 120 temperature sensors One building’s temps for local optimization
Research data lake campus/eng-hall/3/+/+/+ 45 sensors on floor 3 Research lab floor, all sensor types

Step 3: QoS Selection Per Data Flow

Data Flow QoS Justification Overhead
Temperature readings QoS 0 (fire-and-forget) Readings arrive every 60 sec. Missing one is OK – next one comes in 60 sec. 0 extra packets
Door lock/unlock events QoS 1 (at-least-once) Security audit requires no missed events. Duplicates are OK (idempotent: door is either locked or unlocked). +1 PUBACK per message
Energy meter billing reads QoS 2 (exactly-once) Billing cannot duplicate or miss a kWh reading – duplicates = overcharging, misses = undercharging. +3 extra packets per message
Fire alarm QoS 1 + retained Must not miss. Retained ensures late-joining dashboard sees active alarm. +1 PUBACK + retained storage

Step 4: Broker Throughput Sizing

Sensor Type Count Msg/Min QoS Messages/Min Total Fan-Out (avg subscribers) Deliveries/Min
Temperature 8,000 1 QoS 0 8,000 2.1 (dashboard + HVAC) 16,800
Occupancy 3,000 0.5 QoS 0 1,500 1.5 2,250
Energy 500 0.2 QoS 2 100 2.0 200 (but 4x packet overhead)
Door 1,000 0.1 QoS 1 100 2.0 200 (but 2x overhead)
Other 2,500 0.5 QoS 0 1,250 1.2 1,500
Total 15,000 10,950 msg/min ~21,750 deliveries/min

Broker requirement: ~363 deliveries/second average, ~1,000/sec peak (morning arrival). A single Mosquitto instance on a 2-core VM handles 10,000 msg/sec – this campus needs only 10% of a single broker’s capacity.

Result: The hierarchical topic design enables each consumer to subscribe with a single wildcard pattern instead of 15,000 individual subscriptions. QoS is differentiated by data criticality (not applied uniformly), saving 3x packet overhead on the 95% of traffic that tolerates occasional loss. Total broker cost: ~$50/month for a single cloud VM.

Broker throughput scales with sensor count and subscriber fan-out. For \(N_{\text{sensors}}\) sensors publishing at rate \(r\) (msg/sec) to \(M\) subscribers, total deliveries per second:

\[ D = N_{\text{sensors}} \times r \times \text{avg\_subscribers\_per\_topic} \]

Campus example: \(N = 15{,}000\), \(r = \frac{1}{60}\) (1 msg/min average), average fan-out = 1.45 subscribers/topic (facilities gets all, others are selective) yields \(D = 15{,}000 \times \frac{1}{60} \times 1.45 = 362.5\) deliveries/sec. With 20% QoS 1 overhead (PUBACK packets), broker handles \(362.5 \times 1.2 = 435\) packets/sec – trivial for modern brokers (10K+ msg/sec capacity).

Try It: Broker Throughput Calculator

Estimate MQTT broker throughput requirements for your IoT deployment — based on the sizing methodology from the worked example above. See if a single broker can handle your load.

47.10 Summary

This chapter covered the publish-subscribe pattern and topic routing for IoT messaging:

  • Topic Hierarchies: Slash-separated paths for efficient filtering
  • Wildcard Matching: + for single level, # for multi-level matching
  • Subscriber Management: Registration, routing, and per-subscriber inboxes
  • QoS Levels: Trade-offs between speed (QoS 0), reliability (QoS 1), and exactness (QoS 2)
  • Retained Messages: Last-known values for new subscribers
  • Persistence: Buffering during network outages

Key Takeaways:

  • Topic hierarchies enable selective subscriptions at scale
  • Wildcards reduce the need for multiple subscriptions
  • QoS selection is a trade-off between reliability and overhead
  • Retained messages ensure new subscribers get current state

47.11 Knowledge Check

Common Pitfalls

Publishing all sensor data to a single topic (e.g., sensors) forces all consumers to receive and filter all messages rather than subscribing to specific subtopics. Use hierarchical topics (building/floor/room/sensor/metric) from day one to enable selective subscription and support future growth without refactoring.

Using # wildcard subscriptions in production connects a client to every topic on the broker — including high-volume system topics. At scale, a single # subscriber might receive millions of messages/second it doesn’t need. Always subscribe to the narrowest topic that satisfies your requirements.

MQTT retained messages persist on the broker and are immediately delivered to new subscribers. If a sensor publishes a retained alarm message and the alarm condition resolves, subscribers connecting later will immediately receive the stale alarm. Explicitly clear retained messages when their content becomes obsolete.

Pub/Sub systems do not guarantee message delivery order across topics or across multiple publishers to the same topic. A temperature and a humidity reading published 1ms apart may arrive at subscribers in either order. Design consumers to handle out-of-order messages using timestamps, not arrival order.

47.13 What’s Next

If you want to… Read this
Understand message queue fundamentals Message Queue Fundamentals
Learn about protocol bridging concepts Protocol Bridging Fundamentals
See real-world gateway examples Real-World Gateway Examples
Practice with message queue lab Message Queue Lab

Continue to Queue Lab Challenges to apply these concepts in hands-on exercises including dead letter queues, message deduplication, and flow control.