1188 MQTT Publish-Subscribe Basics

1188.1 Learning Objectives

By the end of this chapter, you will be able to:

Understand the Publish-Subscribe Pattern: Grasp how MQTT decouples message producers from consumers
Design Topic Hierarchies: Create efficient, scalable topic naming conventions
Master Wildcards: Use + and # wildcards for flexible subscriptions
Experiment with MQTT: Use interactive simulators to test publish-subscribe messaging

1188.2 Prerequisites

Before diving into this chapter, you should be familiar with:

Basic Networking Concepts: Understanding TCP/IP, client-server architecture, and network protocols
IoT Reference Architectures: Familiarity with IoT system layers and components

The Challenge: Scalable Device Communication

The Problem: Traditional request-response patterns (like HTTP polling) don’t scale for IoT. With millions of devices, each polling a server creates:

N devices x M polls/hour = billions of wasted requests - Most polls return “no new data”
Servers must track every device connection - Memory and connection limits quickly exceeded
Devices waste energy checking for messages that aren’t there - Critical for battery-powered sensors

The Solution: Publish-Subscribe messaging with a central broker. MQTT was designed from the ground up for exactly these IoT constraints, providing lightweight, reliable, and scalable communication.

Key Takeaway

MQTT is the publish-subscribe protocol that lets IoT devices communicate through a central broker instead of directly with each other. By decoupling senders (publishers) from receivers (subscribers), MQTT enables millions of devices to exchange messages efficiently.

1188.3 What is MQTT?

What is MQTT? (Simple Explanation)

Analogy: MQTT is like a newspaper subscription service.

Instead of checking websites for updates constantly, you subscribe to topics you care about, and news gets delivered to you automatically.

TRADITIONAL vs MQTT:

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#E67E22', 'secondaryColor': '#16A085', 'tertiaryColor': '#E67E22', 'noteTextColor': '#2C3E50', 'noteBkgColor': '#FFF9E6', 'noteBorderColor': '#E67E22'}}}%%
graph LR
    subgraph "Traditional HTTP Polling"
        Client1["Client"] -->|"1. Request: Get temp?"| Server1["Server"]
        Server1 -->|"2. Response: 24C"| Client1
        Client1 -->|"3. Request again?"| Server1
        Server1 -->|"4. Response: 24C"| Client1
    end

    subgraph "MQTT Publish-Subscribe"
        Sensor["Sensor"] -->|"Publish: temp=24C"| Broker["Broker"]
        Broker -->|"Auto-deliver"| App["App"]
        Broker -->|"Auto-deliver"| Cloud["Cloud"]
    end

    style Client1 fill:#2C3E50,color:#fff
    style Server1 fill:#7F8C8D,color:#fff
    style Sensor fill:#16A085,color:#fff
    style Broker fill:#E67E22,color:#fff
    style App fill:#2C3E50,color:#fff
    style Cloud fill:#2C3E50,color:#fff

Figure 1188.1: Architecture comparison: HTTP polling versus MQTT publish-subscribe

Definition

MQTT (Message Queue Telemetry Transport) is a lightweight, publish-subscribe messaging protocol designed for constrained devices and low-bandwidth, high-latency networks. It’s the de facto standard for IoT communication.

Key Characteristics:

Lightweight: Minimal packet overhead (2 bytes minimum)
Publish-Subscribe Pattern: Decouples message producers from consumers
Quality of Service: Three levels (QoS 0, 1, 2) for reliability
Retained Messages: Latest message stored for new subscribers
Last Will and Testament: Notifies when clients disconnect unexpectedly

1188.4 The Three Key Players

MQTT has three main components:

Component	Role	Analogy
Publisher	Sends messages	Newspaper writer
Broker	Routes messages	Post office
Subscriber	Receives messages	Newspaper subscriber

HOW MQTT WORKS:

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#16A085', 'primaryTextColor': '#fff', 'primaryBorderColor': '#2C3E50', 'lineColor': '#E67E22', 'secondaryColor': '#2C3E50', 'tertiaryColor': '#E67E22'}}}%%
sequenceDiagram
    participant P as Publisher<br/>(Sensor)
    participant B as Broker<br/>(Post Office)
    participant S as Subscriber<br/>(Dashboard)

    Note over P,S: Step 1: Subscribe
    S->>B: SUBSCRIBE "temperature"
    B->>S: SUBACK

    Note over P,S: Step 2: Publish
    P->>B: PUBLISH "temperature" = "24C"

    Note over P,S: Step 3: Deliver
    B->>S: Deliver "temperature" = "24C"

    Note over P,S: Step 4: New Update
    P->>B: PUBLISH "temperature" = "25C"
    B->>S: Deliver "temperature" = "25C"

Figure 1188.2: Sequence diagram illustrating MQTT communication flow

Key Insight: The Broker Decouples Publishers from Subscribers

flowchart LR
    P1["Sensor 1"] --> B["Broker"]
    P2["Sensor 2"] --> B
    P3["Sensor 3"] --> B
    B --> S1["Dashboard"]
    B --> S2["Database"]
    B --> S3["Alert System"]

    style B fill:#E67E22,color:#fff
    style P1 fill:#16A085,color:#fff
    style P2 fill:#16A085,color:#fff
    style P3 fill:#16A085,color:#fff
    style S1 fill:#2C3E50,color:#fff
    style S2 fill:#2C3E50,color:#fff
    style S3 fill:#2C3E50,color:#fff

Figure 1188.3: MQTT broker routing sensor messages to multiple consumers

In Plain English: Publishers don’t know (or care) who receives their messages. Subscribers don’t know who sends. The broker handles all the routing. This means you can add new sensors or new consumers without changing any existing code.

Minimum Viable Understanding: Publish-Subscribe Pattern

Core Concept: Publishers send messages to named topics without knowing who receives them; subscribers listen to topics without knowing who sends, with a broker routing all messages in between. Why It Matters: This decoupling enables IoT systems to scale from 10 to 10 million devices without changing code - add new sensors or dashboards anytime without modifying existing components. Key Takeaway: Design your topic hierarchy first (e.g., building/floor/room/sensor_type), then build publishers and subscribers independently - the broker handles all discovery and routing.

Show code

{
  const container = document.getElementById('kc-mqtt-pubsub-1');
  if (container && typeof InlineKnowledgeCheck !== 'undefined') {
    container.innerHTML = '';
    container.appendChild(InlineKnowledgeCheck.create({
      question: "A factory has 50 temperature sensors that need to send data to 3 different systems: a real-time dashboard, a database for logging, and an alerting service. Using traditional HTTP, how would you architect this?",
      options: [
        {text: "Each sensor makes 3 HTTP requests per reading (one to each system)", correct: false, feedback: "This works but is inefficient - 50 sensors x 3 requests = 150 HTTP calls per reading cycle, and sensors must know all endpoint addresses."},
        {text: "Sensors publish to an MQTT broker, and all 3 systems subscribe to the same topic", correct: true, feedback: "Correct! With MQTT pub-sub, sensors publish once, and the broker delivers to all subscribers. Sensors don't need to know about the systems, and adding a 4th system requires zero sensor changes."},
        {text: "Create a central HTTP server that fans out to the 3 systems", correct: false, feedback: "This adds a single point of failure and still requires custom fan-out logic. MQTT's broker already provides this routing built-in."},
        {text: "Use WebSockets for real-time streaming to all systems", correct: false, feedback: "WebSockets require maintaining 50 x 3 = 150 persistent connections. MQTT is designed specifically for this IoT pattern with much lower overhead."}
      ],
      difficulty: "medium",
      topic: "mqtt-pubsub"
    }));
  }
}

1188.5 Topics: Organizing Messages

MQTT uses topics to organize messages - like folders for emails:

Topic	What It Means	Example Message
`home/living/temp`	Living room temperature	`"25.5"`
`home/bedroom/light`	Bedroom light status	`"on"`
`factory/machine1/status`	Machine 1 status	`"running"`

Wildcards make subscribing powerful:

home/+/temp = “All room temperatures” (+ = any one level)
home/# = “Everything in home” (# = all levels below)

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#E67E22', 'secondaryColor': '#16A085', 'tertiaryColor': '#E67E22'}}}%%
graph TB
    subgraph "Topic Hierarchy"
        Root["home/"] --> Living["living/"]
        Root --> Bedroom["bedroom/"]
        Root --> Kitchen["kitchen/"]
        Living --> LTemp["temp<br/>24C"]
        Living --> LHumid["humidity<br/>45%"]
        Bedroom --> BTemp["temp<br/>22C"]
        Kitchen --> KTemp["temp<br/>26C"]
    end

    subgraph "Subscriptions"
        Sub1["home/+/temp<br/>Gets: ALL room temps"] -.->|matches| LTemp
        Sub1 -.->|matches| BTemp
        Sub1 -.->|matches| KTemp
        Sub2["home/#<br/>Gets: EVERYTHING"] -.->|matches all| Root
    end

    style Root fill:#E67E22,color:#fff
    style Sub1 fill:#16A085,color:#fff
    style Sub2 fill:#2C3E50,color:#fff

Figure 1188.4: Hierarchical topic structure showing wildcard matching

1188.5.1 Wildcard Reference

Wildcard	Meaning	Example	Matches
`+`	Single level	`home/+/temperature`	`home/bedroom/temperature`, `home/kitchen/temperature`
`#`	Multiple levels	`home/#`	`home/bedroom/temperature`, `home/bedroom/humidity`, `home/kitchen/motion/sensor1`

Show code

{
  const container = document.getElementById('kc-mqtt-pubsub-2');
  if (container && typeof InlineKnowledgeCheck !== 'undefined') {
    container.innerHTML = '';
    container.appendChild(InlineKnowledgeCheck.create({
      question: "Your smart home system has sensors publishing to topics like home/bedroom/temp, home/kitchen/temp, home/bedroom/humidity, and home/kitchen/humidity. You want to subscribe to ALL temperature readings from ALL rooms. Which subscription pattern should you use?",
      options: [
        {text: "home/#/temp", correct: false, feedback: "Invalid! The # wildcard must be at the END of the topic filter. It cannot be followed by other levels."},
        {text: "home/+/temp", correct: true, feedback: "Correct! The + wildcard matches exactly ONE topic level. This matches home/bedroom/temp, home/kitchen/temp, home/living/temp, etc. - all rooms' temperature readings."},
        {text: "home/*/temp", correct: false, feedback: "The * character is NOT a valid MQTT wildcard. MQTT only uses + (single-level) and # (multi-level)."},
        {text: "home/temp/#", correct: false, feedback: "This would match topics starting with home/temp/, but your sensors publish to home/{room}/temp - the room comes before the measurement type."}
      ],
      difficulty: "medium",
      topic: "mqtt-wildcards"
    }));
  }
}

Pitfall: Using Leading Slashes in Topic Names

The Mistake: Developers create topics with leading slashes like /home/temperature or mix conventions, using /home/temp in publishers and home/temp in subscribers.

Why It Happens: Leading slashes look natural from filesystem experience (Unix paths start with /). MQTT does not strip or normalize slashes, so /home/temp and home/temp are completely different topics.

The Fix: Never use leading slashes. Adopt a consistent topic naming convention across your organization.

// WRONG - leading slash creates unexpected first level
mqttClient.publish("/home/temperature", "24.5");  // Topic has 3 levels: "", "home", "temperature"
mqttClient.subscribe("home/temperature");         // 2 levels: "home", "temperature"
// Result: Subscriber never receives message!

// CORRECT - consistent naming without leading slash
mqttClient.publish("home/temperature", "24.5");   // 2 levels: "home", "temperature"
mqttClient.subscribe("home/temperature");         // Matches!

1188.6 Interactive: MQTT Pub/Sub Simulator

Try this hands-on simulator to see MQTT messaging in action. Publish messages to topics and watch how the broker routes them to matching subscribers.

<!-- Publisher Section -->
<div style="flex: 1; min-width: 250px; background: #fff; padding: 15px; border-radius: 8px; box-shadow: 0 2px 4px rgba(0,0,0,0.1);">
  <h4 style="color: #16A085; margin-top: 0;">Publisher</h4>
  <label style="display: block; margin: 10px 0 5px; font-weight: bold; font-size: 0.9em;">Topic:</label>
  <input type="text" id="pub-topic" placeholder="e.g., home/living/temp" style="width: 100%; padding: 8px; border: 1px solid #ddd; border-radius: 4px; box-sizing: border-box;">

  <label style="display: block; margin: 10px 0 5px; font-weight: bold; font-size: 0.9em;">Message:</label>
  <input type="text" id="pub-message" placeholder="e.g., 24.5" style="width: 100%; padding: 8px; border: 1px solid #ddd; border-radius: 4px; box-sizing: border-box;">

  <button type="button" onclick="publishMessage()" style="width: 100%; margin-top: 15px; padding: 10px; background: #16A085; color: white; border: none; border-radius: 4px; cursor: pointer; font-weight: bold; font-size: 1em;">Publish Message</button>

  <div style="margin-top: 15px; padding: 10px; background: #e8f5e9; border-left: 4px solid #16A085; border-radius: 4px; font-size: 0.85em;">
    <strong>Quick Examples:</strong><br>
    <code style="cursor: pointer; color: #16A085;" onclick="document.getElementById('pub-topic').value='home/living/temp'; document.getElementById('pub-message').value='24.5';">home/living/temp</code><br>
    <code style="cursor: pointer; color: #16A085;" onclick="document.getElementById('pub-topic').value='home/bedroom/light'; document.getElementById('pub-message').value='on';">home/bedroom/light</code>
  </div>
</div>

<!-- Broker Section -->
<div style="flex: 0.8; min-width: 200px; text-align: center;">
  <h4 style="color: #E67E22; margin-top: 0;">MQTT Broker</h4>
  <div id="broker-status" style="padding: 30px 20px; background: linear-gradient(135deg, #fff3cd 0%, #ffe8a1 100%); border-radius: 8px; box-shadow: 0 2px 4px rgba(0,0,0,0.1); font-size: 0.95em; min-height: 100px; display: flex; align-items: center; justify-content: center;">
    <div>
      <div style="font-weight: bold; color: #856404;">Status: Ready</div>
      <div style="margin-top: 5px; font-size: 0.85em; color: #666;">Waiting for messages...</div>
    </div>
  </div>
</div>

<!-- Subscribers Section -->
<div style="flex: 1; min-width: 250px; background: #fff; padding: 15px; border-radius: 8px; box-shadow: 0 2px 4px rgba(0,0,0,0.1);">
  <h4 style="color: #2C3E50; margin-top: 0;">Subscribers</h4>
  <div id="subscribers-list" style="margin-bottom: 15px; max-height: 200px; overflow-y: auto;"></div>

  <label style="display: block; margin: 10px 0 5px; font-weight: bold; font-size: 0.9em;">Subscribe to Topic:</label>
  <input type="text" id="sub-topic" placeholder="e.g., home/+/temp or home/#" style="width: 100%; padding: 8px; border: 1px solid #ddd; border-radius: 4px; box-sizing: border-box;">

  <button type="button" onclick="addSubscriber()" style="width: 100%; margin-top: 10px; padding: 10px; background: #2C3E50; color: white; border: none; border-radius: 4px; cursor: pointer; font-weight: bold; font-size: 1em;">+ Add Subscriber</button>

  <div style="margin-top: 15px; padding: 10px; background: #e3f2fd; border-left: 4px solid #2C3E50; border-radius: 4px; font-size: 0.85em;">
    <strong>Wildcard Examples:</strong><br>
    <code style="cursor: pointer; color: #2C3E50;" onclick="document.getElementById('sub-topic').value='home/+/temp';">home/+/temp</code> (all room temps)<br>
    <code style="cursor: pointer; color: #2C3E50;" onclick="document.getElementById('sub-topic').value='home/#';">home/#</code> (everything in home)
  </div>
</div>

<h4 style="color: #333; margin-bottom: 10px;">Message Log</h4>
<div id="message-log" style="padding: 15px; background: #263238; color: #aed581; max-height: 200px; overflow-y: auto; border-radius: 8px; font-family: 'Courier New', monospace; font-size: 0.85em; line-height: 1.6;"></div>
<button type="button" onclick="clearLog()" style="margin-top: 10px; padding: 8px 15px; background: #666; color: white; border: none; border-radius: 4px; cursor: pointer; font-size: 0.9em;">Clear Log</button>

How to Use This Simulator:

Add Subscribers (already includes 2 examples):
- Enter a topic pattern (e.g., home/+/temp for all room temperatures)
- Click “Add Subscriber”
Publish Messages:
- Enter a specific topic (e.g., home/living/temp)
- Enter a message payload (e.g., 24.5)
- Click “Publish Message”
Watch the Flow: See the broker route messages to matching subscribers

1188.7 Retained Messages

Analogy: Retained messages are like leaving the last known value on a whiteboard.

Without retained messages:

10:00 AM - Sensor publishes: "Temperature: 24C"
10:30 AM - Phone app starts up and subscribes
           App sees nothing (missed the 10:00 message!)
10:35 AM - Sensor publishes again: "Temperature: 25C"
           App finally gets first reading

With retained messages:

10:00 AM - Sensor publishes with RETAIN flag: "Temperature: 24C"
           Broker saves this as "latest known value"
10:30 AM - Phone app starts up and subscribes
           Broker immediately sends: "Temperature: 24C" (retained)
           App shows data instantly!

When to use retained messages:

Device status (“online” / “offline”)
Configuration settings (“mode: heating”)
Last known sensor value (“temperature: 24C”)

When NOT to use retained messages:

Streaming data (don’t need every single reading)
Alerts/events (only care about new ones)

Show code

{
  const container = document.getElementById('kc-mqtt-pubsub-3');
  if (container && typeof InlineKnowledgeCheck !== 'undefined') {
    container.innerHTML = '';
    container.appendChild(InlineKnowledgeCheck.create({
      question: "A smart home dashboard app connects to an MQTT broker and subscribes to 'home/thermostat/temperature'. The thermostat published the last temperature (22.5C) 3 hours ago with retain=true. What does the dashboard see immediately after subscribing?",
      options: [
        {text: "Nothing - must wait for next thermostat publish", correct: false, feedback: "With retain=true, the broker stores the last message. New subscribers receive this retained message immediately upon subscribing."},
        {text: "22.5C - the retained message is delivered immediately", correct: true, feedback: "Correct! Retained messages are stored by the broker and delivered to new subscribers as soon as they subscribe."},
        {text: "An error - messages older than 1 hour are deleted", correct: false, feedback: "Retained messages have no automatic expiration in MQTT 3.1.1. They persist until explicitly cleared."},
        {text: "22.5C only if the dashboard was subscribed before", correct: false, feedback: "Retained messages are delivered to ANY subscriber, not just previously connected ones."}
      ],
      difficulty: "medium",
      topic: "mqtt-retained"
    }));
  }
}

1188.8 Last Will and Testament (LWT)

Analogy: LWT is like leaving a sealed letter that only gets opened if you disappear.

The Problem:

Sensor is publishing: "I'm alive! Temperature: 24C"
Network cable gets unplugged...
Sensor is offline, but subscribers don't know!
Dashboard still shows "24C" from 10 minutes ago

The Solution with LWT:

Sensor connects to broker and says:
"If I disconnect unexpectedly, publish this message for me:
  Topic: home/bedroom/status
  Message: OFFLINE"

Normal operation: Sensor publishes normally
Network failure happens...
Broker detects sensor is gone -> publishes LWT automatically
Dashboard receives: "home/bedroom/status: OFFLINE"

Real code:

// ESP32 setting up Last Will and Testament
mqttClient.setWill("home/sensor1/status", "OFFLINE");  // LWT message

mqttClient.connect();  // Connect to broker

// Periodically publish normal status
mqttClient.publish("home/sensor1/status", "ONLINE");

// If ESP32 crashes or loses Wi-Fi, broker automatically publishes:
// Topic: "home/sensor1/status"
// Message: "OFFLINE"

Show code

{
  const container = document.getElementById('kc-mqtt-pubsub-4');
  if (container && typeof InlineKnowledgeCheck !== 'undefined') {
    container.innerHTML = '';
    container.appendChild(InlineKnowledgeCheck.create({
      question: "An industrial IoT system monitors 100 pressure sensors in a chemical plant. The monitoring dashboard must know immediately when any sensor goes offline. How should you implement offline detection?",
      options: [
        {text: "Poll each sensor every 10 seconds with HTTP requests", correct: false, feedback: "Polling 100 sensors every 10 seconds = 600 requests/minute. This wastes bandwidth and doesn't scale."},
        {text: "Have sensors publish 'I am alive' messages every 5 seconds", correct: false, feedback: "This works but wastes bandwidth (1200 messages/minute just for health checks)."},
        {text: "Configure Last Will Testament (LWT) with 'offline' status for each sensor", correct: true, feedback: "Correct! LWT is triggered automatically by the broker when a client disconnects unexpectedly. Zero polling overhead, instant notification."},
        {text: "Subscribe to MQTT $SYS topics to detect disconnections", correct: false, feedback: "$SYS topics show broker statistics but don't provide per-device offline notifications."}
      ],
      difficulty: "medium",
      topic: "mqtt-lwt"
    }));
  }
}

1188.9 For Kids: Meet the Sensor Squad!

Sensor Squad Adventure: The Magic Message Board

MQTT is like a magical bulletin board where friends can post notes and everyone who cares about that topic gets a copy automatically!

One day, the Sensor Squad discovered a problem in their smart treehouse. Thermo the temperature sensor wanted to tell everyone when it got too hot, Lumi the light sensor needed to announce when it got dark, Speedy the motion detector wanted to alert everyone about visitors, and Droplet the humidity sensor had to warn about rain coming. But with everyone shouting messages at the same time, nobody could hear anything!

Then wise old Broker Bear arrived with a magical bulletin board. “Instead of shouting,” Broker Bear explained, “you can publish your messages to different sections of my board. Thermo, you post to the ‘temperature’ section. Lumi, you use the ‘light’ section.”

The Sensor Squad loved this idea! Broker Bear continued, “And here’s the magic part - anyone who wants to know about temperature can subscribe to that section. Whenever Thermo posts a new message, my magic board automatically sends a copy to everyone who subscribed!”

1188.9.1 Key Words for Kids

Word	What It Means
Publish	Posting a message to the bulletin board
Subscribe	Signing up to receive messages from a topic
Broker	The helpful friend who runs the bulletin board
Topic	A labeled section on the board

1188.9.2 Try This at Home!

Create your own family message board! Get a cork board and divide it into sections like “Kitchen,” “Living Room,” and “Bedrooms.” Family members can post sticky notes to sections, and anyone interested checks for new messages. This is exactly how MQTT works in smart homes!

1188.10 Summary

MQTT’s publish-subscribe pattern provides:

Decoupling: Publishers and subscribers don’t need to know about each other
Topics: Hierarchical naming for message organization
Wildcards: + for single-level, # for multi-level matching
Retained Messages: Instant state for new subscribers
Last Will Testament: Automatic offline detection

1188.11 What’s Next

Now that you understand MQTT’s publish-subscribe fundamentals:

Next Chapter: MQTT Quality of Service - Master QoS levels for reliable message delivery
Hands-on Practice: MQTT Labs - Build real MQTT clients with ESP32 and Python
Alternative Protocol: CoAP Fundamentals - Compare with REST-style IoT messaging