32  MQTT Implementation - Hands-On Labs

In 60 Seconds

These hands-on labs cover building complete MQTT systems with real hardware: ESP32 DHT22 temperature publishers with configurable QoS levels, multi-device motion-to-lighting automation pipelines, secure broker deployment with TLS and ACLs, and Last Will Testament (LWT) for automatic device disconnection detection.

32.1 Learning Objectives

By the end of this chapter, you will be able to:

  • Construct ESP32 MQTT Publishers: Implement complete sensor-to-broker data pipelines with DHT22 temperature sensors using the PubSubClient library
  • Design Multi-Device Automation Systems: Configure motion sensors and lighting actuators to communicate through an MQTT broker backbone
  • Select and Justify QoS Levels: Evaluate QoS 0, 1, and 2 trade-offs and select the appropriate level based on reliability requirements and power constraints
  • Configure Secure MQTT Brokers: Deploy Mosquitto with TLS certificates, username/password authentication, and ACLs for production-ready operation
  • Implement Last Will Testament: Configure LWT messages to automatically detect and report unexpected device disconnections
  • Calculate Battery Life Impact: Analyze the energy cost of different QoS levels and compare always-on versus deep-sleep publishing strategies
  • MQTT: Message Queuing Telemetry Transport — pub/sub protocol optimized for constrained IoT devices over unreliable networks
  • Broker: Central server routing messages from publishers to all matching subscribers by topic pattern
  • Topic: Hierarchical string (e.g., home/bedroom/temperature) used to route messages to interested subscribers
  • QoS Level: Quality of Service 0/1/2 trading delivery guarantee for message overhead
  • Retained Message: Last message on a topic stored by broker for immediate delivery to new subscribers
  • Last Will and Testament: Pre-configured message published by broker when a client disconnects ungracefully
  • Persistent Session: Broker stores subscriptions and pending messages allowing clients to resume after disconnection

32.2 For Beginners: MQTT Hands-On Labs

These labs guide you through building real MQTT systems step by step. You will set up a broker, connect devices, publish sensor data, and subscribe to updates. It is like assembling furniture with instructions – each step builds on the previous one until you have a complete, working MQTT application.

32.3 Prerequisites

Before diving into this chapter, you should be familiar with:

32.4 Lab 1: ESP32 DHT22 MQTT Publisher with QoS Levels

Objective: Build a temperature/humidity sensor that publishes to MQTT with different QoS levels.

Materials:

  • ESP32 development board
  • DHT22 temperature/humidity sensor
  • 10k ohm pull-up resistor
  • Breadboard and jumper wires
  • Wi-Fi connection

Circuit Diagram:

DHT22          ESP32
-----          -----
VCC   ------>  3.3V
DATA  ------>  GPIO 4 (with 10k ohm pull-up to 3.3V)
GND   ------>  GND

Complete Code:

#include <WiFi.h>
#include <PubSubClient.h>
#include <DHT.h>

// Wi-Fi credentials
const char* ssid = "YOUR_WIFI_SSID";
const char* password = "YOUR_WIFI_PASSWORD";

// MQTT Broker settings
const char* mqtt_server = "test.mosquitto.org";
const int mqtt_port = 1883;
const char* mqtt_client_id = "ESP32_DHT22_001";

// DHT22 sensor
#define DHTPIN 4
#define DHTTYPE DHT22
DHT dht(DHTPIN, DHTTYPE);

// MQTT topics
const char* topic_temp = "iotclass/lab1/temperature";
const char* topic_humidity = "iotclass/lab1/humidity";
const char* topic_status = "iotclass/lab1/status";

WiFiClient espClient;
PubSubClient client(espClient);

// Forward declaration (defined below loop)
void mqtt_callback(char* topic, byte* payload, unsigned int length);

unsigned long lastPublish = 0;
const long publishInterval = 5000; // 5 seconds

void setup_wifi() {
  Serial.println("\nConnecting to Wi-Fi...");
  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("\nWi-Fi connected");
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());
}

void reconnect_mqtt() {
  while (!client.connected()) {
    Serial.print("Connecting to MQTT broker...");

    if (client.connect(mqtt_client_id)) {
      Serial.println(" Connected");

      // Publish online status as retained message
      client.publish(topic_status, "online", true);

      // Subscribe to commands (optional)
      client.subscribe("iotclass/lab1/command");
    } else {
      Serial.print(" Failed, rc=");
      Serial.println(client.state());
      delay(5000);
    }
  }
}

void setup() {
  Serial.begin(115200);
  dht.begin();

  setup_wifi();

  client.setServer(mqtt_server, mqtt_port);
  client.setCallback(mqtt_callback);
}

void mqtt_callback(char* topic, byte* payload, unsigned int length) {
  Serial.print("Message received on ");
  Serial.print(topic);
  Serial.print(": ");

  String message = "";
  for (int i = 0; i < length; i++) {
    message += (char)payload[i];
  }
  Serial.println(message);
}

void loop() {
  if (!client.connected()) {
    reconnect_mqtt();
  }
  client.loop();

  unsigned long now = millis();
  if (now - lastPublish >= publishInterval) {
    lastPublish = now;

    // Read sensor
    float humidity = dht.readHumidity();
    float temperature = dht.readTemperature();

    if (isnan(humidity) || isnan(temperature)) {
      Serial.println("Failed to read from DHT sensor!");
      return;
    }

    // Publish temperature with QoS 0
    char tempStr[8];
    dtostrf(temperature, 6, 2, tempStr);
    bool temp_success = client.publish(topic_temp, tempStr, false);
    Serial.print("Temperature: ");
    Serial.print(tempStr);
    Serial.print("C ");
    Serial.println(temp_success ? "OK" : "FAIL");

    // Publish humidity with QoS 0
    char humStr[8];
    dtostrf(humidity, 6, 2, humStr);
    bool hum_success = client.publish(topic_humidity, humStr, false);
    Serial.print("Humidity: ");
    Serial.print(humStr);
    Serial.print("% ");
    Serial.println(hum_success ? "OK" : "FAIL");

    Serial.println("---");
  }
}

Expected Output (Serial Monitor):

Connecting to Wi-Fi...
Wi-Fi connected
IP address: 192.168.1.100
Connecting to MQTT broker... Connected
Temperature: 22.50C OK
Humidity: 45.30% OK
***
Temperature: 22.48C OK
Humidity: 45.35% OK
***
Interactive Simulator: MQTT Publisher (ESP32 + DHT22)

Try it yourself! See a complete IoT system publishing sensor data to an MQTT broker.

What This Simulates: An ESP32 with DHT22 sensor connecting to Wi-Fi, then publishing temperature and humidity data to an MQTT broker every 5 seconds.

How to Use:

  1. Click the Play button to start simulation
  2. Watch the Serial Monitor show Wi-Fi connection
  3. Observe MQTT broker connection with client ID
  4. See temperature/humidity published to topics
  5. Notice QoS 0 delivery with confirmation
  6. Open MQTT Explorer or subscriber to see messages in real-time!

Learning Points

Observe:

  • PubSubClient Library: Arduino MQTT client for ESP32
  • client.connect(): Establishes connection to broker with unique client ID
  • client.publish(): Sends message to topic (returns true/false)
  • Topic Hierarchy: iotclass/lab1/temperature uses / separators
  • QoS 0 (At Most Once): Fire-and-forget, fastest but no guarantee
  • Retained Messages: client.publish(topic, msg, true) stores last value for new subscribers
  • Auto-Reconnect: if (!client.connected()) reconnect_mqtt()

MQTT Publishing Flow:

1. ESP32 connects to Wi-Fi network
2. ESP32 connects to MQTT broker (test.mosquitto.org:1883)
3. ESP32 publishes "online" status as retained message
4. Every 5 seconds:
   a. Read DHT22 sensor (temperature, humidity)
   b. Convert float to string
   c. Publish to "iotclass/lab1/temperature" topic
   d. Publish to "iotclass/lab1/humidity" topic
   e. Print success/failure indicators
5. Maintain connection with client.loop()

MQTT Topic Structure:

MQTT topic tree structure
Figure 32.1: MQTT topic hierarchy for lab sensor data

Quality of Service Levels:

QoS 0 (At Most Once): Fast, no acknowledgment - used here
QoS 1 (At Least Once): Acknowledged, may deliver duplicates
QoS 2 (Exactly Once): Slowest, guaranteed single delivery

Real-World Applications:

  • Smart Agriculture: Soil moisture sensors publishing to cloud dashboard
  • Industrial Monitoring: Temperature/pressure sensors in manufacturing
  • Home Automation: Smart thermostats publishing temperature readings
  • Environmental Monitoring: Weather stations sending data to aggregation service
  • Asset Tracking: GPS devices publishing location updates

Experiment:

  • Add more sensors (motion, light) and publish to separate topics
  • Implement QoS 1 or 2 to see acknowledgments
  • Add Last Will and Testament message for disconnect detection
  • Publish JSON payload with multiple values: {"temp":22.5,"hum":45.3}
  • Add timestamp to messages for data logging

Learning Outcomes:

  • Configure ESP32 Wi-Fi connection
  • Integrate DHT22 sensor with MQTT
  • Implement publish with QoS levels
  • Handle MQTT reconnection
  • Use retained messages for status

Challenges:

  1. Modify to publish only when temperature changes by +/-0.5C (reduce traffic)
  2. Add battery voltage monitoring and publish with QoS 2
  3. Implement Last Will and Testament to detect unexpected disconnections
  4. Add JSON payload with multiple sensor readings
Try It: MQTT Topic Hierarchy Builder

Design and visualize your own MQTT topic structure. Choose a domain and configure the hierarchy depth to see how topics scale.

Problem: Your DHT22 ESP32 sensors are publishing every 5 seconds (17,280 messages/day per sensor). With 50 sensors, that’s 864,000 messages/day. AWS IoT Core charges $1/million messages, but you’re also hitting broker CPU limits.

Analysis:

Temperature in a room rarely changes more than 0.5°C per minute. Publishing every 5 seconds means 99% of messages report identical values.

Step 1: Calculate Current Costs

  • Messages/month: 864,000 × 30 = 25.9M messages
  • AWS IoT Core: 25.9 × $1 = $25.90/month
  • Broker CPU: Mosquitto on m5.large = $70/month
  • Total: $95.90/month

Step 2: Design Optimization

Publish only when temperature changes by ≥0.5°C OR 5 minutes elapse (heartbeat):

float lastTemp = 0.0;
unsigned long lastPublish = 0;
const float THRESHOLD = 0.5;  // 0.5°C change triggers publish
const unsigned long HEARTBEAT = 300000;  // 5 min max interval

void loop() {
    float temp = dht.readTemperature();
    unsigned long now = millis();

    // Publish if temp changed OR heartbeat timeout
    if (abs(temp - lastTemp) >= THRESHOLD ||
        (now - lastPublish >= HEARTBEAT)) {

        client.publish(topic_temp, String(temp).c_str());
        lastTemp = temp;
        lastPublish = now;
    }

    delay(5000);  // Still sample every 5s, but don't always publish
}

Step 3: Calculate New Costs

Typical office temperature changes 2-3 times/hour (HVAC cycles). With 5-minute heartbeat:

  • Meaningful changes: ~3/hour = 72/day
  • Heartbeats: (1440 min/day) / 5 = 288/day
  • Total: 360 messages/day per sensor (was 17,280)

For 50 sensors: - Messages/month: 360 × 50 × 30 = 540,000 messages - AWS IoT Core: 0.54 × $1 = $0.54/month - Broker CPU: Can downgrade to t3.small = $15/month - New total: $15.54/month → 83% cost reduction

Step 4: Additional Benefits

  • Bandwidth saved: 98% reduction (25.9M → 540K messages/month)
  • Battery life: Publishing less often extends battery 48x (still sampling, but not transmitting)
  • Storage costs: Time-series DB costs drop proportionally

Key Insight: Don’t blindly publish on a timer. Publish on change with a heartbeat fallback. This pattern works for most slowly-changing telemetry (temperature, humidity, pressure, battery voltage).

Pitfall: Broker Connection Limits Causing Silent Failures

The Mistake: Developers deploy IoT systems without considering broker connection limits. They test with 10-20 devices successfully, then deploy 500+ devices to production. New devices fail to connect with cryptic errors like “connection refused” or timeout, while existing connections work fine.

Why It Happens: MQTT brokers have configurable maximum connection limits, often defaulting to 1,024 (OS file descriptor limit) or lower. Each MQTT connection consumes a file descriptor, memory for session state (~2-10KB), and a TCP socket. When limits are reached, new connections are silently rejected without clear error messages.

The Fix: Calculate connection requirements before deployment. Configure broker limits explicitly. Implement connection health monitoring and alerting when approaching 80% capacity. Use connection pooling or MQTT bridge patterns for high-scale deployments.

Capacity Planning Formula: Required connections = (devices x 1.2) + (backend_services x 2) + (monitoring x 3). For 1,000 devices with 5 backend services and 2 monitoring tools, plan for: (1000 x 1.2) + (5 x 2) + (2 x 3) = 1,216 connections minimum.

32.5 Lab 2: Python MQTT Dashboard with Multiple Sensors

Objective: Create a real-time dashboard that subscribes to multiple sensor topics and displays data.

Materials:

  • Python 3.7+
  • paho-mqtt library
  • (Optional) Running ESP32 from Lab 1

Expected Output:

IoT MQTT Dashboard - Real-Time Sensor Data

LAB1
Temperature      22.50C              [14:32:15] QoS:0
Humidity         45.30%               [14:32:15] QoS:0
Status           Status: online       [14:32:10] QoS:0

Total messages received: 127
Last update: 14:32:15

Press Ctrl+C to exit

Learning Outcomes:

  • Subscribe to multiple MQTT topics with wildcards
  • Build real-time data visualization
  • Implement alert systems based on sensor data
  • Handle MQTT callbacks and data processing
  • Create user-friendly console interfaces

32.6 Lab 3: MQTT Home Automation - Lights and Motion

Objective: Build a complete home automation system with motion detection and automated lighting control.

Materials:

  • 2x ESP32 boards (one for motion sensor, one for light control)
  • PIR motion sensor (HC-SR501)
  • LED (or relay module for real lights)
  • 220 ohm resistor
  • Breadboard and wires

Circuit 1 - Motion Sensor (ESP32 #1):

PIR Sensor     ESP32
----------     -----
VCC    ------>  5V
OUT    ------>  GPIO 13
GND    ------>  GND

Circuit 2 - Light Control (ESP32 #2):

ESP32         LED
-----         ---
GPIO 2  ----> LED Anode (through 220 ohm resistor)
GND     ----> LED Cathode

Code for Motion Sensor (ESP32 #1):

#include <WiFi.h>
#include <PubSubClient.h>

const char* ssid = "YOUR_WIFI_SSID";
const char* password = "YOUR_WIFI_PASSWORD";
const char* mqtt_server = "test.mosquitto.org";

WiFiClient espClient;
PubSubClient client(espClient);

#define PIR_PIN 13
#define ROOM_ID "living_room"

char motion_topic[50];
char light_command_topic[50];

bool last_motion_state = false;
unsigned long motion_start_time = 0;
const unsigned long AUTO_OFF_DELAY = 30000; // 30 seconds

void setup() {
  Serial.begin(115200);
  pinMode(PIR_PIN, INPUT);

  sprintf(motion_topic, "home/%s/motion", ROOM_ID);
  sprintf(light_command_topic, "home/%s/light/command", ROOM_ID);

  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("\nWi-Fi connected");

  client.setServer(mqtt_server, 1883);
  reconnect();
}

void reconnect() {
  while (!client.connected()) {
    Serial.print("Connecting to MQTT...");
    if (client.connect("ESP32_MotionSensor")) {
      Serial.println(" Connected");
    } else {
      delay(5000);
    }
  }
}

void loop() {
  if (!client.connected()) {
    reconnect();
  }
  client.loop();

  bool motion_detected = digitalRead(PIR_PIN) == HIGH;

  // Motion started
  if (motion_detected && !last_motion_state) {
    Serial.println("Motion detected!");
    client.publish(motion_topic, "true", true);

    // Turn on light
    client.publish(light_command_topic, "ON");
    motion_start_time = millis();
    last_motion_state = true;
  }

  // Motion stopped
  if (!motion_detected && last_motion_state) {
    Serial.println("   Motion cleared");
    client.publish(motion_topic, "false", true);
    last_motion_state = false;
  }

  // Auto turn off light after delay
  if (!motion_detected && (millis() - motion_start_time > AUTO_OFF_DELAY)) {
    client.publish(light_command_topic, "OFF");
    Serial.println("Auto turning off light");
    motion_start_time = millis() + 1000000; // Prevent repeated commands
  }

  delay(200);
}

Code for Light Control (ESP32 #2):

#include <WiFi.h>
#include <PubSubClient.h>

const char* ssid = "YOUR_WIFI_SSID";
const char* password = "YOUR_WIFI_PASSWORD";
const char* mqtt_server = "test.mosquitto.org";

WiFiClient espClient;
PubSubClient client(espClient);

#define LED_PIN 2
#define ROOM_ID "living_room"

char light_command_topic[50];
char light_state_topic[50];

void mqtt_callback(char* topic, byte* payload, unsigned int length) {
  String message = "";
  for (int i = 0; i < length; i++) {
    message += (char)payload[i];
  }

  Serial.print("Received command: ");
  Serial.println(message);

  if (message == "ON") {
    digitalWrite(LED_PIN, HIGH);
    client.publish(light_state_topic, "ON", true);
    Serial.println("Light turned ON");
  } else if (message == "OFF") {
    digitalWrite(LED_PIN, LOW);
    client.publish(light_state_topic, "OFF", true);
    Serial.println("Light turned OFF");
  }
}

void setup() {
  Serial.begin(115200);
  pinMode(LED_PIN, OUTPUT);
  digitalWrite(LED_PIN, LOW);

  sprintf(light_command_topic, "home/%s/light/command", ROOM_ID);
  sprintf(light_state_topic, "home/%s/light/state", ROOM_ID);

  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("\nWi-Fi connected");

  client.setServer(mqtt_server, 1883);
  client.setCallback(mqtt_callback);

  reconnect();
}

void reconnect() {
  while (!client.connected()) {
    Serial.print("Connecting to MQTT...");
    if (client.connect("ESP32_LightControl")) {
      Serial.println(" Connected");

      // Subscribe to light commands
      client.subscribe(light_command_topic);
      Serial.print("Subscribed to: ");
      Serial.println(light_command_topic);

      // Publish initial state
      client.publish(light_state_topic, "OFF", true);
    } else {
      delay(5000);
    }
  }
}

void loop() {
  if (!client.connected()) {
    reconnect();
  }
  client.loop();
}

Expected Serial Output (Motion Sensor):

Wi-Fi connected
Connecting to MQTT... Connected
Motion detected!
   Motion cleared
Auto turning off light

Expected Serial Output (Light Control):

Wi-Fi connected
Connecting to MQTT... Connected
Subscribed to: home/living_room/light/command
Received command: ON
Light turned ON
Received command: OFF
Light turned OFF
Interactive Simulator: MQTT Subscriber (Light Control)

What This Simulates: ESP32 subscribing to MQTT commands and controlling an LED based on received messages - the other half of publish-subscribe.

Key Points

MQTT Subscription Flow:

  1. ESP32 subscribes to home/living_room/light/command
  2. Broker forwards matching messages to this client
  3. mqtt_callback() function executes when message arrives
  4. LED state changes based on “ON”/“OFF” payload
  5. ESP32 publishes new state to home/living_room/light/state (retained)

Real-World: Smart home lights, automated curtains, door locks, HVAC controls

Experiment: Add dimming levels (0-100), multiple rooms, motion sensor integration

Learning Outcomes:

  • Build multi-device MQTT communication
  • Implement automation logic with sensors and actuators
  • Use retained messages for state synchronization
  • Create topic naming conventions for home automation
  • Handle timing and auto-off functionality

Challenges:

  1. Add manual control via MQTT (smartphone app or Node-RED)
  2. Implement brightness control with PWM
  3. Add multiple rooms with independent control
  4. Create schedules (morning/evening modes)
Try It: Motion-to-Light Automation Timing

Experiment with motion sensor timing parameters to understand how AUTO_OFF_DELAY, PIR hold time, and sensor polling rate affect automation behavior and power consumption.

32.7 Lab 4: Secure MQTT with TLS and Authentication

Objective: Implement MQTT security using TLS encryption and username/password authentication.

Materials:

  • ESP32 or Python client
  • Private MQTT broker (Mosquitto installed locally or on Raspberry Pi)
  • SSL/TLS certificates

Step 1: Install and Configure Mosquitto Broker

Install Mosquitto on Linux/Raspberry Pi:

sudo apt update
sudo apt install mosquitto mosquitto-clients

Step 2: Generate SSL/TLS Certificates

# Create certificate directory
sudo mkdir -p /etc/mosquitto/certs
cd /etc/mosquitto/certs

# Generate CA certificate
sudo openssl req -new -x509 -days 365 -extensions v3_ca \
  -keyout ca.key -out ca.crt \
  -subj "/C=US/ST=State/L=City/O=IoTClass/CN=CA"

# Generate server key and certificate
sudo openssl genrsa -out server.key 2048
sudo openssl req -new -out server.csr -key server.key \
  -subj "/C=US/ST=State/L=City/O=IoTClass/CN=mqtt.local"
sudo openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out server.crt -days 365

# Set permissions
sudo chmod 644 /etc/mosquitto/certs/*.crt
sudo chmod 600 /etc/mosquitto/certs/*.key

Step 3: Configure Mosquitto with Security

Edit /etc/mosquitto/mosquitto.conf:

# Default listener (disabled)
listener 1883
allow_anonymous false

# TLS listener
listener 8883
cafile /etc/mosquitto/certs/ca.crt
certfile /etc/mosquitto/certs/server.crt
keyfile /etc/mosquitto/certs/server.key
require_certificate false

# Password file
password_file /etc/mosquitto/passwd

# Logging
log_dest file /var/log/mosquitto/mosquitto.log
log_type all

Step 4: Create User Accounts

# Create password file with first user
sudo mosquitto_passwd -c /etc/mosquitto/passwd iotuser

# Add more users (without -c flag)
sudo mosquitto_passwd /etc/mosquitto/passwd admin

Step 5: Restart Mosquitto

sudo systemctl restart mosquitto
sudo systemctl status mosquitto
MQTT security architecture with four defense layers: TLS encryption preventing eavesdropping, authentication verifying client identity, access control lists enforcing topic permissions, and monitoring detecting anomalous behavior patterns
(a) MQTT security defense layers
Figure 32.2: MQTT security architecture with four defense layers: TLS encryption prevents eavesdropping, authentication verifies client identity, ACLs enforce topic permissions, and monitoring detects anomalous behavior, blocking four common attack vectors.

Testing Connection Security:

# Test with mosquitto_pub (with authentication)
mosquitto_pub -h mqtt.local -p 8883 \
  -u iotuser -P your_password \
  --cafile /etc/mosquitto/certs/ca.crt \
  -t "secure/test" -m "Hello from terminal" -q 1

# Test without authentication (should fail)
mosquitto_pub -h mqtt.local -p 8883 \
  --cafile /etc/mosquitto/certs/ca.crt \
  -t "secure/test" -m "This will fail"

Learning Outcomes:

  • Generate SSL/TLS certificates for MQTT
  • Configure Mosquitto broker with security
  • Implement username/password authentication
  • Use encrypted MQTT connections
  • Understand certificate-based security
  • Test and troubleshoot secure MQTT

Security Best Practices:

  1. Always use TLS in production (port 8883)
  2. Disable anonymous access
  3. Use strong passwords (12+ characters)
  4. Implement topic-level ACLs (Access Control Lists)
  5. Keep certificates updated (renew before expiration)
  6. Monitor broker logs for suspicious activity
  7. Use client certificates for enhanced security (mutual TLS)
Try It: MQTT Security Configuration Advisor

Select your deployment scenario and security requirements to get a tailored Mosquitto configuration with an overall security score.

32.8 Interactive Simulator: MQTT QoS Levels and Last Will Testament

What This Simulates: ESP32 demonstrating MQTT Quality of Service levels and Last Will and Testament for reliable communication

QoS Levels Explained:

QoS 0 (At Most Once):          QoS 1 (At Least Once):      QoS 2 (Exactly Once):
Publisher -> Message ->         Publisher -> Message ->      Publisher -> Message ->
Broker -> Subscriber            Broker -> Subscriber         Broker -> PUBREC <-
                                Broker <- PUBACK             Publisher -> PUBREL ->
Fast, no guarantee              May duplicate                Broker -> PUBCOMP <-
                                Acknowledged                 Guaranteed once

Use for: sensor readings        Use for: commands            Use for: billing data
Power: Lowest                   Power: Medium                Power: Highest
Latency: ~5ms                   Latency: ~15ms               Latency: ~30ms
Quick Check: QoS Levels

Last Will and Testament (LWT):

Device connects with LWT set:
{
  topic: "devices/sensor01/status",
  message: "offline",
  qos: 1,
  retain: true
}

Normal flow:                    Unexpected disconnect:
1. Device connects              1. Device crashes
2. Publishes "online"           2. Network timeout (60s)
3. Sends data periodically      3. Broker detects disconnect
4. Publishes "offline"          4. Broker publishes LWT:
5. Disconnects gracefully          "devices/sensor01/status" -> "offline"
                                5. Monitoring system alerted

Result: Device status always known, even after crash

How to Use:

  1. Click the Play button to start simulation
  2. Watch LWT being set during connection
  3. Observe messages sent with different QoS levels
  4. See PUBACK responses for QoS 1
  5. Monitor retained status messages

Learning Points

QoS Level Comparison:

Metric QoS 0 QoS 1 QoS 2
Guarantee None At least once Exactly once
Duplicates No Possible Never
ACK overhead 0 bytes 4 bytes (PUBACK) 12 bytes (PUBREC + PUBREL + PUBCOMP)
Use case Sensors Commands Billing

Last Will and Testament Flow:

  1. CONNECT packet with LWT fields, for example:
    • Client ID: sensor01
    • LWT topic: devices/sensor01/lwt
    • LWT message: "offline"
    • LWT QoS: 1
    • LWT retain flag: true
    • Keep-alive: 60 seconds
  2. The broker stores the LWT but does not publish it yet.
  3. If the client disconnects gracefully, the LWT is discarded. If the client times out or crashes, the broker publishes the LWT message.
  4. Monitoring clients subscribed to devices/+/lwt receive the notification and can alert an operator or trigger remediation.

Real-World Applications:

  1. Healthcare - Patient monitor disconnection alerts (QoS 2)
  2. Industrial - Equipment status tracking with LWT
  3. Smart Home - Door/window sensor states (QoS 1, retained)
  4. Fleet Management - Vehicle tracking with offline detection
  5. Agriculture - Irrigation system failure notification

Experiments to Try:

  1. Compare QoS - Send 100 messages with each QoS, measure time
  2. Network Interruption - Disconnect Wi-Fi, observe LWT trigger
  3. Retained Status - New subscriber receives last status immediately
  4. Duplicate Detection - QoS 1 may deliver twice, handle with message IDs
  5. Battery Impact - Calculate battery life with different QoS levels

Choosing QoS Level (Rule of Thumb):

  • Telemetry (temperature, heartbeat): QoS 0 – frequent, replaceable
  • Commands (door sensor, firmware): QoS 1 – important, idempotent
  • Transactions (payment, config change): QoS 2 – critical, non-idempotent

LWT Best Practices:

DO:
- Set LWT on critical devices
- Use retain=true for status topics
- Set QoS 1 for LWT
- Include timestamp in LWT message
- Subscribe to +/status for monitoring

DON'T:
- Use large LWT messages (keep <100 bytes)
- Set very short keep-alive (<30s)
- Forget to publish "online" after connect
- Use QoS 0 for LWT (unreliable)

Example LWT message:
{
  "status": "offline",
  "reason": "unexpected",
  "timestamp": 1698765432,
  "last_seen": "2024-10-28T10:23:52Z"
}
Try It: MQTT Keep-Alive and LWT Timeout Planner

Configure keep-alive and LWT settings to balance between fast disconnect detection and network overhead. The broker detects a dead client after 1.5x the keep-alive interval (per MQTT spec).

Pitfall: Stale Retained Messages After Device Removal

The Mistake: Developers use retained messages for device status (e.g., devices/sensor42/status with retain=true), but when devices are decommissioned or replaced, the old retained messages remain on the broker indefinitely. New subscribers receive stale “online” status for devices that no longer exist.

Why It Happens: Retained messages persist until explicitly cleared with an empty payload. Most developers focus on the “publish” side and forget that retained messages require lifecycle management. Brokers like Mosquitto will keep retained messages forever unless configured otherwise.

The Fix: Implement device decommissioning that clears retained messages. Publish an empty payload (zero-length) with retain=true to delete the retained message. For MQTT 5.0, use Message Expiry Interval to auto-expire stale status.

# MQTT 3.1.1: Clear retained message on device removal
def decommission_device(client, device_id):
    # Publish empty payload with retain=true to clear
    client.publish(f"devices/{device_id}/status", payload="", qos=1, retain=True)
    client.publish(f"devices/{device_id}/config", payload="", qos=1, retain=True)
    client.publish(f"devices/{device_id}/lwt", payload="", qos=1, retain=True)
    print(f"Cleared all retained messages for {device_id}")

# MQTT 5.0: Auto-expire status messages
from paho.mqtt.properties import Properties
from paho.mqtt.packettypes import PacketTypes

props = Properties(PacketTypes.PUBLISH)
props.MessageExpiryInterval = 300  # Expire after 5 minutes of no update

client.publish(
    "devices/sensor01/status",
    payload='{"status": "online", "timestamp": 1699123456}',
    qos=1,
    retain=True,
    properties=props
)
# If device stops publishing, status auto-clears after 5 minutes

Broker Configuration (Mosquitto): Set retain_available false to disable retained messages entirely if your application doesn’t need them, or use $SYS/broker/retained messages/count to monitor accumulation.

Pitfall: Last Will Triggered on Graceful Disconnect

The Mistake: Developers configure Last Will and Testament (LWT) to publish “offline” status, expecting it only fires on crashes or network failures. But they observe LWT messages being published even during normal shutdown sequences, flooding monitoring systems with false “offline” alerts.

Why It Happens: The MQTT spec states LWT is published when the broker closes the connection “without receiving a DISCONNECT packet.” Many developers call client.disconnect() but don’t wait for completion, or the TCP connection closes before the DISCONNECT packet is sent. Network issues during graceful shutdown can also prevent DISCONNECT delivery.

The Fix: Always explicitly publish “offline” status before disconnecting, then send DISCONNECT. Use blocking disconnect or wait for confirmation. For Python paho-mqtt, use disconnect() followed by loop_stop() in the correct order.

// ESP32: Graceful shutdown with explicit status
void gracefulShutdown() {
    // Step 1: Publish explicit offline status
    mqttClient.publish("devices/sensor01/status", "offline", true);  // retain=true
    delay(100);  // Allow time for publish to complete

    // Step 2: Send DISCONNECT packet (prevents LWT)
    mqttClient.disconnect();
    delay(100);  // Allow TCP to close cleanly

    // Step 3: Now safe to power down
    WiFi.disconnect(true);
    esp_deep_sleep_start();
}

// If LWT fires anyway, it's redundant (already sent "offline")
// Better than missing offline status if crash occurs before explicit publish
# Python paho-mqtt 2.0+: Proper disconnect sequence
import time

def graceful_disconnect(client):
    # Publish explicit status first
    result = client.publish("devices/sensor01/status", "offline", qos=1, retain=True)
    result.wait_for_publish(timeout=5.0)  # Block until published

    # Now disconnect - this prevents LWT from firing
    client.disconnect()
    client.loop_stop()  # Stop network thread AFTER disconnect

# Common mistake: loop_stop() before disconnect()
# This kills the network thread before DISCONNECT packet is sent
# Result: Broker never receives DISCONNECT, triggers LWT

MQTT 5.0 Enhancement: Use Reason Code in DISCONNECT to tell broker why you’re disconnecting. Reason Code 0x00 (Normal disconnection) explicitly signals “don’t publish LWT.”

Battery Life Calculation:

Scenario: Send temperature every 60 seconds

QoS 0:
- Assumptions: ~120 mA radio current, ~50 ms radio-on time per publish
- Per message: 120 mA x 50 ms / 3,600,000 = 0.0017 mAh
- Per day: 1440 x 0.0017 = 2.4 mAh/day
- Battery (2000 mAh): 2000 / 2.4 = ~833 days (~2.3 years)

QoS 1:
- Assumptions: ~120 mA radio current, ~150 ms radio-on time (PUBLISH + PUBACK)
- Per message: 120 mA x 150 ms / 3,600,000 = 0.0050 mAh
- Per day: 1440 x 0.0050 = 7.2 mAh/day
- Battery (2000 mAh): 2000 / 7.2 = ~278 days (~9 months)

QoS 2:
- Assumptions: ~120 mA radio current, ~300 ms radio-on time (4-way handshake)
- Per message: 120 mA x 300 ms / 3,600,000 = 0.0100 mAh
- Per day: 1440 x 0.0100 = 14.4 mAh/day
- Battery (2000 mAh): 2000 / 14.4 = ~139 days (~4.6 months)

Conclusion: QoS 0 uses ~67% less TX energy than QoS 1; QoS 2 has the highest overhead.

Note: These estimates isolate MQTT exchange time. If the device reconnects Wi-Fi/TLS each minute (common with deep sleep), connection overhead will dominate and battery life will be much lower.

The QoS battery calculations above assume “always-on” Wi-Fi. With ESP32 deep sleep, connection overhead dominates:

Deep sleep scenario (ESP32 wakes every 60 s, sends 1 reading, sleeps):

Wi-Fi connection overhead:

  • Scan channels: 200 ms @ 80 mA = 4.44 µAh
  • Associate + DHCP: 1,500 ms @ 100 mA = 41.67 µAh
  • Total connection: 46.11 µAh per wake

MQTT connection (clean session): - TCP handshake: 150 ms @ 80 mA = 3.33 µAh - CONNECT + CONNACK: 100 ms @ 80 mA = 2.22 µAh - Total MQTT setup: 5.55 µAh

Message transmission (QoS 0):

  • PUBLISH: 50 ms @ 120 mA = 1.67 µAh
  • Total message: 1.67 µAh

Per-wake cycle: \[E_{total} = 46.11 + 5.55 + 1.67 = 53.33 \text{ µAh}\]

Daily energy (1,440 wakes): \[E_{day} = 1{,}440 \times 53.33 = 76{,}795 \text{ µAh} = 76.8 \text{ mAh}\]

Battery life (2000 mAh): \[\frac{2{,}000}{76.8} = 26 \text{ days}\]

Key insight: Wi-Fi connection (46 µAh) is 28× larger than the MQTT message (1.67 µAh). Persistent session with keepalive = 5 min eliminates reconnection overhead → 833 days battery life!

32.9 Interactive Design Tools

Use these calculators to plan your MQTT deployments with real numbers.

32.9.1 MQTT QoS Battery Life Calculator

Estimate how long your battery-powered MQTT device will last at each QoS level.

32.9.2 MQTT Broker Capacity Planner

Plan your MQTT broker resources based on device count and connection patterns.

32.9.3 MQTT Message Cost Estimator

Estimate monthly cloud MQTT messaging costs and potential savings from change-based publishing.

32.9.4 MQTT Deep Sleep Energy Calculator

Model the full wake cycle energy for ESP32 with deep sleep, including Wi-Fi and MQTT connection overhead.

32.10 Knowledge Check

Test your understanding of MQTT implementations with these questions:

32.11 Summary

This chapter provided complete hands-on MQTT implementation labs:

  • Lab 1 - ESP32 Publisher: Built a DHT22 temperature sensor system with QoS 0 publishing, retained status messages, and automatic reconnection
  • Lab 2 - Python Dashboard: Created multi-topic subscription with wildcards, real-time display, and alert thresholds
  • Lab 3 - Home Automation: Implemented motion-controlled lighting using two ESP32 devices communicating via MQTT
  • Lab 4 - Secure MQTT: Configured TLS certificates, username/password authentication, and ACLs for production deployment
  • QoS and LWT: Demonstrated Quality of Service levels (0/1/2) with battery impact calculations and Last Will Testament for disconnect detection
  • Common Pitfalls: Identified stale retained messages, LWT timing issues, and TLS timeout problems with solutions

32.12 Concept Relationships

MQTT Hands-On Labs connect to:

Lab progression: Lab 1 (basic publisher) → Lab 2 (Python dashboard) → Lab 3 (multi-device automation) → Lab 4 (secure deployment). Each builds on previous concepts while adding complexity.

32.13 See Also

32.14 What’s Next

Chapter Focus Why Read It
MQTT Comprehensive Review Scenario-based questions, MQTT vs CoAP vs HTTP comparisons, broker clustering Synthesizes all MQTT concepts from hands-on labs into production-ready knowledge
MQTT QoS and Session QoS 0/1/2 mechanics, clean vs persistent sessions, message queuing Deepens understanding of the QoS trade-offs applied throughout these labs
MQTT Security Fundamentals TLS mutual authentication, certificate management, threat modelling Expands on the Lab 4 security configuration with production security design
MQTT Python Patterns Reconnection logic, async callbacks, error handling, message batching Builds production-quality Python clients beyond the dashboard pattern in Lab 2
ESP32 Prototyping Hardware ESP32 pinouts, sensor wiring, power management, deep sleep Supports hardware setup for Labs 1 and 3 and extends to battery-powered designs
Network Design and Simulation Load testing, network simulation, capacity validation Validates MQTT broker capacity planning before production deployment