1205  MQTT Implementation - Hands-On Labs

1205.1 Learning Objectives

By the end of this chapter, you will be able to:

  • Build ESP32 MQTT Publishers: Create complete sensor-to-broker data pipelines with DHT22 temperature sensors
  • Implement Multi-Device Systems: Connect motion sensors to automated lighting using MQTT as the communication backbone
  • Configure QoS Levels: Choose appropriate QoS (0, 1, 2) based on reliability requirements and power constraints
  • Deploy Secure MQTT: Set up TLS certificates, authentication, and ACLs for production-ready brokers
  • Use Last Will Testament: Detect device disconnections automatically with LWT messages

1205.2 Prerequisites

Before diving into this chapter, you should be familiar with:

1205.3 Lab 1: ESP32 DHT22 MQTT Publisher with QoS Levels

Objective: Build a temperature/humidity sensor that publishes to MQTT with different QoS levels.

Materials: - ESP32 development board - DHT22 temperature/humidity sensor - 10k ohm pull-up resistor - Breadboard and jumper wires - Wi-Fi connection

Circuit Diagram:

DHT22          ESP32
-----          -----
VCC   ------>  3.3V
DATA  ------>  GPIO 4 (with 10k ohm pull-up to 3.3V)
GND   ------>  GND

Complete Code:

#include <WiFi.h>
#include <PubSubClient.h>
#include <DHT.h>

// Wi-Fi credentials
const char* ssid = "YOUR_WIFI_SSID";
const char* password = "YOUR_WIFI_PASSWORD";

// MQTT Broker settings
const char* mqtt_server = "test.mosquitto.org";
const int mqtt_port = 1883;
const char* mqtt_client_id = "ESP32_DHT22_001";

// DHT22 sensor
#define DHTPIN 4
#define DHTTYPE DHT22
DHT dht(DHTPIN, DHTTYPE);

// MQTT topics
const char* topic_temp = "iotclass/lab1/temperature";
const char* topic_humidity = "iotclass/lab1/humidity";
const char* topic_status = "iotclass/lab1/status";

WiFiClient espClient;
PubSubClient client(espClient);

unsigned long lastPublish = 0;
const long publishInterval = 5000; // 5 seconds

void setup_wifi() {
  Serial.println("\nConnecting to Wi-Fi...");
  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("\nWi-Fi connected");
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());
}

void reconnect_mqtt() {
  while (!client.connected()) {
    Serial.print("Connecting to MQTT broker...");

    if (client.connect(mqtt_client_id)) {
      Serial.println(" Connected");

      // Publish online status as retained message
      client.publish(topic_status, "online", true);

      // Subscribe to commands (optional)
      client.subscribe("iotclass/lab1/command");
    } else {
      Serial.print(" Failed, rc=");
      Serial.println(client.state());
      delay(5000);
    }
  }
}

void setup() {
  Serial.begin(115200);
  dht.begin();

  setup_wifi();

  client.setServer(mqtt_server, mqtt_port);
  client.setCallback(mqtt_callback);
}

void mqtt_callback(char* topic, byte* payload, unsigned int length) {
  Serial.print("Message received on ");
  Serial.print(topic);
  Serial.print(": ");

  String message = "";
  for (int i = 0; i < length; i++) {
    message += (char)payload[i];
  }
  Serial.println(message);
}

void loop() {
  if (!client.connected()) {
    reconnect_mqtt();
  }
  client.loop();

  unsigned long now = millis();
  if (now - lastPublish >= publishInterval) {
    lastPublish = now;

    // Read sensor
    float humidity = dht.readHumidity();
    float temperature = dht.readTemperature();

    if (isnan(humidity) || isnan(temperature)) {
      Serial.println("Failed to read from DHT sensor!");
      return;
    }

    // Publish temperature with QoS 0
    char tempStr[8];
    dtostrf(temperature, 6, 2, tempStr);
    bool temp_success = client.publish(topic_temp, tempStr, false);
    Serial.print("Temperature: ");
    Serial.print(tempStr);
    Serial.print("C ");
    Serial.println(temp_success ? "OK" : "FAIL");

    // Publish humidity with QoS 0
    char humStr[8];
    dtostrf(humidity, 6, 2, humStr);
    bool hum_success = client.publish(topic_humidity, humStr, false);
    Serial.print("Humidity: ");
    Serial.print(humStr);
    Serial.print("% ");
    Serial.println(hum_success ? "OK" : "FAIL");

    Serial.println("---");
  }
}

Expected Output (Serial Monitor):

Connecting to Wi-Fi...
Wi-Fi connected
IP address: 192.168.1.100
Connecting to MQTT broker... Connected
Temperature: 22.50C OK
Humidity: 45.30% OK
---
Temperature: 22.48C OK
Humidity: 45.35% OK
---
TipInteractive Simulator: MQTT Publisher (ESP32 + DHT22)

Try it yourself! See a complete IoT system publishing sensor data to an MQTT broker.

What This Simulates: An ESP32 with DHT22 sensor connecting to Wi-Fi, then publishing temperature and humidity data to an MQTT broker every 5 seconds.

How to Use: 1. Click the Play button to start simulation 2. Watch the Serial Monitor show Wi-Fi connection 3. Observe MQTT broker connection with client ID 4. See temperature/humidity published to topics 5. Notice QoS 0 delivery with confirmation 6. Open MQTT Explorer or subscriber to see messages in real-time!

NoteLearning Points

Observe: - PubSubClient Library: Arduino MQTT client for ESP32 - client.connect(): Establishes connection to broker with unique client ID - client.publish(): Sends message to topic (returns true/false) - Topic Hierarchy: iotclass/lab1/temperature uses / separators - QoS 0 (At Most Once): Fire-and-forget, fastest but no guarantee - Retained Messages: client.publish(topic, msg, true) stores last value for new subscribers - Auto-Reconnect: if (!client.connected()) reconnect_mqtt()

MQTT Publishing Flow:

1. ESP32 connects to Wi-Fi network
2. ESP32 connects to MQTT broker (test.mosquitto.org:1883)
3. ESP32 publishes "online" status as retained message
4. Every 5 seconds:
   a. Read DHT22 sensor (temperature, humidity)
   b. Convert float to string
   c. Publish to "iotclass/lab1/temperature" topic
   d. Publish to "iotclass/lab1/humidity" topic
   e. Print success/failure indicators
5. Maintain connection with client.loop()

MQTT Topic Structure:

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#16A085', 'secondaryColor': '#E67E22', 'tertiaryColor': '#ecf0f1'}}}%%
graph TB
    Root["iotclass/"]
    Lab1["lab1/"]
    Temp["temperature"]
    Hum["humidity"]
    Status["status"]

    Root --> Lab1
    Lab1 --> Temp
    Lab1 --> Hum
    Lab1 --> Status

    Temp -.->|"QoS 0"| Value1["22.50C"]
    Hum -.->|"QoS 0"| Value2["45.30%"]
    Status -.->|"Retained"| Value3["online"]

    style Root fill:#2C3E50,stroke:#16A085,stroke-width:2px,color:#fff
    style Lab1 fill:#16A085,stroke:#2C3E50,stroke-width:2px,color:#fff
    style Temp fill:#E67E22,stroke:#2C3E50,stroke-width:2px
    style Hum fill:#E67E22,stroke:#2C3E50,stroke-width:2px
    style Status fill:#E67E22,stroke:#2C3E50,stroke-width:2px

Figure 1205.1: MQTT topic hierarchy for lab sensor data

Quality of Service Levels:

QoS 0 (At Most Once): Fast, no acknowledgment - used here
QoS 1 (At Least Once): Acknowledged, may deliver duplicates
QoS 2 (Exactly Once): Slowest, guaranteed single delivery

Real-World Applications: - Smart Agriculture: Soil moisture sensors publishing to cloud dashboard - Industrial Monitoring: Temperature/pressure sensors in manufacturing - Home Automation: Smart thermostats publishing temperature readings - Environmental Monitoring: Weather stations sending data to aggregation service - Asset Tracking: GPS devices publishing location updates

Experiment: - Add more sensors (motion, light) and publish to separate topics - Implement QoS 1 or 2 to see acknowledgments - Add Last Will and Testament message for disconnect detection - Publish JSON payload with multiple values: {"temp":22.5,"hum":45.3} - Add timestamp to messages for data logging

Learning Outcomes: - Configure ESP32 Wi-Fi connection - Integrate DHT22 sensor with MQTT - Implement publish with QoS levels - Handle MQTT reconnection - Use retained messages for status

Challenges: 1. Modify to publish only when temperature changes by +/-0.5C (reduce traffic) 2. Add battery voltage monitoring and publish with QoS 2 3. Implement Last Will and Testament to detect unexpected disconnections 4. Add JSON payload with multiple sensor readings

CautionPitfall: Broker Connection Limits Causing Silent Failures

The Mistake: Developers deploy IoT systems without considering broker connection limits. They test with 10-20 devices successfully, then deploy 500+ devices to production. New devices fail to connect with cryptic errors like “connection refused” or timeout, while existing connections work fine.

Why It Happens: MQTT brokers have configurable maximum connection limits, often defaulting to 1,024 (OS file descriptor limit) or lower. Each MQTT connection consumes a file descriptor, memory for session state (~2-10KB), and a TCP socket. When limits are reached, new connections are silently rejected without clear error messages.

The Fix: Calculate connection requirements before deployment. Configure broker limits explicitly. Implement connection health monitoring and alerting when approaching 80% capacity. Use connection pooling or MQTT bridge patterns for high-scale deployments.

Capacity Planning Formula: Required connections = (devices x 1.2) + (backend_services x 2) + (monitoring x 3). For 1,000 devices with 5 backend services and 2 monitoring tools, plan for: (1000 x 1.2) + (5 x 2) + (2 x 3) = 1,216 connections minimum.


1205.4 Lab 2: Python MQTT Dashboard with Multiple Sensors

Objective: Create a real-time dashboard that subscribes to multiple sensor topics and displays data.

Materials: - Python 3.7+ - paho-mqtt library - (Optional) Running ESP32 from Lab 1

Expected Output:

IoT MQTT Dashboard - Real-Time Sensor Data

LAB1
Temperature      22.50C              [14:32:15] QoS:0
Humidity         45.30%               [14:32:15] QoS:0
Status           Status: online       [14:32:10] QoS:0

Total messages received: 127
Last update: 14:32:15

Press Ctrl+C to exit

Learning Outcomes: - Subscribe to multiple MQTT topics with wildcards - Build real-time data visualization - Implement alert systems based on sensor data - Handle MQTT callbacks and data processing - Create user-friendly console interfaces


1205.5 Lab 3: MQTT Home Automation - Lights and Motion

Objective: Build a complete home automation system with motion detection and automated lighting control.

Materials: - 2x ESP32 boards (one for motion sensor, one for light control) - PIR motion sensor (HC-SR501) - LED (or relay module for real lights) - 220 ohm resistor - Breadboard and wires

Circuit 1 - Motion Sensor (ESP32 #1):

PIR Sensor     ESP32
----------     -----
VCC    ------>  5V
OUT    ------>  GPIO 13
GND    ------>  GND

Circuit 2 - Light Control (ESP32 #2):

ESP32         LED
-----         ---
GPIO 2  ----> LED Anode (through 220 ohm resistor)
GND     ----> LED Cathode

Code for Motion Sensor (ESP32 #1):

#include <WiFi.h>
#include <PubSubClient.h>

const char* ssid = "YOUR_WIFI_SSID";
const char* password = "YOUR_WIFI_PASSWORD";
const char* mqtt_server = "test.mosquitto.org";

WiFiClient espClient;
PubSubClient client(espClient);

#define PIR_PIN 13
#define ROOM_ID "living_room"

char motion_topic[50];
char light_command_topic[50];

bool last_motion_state = false;
unsigned long motion_start_time = 0;
const unsigned long AUTO_OFF_DELAY = 30000; // 30 seconds

void setup() {
  Serial.begin(115200);
  pinMode(PIR_PIN, INPUT);

  sprintf(motion_topic, "home/%s/motion", ROOM_ID);
  sprintf(light_command_topic, "home/%s/light/command", ROOM_ID);

  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("\nWi-Fi connected");

  client.setServer(mqtt_server, 1883);
  reconnect();
}

void reconnect() {
  while (!client.connected()) {
    Serial.print("Connecting to MQTT...");
    if (client.connect("ESP32_MotionSensor")) {
      Serial.println(" Connected");
    } else {
      delay(5000);
    }
  }
}

void loop() {
  if (!client.connected()) {
    reconnect();
  }
  client.loop();

  bool motion_detected = digitalRead(PIR_PIN) == HIGH;

  // Motion started
  if (motion_detected && !last_motion_state) {
    Serial.println("Motion detected!");
    client.publish(motion_topic, "true", true);

    // Turn on light
    client.publish(light_command_topic, "ON");
    motion_start_time = millis();
    last_motion_state = true;
  }

  // Motion stopped
  if (!motion_detected && last_motion_state) {
    Serial.println("   Motion cleared");
    client.publish(motion_topic, "false", true);
    last_motion_state = false;
  }

  // Auto turn off light after delay
  if (!motion_detected && (millis() - motion_start_time > AUTO_OFF_DELAY)) {
    client.publish(light_command_topic, "OFF");
    Serial.println("Auto turning off light");
    motion_start_time = millis() + 1000000; // Prevent repeated commands
  }

  delay(200);
}

Code for Light Control (ESP32 #2):

#include <WiFi.h>
#include <PubSubClient.h>

const char* ssid = "YOUR_WIFI_SSID";
const char* password = "YOUR_WIFI_PASSWORD";
const char* mqtt_server = "test.mosquitto.org";

WiFiClient espClient;
PubSubClient client(espClient);

#define LED_PIN 2
#define ROOM_ID "living_room"

char light_command_topic[50];
char light_state_topic[50];

void mqtt_callback(char* topic, byte* payload, unsigned int length) {
  String message = "";
  for (int i = 0; i < length; i++) {
    message += (char)payload[i];
  }

  Serial.print("Received command: ");
  Serial.println(message);

  if (message == "ON") {
    digitalWrite(LED_PIN, HIGH);
    client.publish(light_state_topic, "ON", true);
    Serial.println("Light turned ON");
  } else if (message == "OFF") {
    digitalWrite(LED_PIN, LOW);
    client.publish(light_state_topic, "OFF", true);
    Serial.println("Light turned OFF");
  }
}

void setup() {
  Serial.begin(115200);
  pinMode(LED_PIN, OUTPUT);
  digitalWrite(LED_PIN, LOW);

  sprintf(light_command_topic, "home/%s/light/command", ROOM_ID);
  sprintf(light_state_topic, "home/%s/light/state", ROOM_ID);

  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("\nWi-Fi connected");

  client.setServer(mqtt_server, 1883);
  client.setCallback(mqtt_callback);

  reconnect();
}

void reconnect() {
  while (!client.connected()) {
    Serial.print("Connecting to MQTT...");
    if (client.connect("ESP32_LightControl")) {
      Serial.println(" Connected");

      // Subscribe to light commands
      client.subscribe(light_command_topic);
      Serial.print("Subscribed to: ");
      Serial.println(light_command_topic);

      // Publish initial state
      client.publish(light_state_topic, "OFF", true);
    } else {
      delay(5000);
    }
  }
}

void loop() {
  if (!client.connected()) {
    reconnect();
  }
  client.loop();
}

Expected Serial Output (Motion Sensor):

Wi-Fi connected
Connecting to MQTT... Connected
Motion detected!
   Motion cleared
Auto turning off light

Expected Serial Output (Light Control):

Wi-Fi connected
Connecting to MQTT... Connected
Subscribed to: home/living_room/light/command
Received command: ON
Light turned ON
Received command: OFF
Light turned OFF
TipInteractive Simulator: MQTT Subscriber (Light Control)

What This Simulates: ESP32 subscribing to MQTT commands and controlling an LED based on received messages - the other half of publish-subscribe.

NoteKey Points

MQTT Subscription Flow: 1. ESP32 subscribes to home/living_room/light/command 2. Broker forwards matching messages to this client 3. mqtt_callback() function executes when message arrives 4. LED state changes based on “ON”/“OFF” payload 5. ESP32 publishes new state to home/living_room/light/state (retained)

Real-World: Smart home lights, automated curtains, door locks, HVAC controls

Experiment: Add dimming levels (0-100), multiple rooms, motion sensor integration

Learning Outcomes: - Build multi-device MQTT communication - Implement automation logic with sensors and actuators - Use retained messages for state synchronization - Create topic naming conventions for home automation - Handle timing and auto-off functionality

Challenges: 1. Add manual control via MQTT (smartphone app or Node-RED) 2. Implement brightness control with PWM 3. Add multiple rooms with independent control 4. Create schedules (morning/evening modes)


1205.6 Lab 4: Secure MQTT with TLS and Authentication

Objective: Implement MQTT security using TLS encryption and username/password authentication.

Materials: - ESP32 or Python client - Private MQTT broker (Mosquitto installed locally or on Raspberry Pi) - SSL/TLS certificates

Step 1: Install and Configure Mosquitto Broker

Install Mosquitto on Linux/Raspberry Pi:

sudo apt update
sudo apt install mosquitto mosquitto-clients

Step 2: Generate SSL/TLS Certificates

# Create certificate directory
sudo mkdir -p /etc/mosquitto/certs
cd /etc/mosquitto/certs

# Generate CA certificate
sudo openssl req -new -x509 -days 365 -extensions v3_ca \
  -keyout ca.key -out ca.crt \
  -subj "/C=US/ST=State/L=City/O=IoTClass/CN=CA"

# Generate server key and certificate
sudo openssl genrsa -out server.key 2048
sudo openssl req -new -out server.csr -key server.key \
  -subj "/C=US/ST=State/L=City/O=IoTClass/CN=mqtt.local"
sudo openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out server.crt -days 365

# Set permissions
sudo chmod 644 /etc/mosquitto/certs/*.crt
sudo chmod 600 /etc/mosquitto/certs/*.key

Step 3: Configure Mosquitto with Security

Edit /etc/mosquitto/mosquitto.conf:

# Default listener (disabled)
listener 1883
allow_anonymous false

# TLS listener
listener 8883
cafile /etc/mosquitto/certs/ca.crt
certfile /etc/mosquitto/certs/server.crt
keyfile /etc/mosquitto/certs/server.key
require_certificate false

# Password file
password_file /etc/mosquitto/passwd

# Logging
log_dest file /var/log/mosquitto/mosquitto.log
log_type all

Step 4: Create User Accounts

# Create password file with first user
sudo mosquitto_passwd -c /etc/mosquitto/passwd iotuser

# Add more users (without -c flag)
sudo mosquitto_passwd /etc/mosquitto/passwd admin

Step 5: Restart Mosquitto

sudo systemctl restart mosquitto
sudo systemctl status mosquitto

Graph diagram

Graph diagram
Figure 1205.2: MQTT security architecture with four defense layers: TLS encryption prevents eavesdropping, authentication verifies client identity, ACLs enforce topic permissions, and monitoring detects anomalous behavior, blocking four common attack vectors.

Testing Connection Security:

# Test with mosquitto_pub (with authentication)
mosquitto_pub -h mqtt.local -p 8883 \
  -u iotuser -P your_password \
  --cafile /etc/mosquitto/certs/ca.crt \
  -t "secure/test" -m "Hello from terminal" -q 1

# Test without authentication (should fail)
mosquitto_pub -h mqtt.local -p 8883 \
  --cafile /etc/mosquitto/certs/ca.crt \
  -t "secure/test" -m "This will fail"

Learning Outcomes: - Generate SSL/TLS certificates for MQTT - Configure Mosquitto broker with security - Implement username/password authentication - Use encrypted MQTT connections - Understand certificate-based security - Test and troubleshoot secure MQTT

Security Best Practices: 1. Always use TLS in production (port 8883) 2. Disable anonymous access 3. Use strong passwords (12+ characters) 4. Implement topic-level ACLs (Access Control Lists) 5. Keep certificates updated (renew before expiration) 6. Monitor broker logs for suspicious activity 7. Use client certificates for enhanced security (mutual TLS)

1205.7 Interactive Simulator: MQTT QoS Levels and Last Will Testament

What This Simulates: ESP32 demonstrating MQTT Quality of Service levels and Last Will and Testament for reliable communication

QoS Levels Explained:

QoS 0 (At Most Once):          QoS 1 (At Least Once):      QoS 2 (Exactly Once):
Publisher -> Message ->         Publisher -> Message ->      Publisher -> Message ->
Broker -> Subscriber            Broker -> Subscriber         Broker -> PUBREC <-
                                Broker <- PUBACK             Publisher -> PUBREL ->
Fast, no guarantee              May duplicate                Broker -> PUBCOMP <-
                                Acknowledged                 Guaranteed once

Use for: sensor readings        Use for: commands            Use for: billing data
Power: Lowest                   Power: Medium                Power: Highest
Latency: ~5ms                   Latency: ~15ms               Latency: ~30ms

Last Will and Testament (LWT):

Device connects with LWT set:
{
  topic: "devices/sensor01/status",
  message: "offline",
  qos: 1,
  retain: true
}

Normal flow:                    Unexpected disconnect:
1. Device connects              1. Device crashes
2. Publishes "online"           2. Network timeout (60s)
3. Sends data periodically      3. Broker detects disconnect
4. Publishes "offline"          4. Broker publishes LWT:
5. Disconnects gracefully          "devices/sensor01/status" -> "offline"
                                5. Monitoring system alerted

Result: Device status always known, even after crash

How to Use: 1. Click the Play button to start simulation 2. Watch LWT being set during connection 3. Observe messages sent with different QoS levels 4. See PUBACK responses for QoS 1 5. Monitor retained status messages

NoteLearning Points

What You’ll Observe:

  1. QoS 0 - Fire-and-forget, fastest, no acknowledgment
  2. QoS 1 - Acknowledged delivery, may duplicate
  3. QoS 2 - Guaranteed exactly-once delivery
  4. Last Will - Automatic status update on unexpected disconnect
  5. Retained Messages - Last status available to new subscribers

QoS Level Comparison:

Metric          QoS 0    QoS 1     QoS 2
Messages/sec    100      60        30
Battery life    10 days  7 days    4 days
Guarantee       None     At least  Exactly once
Duplicates      No       Possible  Never
Bandwidth       4 bytes  8 bytes   12 bytes
Use case        Sensors  Commands  Billing

Power consumption per message:
QoS 0: ~10 mA for 50 ms  = 0.14 mAh
QoS 1: ~10 mA for 150 ms = 0.42 mAh (3x)
QoS 2: ~10 mA for 300 ms = 0.83 mAh (6x)

Last Will and Testament Flow:

  1. CONNECT packet with LWT fields, for example:
    • Client ID: sensor01
    • LWT topic: devices/sensor01/lwt
    • LWT message: "offline"
    • LWT QoS: 1
    • LWT retain flag: true
    • Keep-alive: 60 seconds
  2. The broker stores the LWT but does not publish it yet.
  3. If the client disconnects gracefully, the LWT is discarded. If the client times out or crashes, the broker publishes the LWT message.
  4. Monitoring clients subscribed to devices/+/lwt receive the notification and can alert an operator or trigger remediation.

Real-World Applications:

  1. Healthcare - Patient monitor disconnection alerts (QoS 2)
  2. Industrial - Equipment status tracking with LWT
  3. Smart Home - Door/window sensor states (QoS 1, retained)
  4. Fleet Management - Vehicle tracking with offline detection
  5. Agriculture - Irrigation system failure notification

Experiments to Try:

  1. Compare QoS - Send 100 messages with each QoS, measure time
  2. Network Interruption - Disconnect Wi-Fi, observe LWT trigger
  3. Retained Status - New subscriber receives last status immediately
  4. Duplicate Detection - QoS 1 may deliver twice, handle with message IDs
  5. Battery Impact - Calculate battery life with different QoS levels

Choosing QoS Level:

Sensor Type             QoS    Why
Temperature (periodic)   0     Frequent updates, next reading soon
Door sensor (events)     1     Important state changes
Payment processing       2     Financial accuracy required
Heartbeat/ping          0     Frequent, loss acceptable
Firmware update         1     Must arrive, duplication OK
Configuration change    2     Must apply exactly once

Rule of thumb:
- Telemetry: QoS 0 (frequent, replaceable)
- Commands: QoS 1 (important, idempotent)
- Transactions: QoS 2 (critical, non-idempotent)

LWT Best Practices:

DO:
- Set LWT on critical devices
- Use retain=true for status topics
- Set QoS 1 for LWT
- Include timestamp in LWT message
- Subscribe to +/status for monitoring

DON'T:
- Use large LWT messages (keep <100 bytes)
- Set very short keep-alive (<30s)
- Forget to publish "online" after connect
- Use QoS 0 for LWT (unreliable)

Example LWT message:
{
  "status": "offline",
  "reason": "unexpected",
  "timestamp": 1698765432,
  "last_seen": "2024-10-28T10:23:52Z"
}
CautionPitfall: Stale Retained Messages After Device Removal

The Mistake: Developers use retained messages for device status (e.g., devices/sensor42/status with retain=true), but when devices are decommissioned or replaced, the old retained messages remain on the broker indefinitely. New subscribers receive stale “online” status for devices that no longer exist.

Why It Happens: Retained messages persist until explicitly cleared with an empty payload. Most developers focus on the “publish” side and forget that retained messages require lifecycle management. Brokers like Mosquitto will keep retained messages forever unless configured otherwise.

The Fix: Implement device decommissioning that clears retained messages. Publish an empty payload (zero-length) with retain=true to delete the retained message. For MQTT 5.0, use Message Expiry Interval to auto-expire stale status.

# MQTT 3.1.1: Clear retained message on device removal
def decommission_device(client, device_id):
    # Publish empty payload with retain=true to clear
    client.publish(f"devices/{device_id}/status", payload="", qos=1, retain=True)
    client.publish(f"devices/{device_id}/config", payload="", qos=1, retain=True)
    client.publish(f"devices/{device_id}/lwt", payload="", qos=1, retain=True)
    print(f"Cleared all retained messages for {device_id}")

# MQTT 5.0: Auto-expire status messages
from paho.mqtt.properties import Properties
from paho.mqtt.packettypes import PacketTypes

props = Properties(PacketTypes.PUBLISH)
props.MessageExpiryInterval = 300  # Expire after 5 minutes of no update

client.publish(
    "devices/sensor01/status",
    payload='{"status": "online", "timestamp": 1699123456}',
    qos=1,
    retain=True,
    properties=props
)
# If device stops publishing, status auto-clears after 5 minutes

Broker Configuration (Mosquitto): Set retain_available false to disable retained messages entirely if your application doesn’t need them, or use $SYS/broker/retained messages/count to monitor accumulation.

CautionPitfall: Last Will Triggered on Graceful Disconnect

The Mistake: Developers configure Last Will and Testament (LWT) to publish “offline” status, expecting it only fires on crashes or network failures. But they observe LWT messages being published even during normal shutdown sequences, flooding monitoring systems with false “offline” alerts.

Why It Happens: The MQTT spec states LWT is published when the broker closes the connection “without receiving a DISCONNECT packet.” Many developers call client.disconnect() but don’t wait for completion, or the TCP connection closes before the DISCONNECT packet is sent. Network issues during graceful shutdown can also prevent DISCONNECT delivery.

The Fix: Always explicitly publish “offline” status before disconnecting, then send DISCONNECT. Use blocking disconnect or wait for confirmation. For Python paho-mqtt, use disconnect() followed by loop_stop() in the correct order.

// ESP32: Graceful shutdown with explicit status
void gracefulShutdown() {
    // Step 1: Publish explicit offline status
    mqttClient.publish("devices/sensor01/status", "offline", true);  // retain=true
    delay(100);  // Allow time for publish to complete

    // Step 2: Send DISCONNECT packet (prevents LWT)
    mqttClient.disconnect();
    delay(100);  // Allow TCP to close cleanly

    // Step 3: Now safe to power down
    WiFi.disconnect(true);
    esp_deep_sleep_start();
}

// If LWT fires anyway, it's redundant (already sent "offline")
// Better than missing offline status if crash occurs before explicit publish
# Python paho-mqtt 2.0+: Proper disconnect sequence
import time

def graceful_disconnect(client):
    # Publish explicit status first
    result = client.publish("devices/sensor01/status", "offline", qos=1, retain=True)
    result.wait_for_publish(timeout=5.0)  # Block until published

    # Now disconnect - this prevents LWT from firing
    client.disconnect()
    client.loop_stop()  # Stop network thread AFTER disconnect

# Common mistake: loop_stop() before disconnect()
# This kills the network thread before DISCONNECT packet is sent
# Result: Broker never receives DISCONNECT, triggers LWT

MQTT 5.0 Enhancement: Use Reason Code in DISCONNECT to tell broker why you’re disconnecting. Reason Code 0x00 (Normal disconnection) explicitly signals “don’t publish LWT.”

Battery Life Calculation:

Scenario: Send temperature every 60 seconds

QoS 0:
- Assumptions: ~120 mA radio current, ~50 ms radio-on time per publish
- Per message: 120 mA x 50 ms / 3,600,000 = 0.0017 mAh
- Per day: 1440 x 0.0017 = 2.4 mAh/day
- Battery (2000 mAh): 2000 / 2.4 = ~833 days (~2.3 years)

QoS 1:
- Assumptions: ~120 mA radio current, ~150 ms radio-on time (PUBLISH + PUBACK)
- Per message: 120 mA x 150 ms / 3,600,000 = 0.0050 mAh
- Per day: 1440 x 0.0050 = 7.2 mAh/day
- Battery (2000 mAh): 2000 / 7.2 = ~278 days (~9 months)

QoS 2:
- Assumptions: ~120 mA radio current, ~300 ms radio-on time (4-way handshake)
- Per message: 120 mA x 300 ms / 3,600,000 = 0.0100 mAh
- Per day: 1440 x 0.0100 = 14.4 mAh/day
- Battery (2000 mAh): 2000 / 14.4 = ~139 days (~4.6 months)

Conclusion: QoS 0 uses ~67% less TX energy than QoS 1; QoS 2 has the highest overhead.

Note: These estimates isolate MQTT exchange time. If the device reconnects Wi-Fi/TLS each minute (common with deep sleep), connection overhead will dominate and battery life will be much lower.

1205.8 Knowledge Check

Test your understanding of MQTT implementations with these questions:

Question: You’re building a production monitoring system for a data center. Which MQTT broker choice is most appropriate?

Explanation: C. Production deployments typically require authentication, TLS, ACLs, availability guarantees, and operational monitoring - public test brokers don’t provide these.

Question: Which MQTT QoS level is best for a security-critical “UNLOCK” command where duplicates are unacceptable?

Explanation: C. QoS 2 provides exactly-once delivery semantics, reducing the risk of duplicates for non-idempotent commands.

Question: Your ESP32 loses Wi-Fi and is disconnected from the broker. What happens to messages it tries to publish during the disconnection?

Explanation: B. If the client isn’t connected, PUBLISH can’t reach the broker; reliable publishing requires local buffering plus retry on reconnect.

Question: In paho-mqtt, what’s a correct way to receive MQTT messages continuously for a subscriber-only dashboard?

Explanation: B. The network loop must run continuously to process incoming packets and invoke callbacks; loop_forever() is simplest for dedicated subscribers.

1205.9 Summary

This chapter provided complete hands-on MQTT implementation labs:

  • Lab 1 - ESP32 Publisher: Built a DHT22 temperature sensor system with QoS 0 publishing, retained status messages, and automatic reconnection
  • Lab 2 - Python Dashboard: Created multi-topic subscription with wildcards, real-time display, and alert thresholds
  • Lab 3 - Home Automation: Implemented motion-controlled lighting using two ESP32 devices communicating via MQTT
  • Lab 4 - Secure MQTT: Configured TLS certificates, username/password authentication, and ACLs for production deployment
  • QoS and LWT: Demonstrated Quality of Service levels (0/1/2) with battery impact calculations and Last Will Testament for disconnect detection
  • Common Pitfalls: Identified stale retained messages, LWT timing issues, and TLS timeout problems with solutions

1205.10 What’s Next

Continue with MQTT Comprehensive Review to synthesize knowledge from all MQTT chapters with scenario-based questions, protocol comparisons (MQTT vs CoAP vs HTTP), production deployment architectures, and advanced topics like broker clustering and load balancing.