11  Lab: MQTT Hands-On

mqtt
impl
hands
on
In 60 Seconds

These hands-on labs cover building complete MQTT systems with real hardware: ESP32 DHT22 temperature publishers with configurable QoS levels, multi-device motion-to-lighting automation pipelines, secure broker deployment with TLS and ACLs, and Last Will Testament (LWT) for automatic device disconnection detection.

Phoebe the physics guide

Phoebe’s Why

This chapter is already suspicious of its own always-on QoS0 estimate – 833 days from 2.4 mAh/day – and rightly points to Wi-Fi reconnect overhead as the reason the deep-sleep model (76.8 mAh/day, about 26 days) is more honest. There is a second, independent reason to distrust “833 days \(\approx\) 2.3 years”: that arithmetic divides a fixed 2000 mAh by a fixed daily current forever, as if the cell were a bucket that only empties from the bottom. A real lithium cell also empties itself slowly just sitting on the shelf, and over 2.3 years that self-discharge is not a rounding error – it competes with the load current for the same milliamp-hours.

The Derivation

Charge-budget life model:

\[t_{life}=\frac{C_{batt}}{I_{avg}}\]

Self-discharge, compounding over elapsed time \(t\) at monthly rate \(r\):

\[Q(t)=Q_0(1-r)^{t}\]

Terminal voltage under a radio pulse current \(I\):

\[V_{term}=V_{oc}-I\,R_{int}\]

Worked Numbers: This Chapter’s Own QoS Scenarios

  • Deep-sleep life (this chapter’s own 76.8 mAh/day, 2000 mAh pack): \(2000/76.8=26.0\) days – short enough that self-discharge barely registers: at catalog-typical LiPo \(r\approx2\%\)/month, 26 days is 0.856 months, leaving \((0.98)^{0.856}=98.3\%\) of nameplate, a 1.71% loss.
  • The always-on QoS0 claim (this chapter’s own 2.4 mAh/day, 833 days): that duration is 27.4 months, and compounding the same 2%/month gives \((0.98)^{27.4}=57.5\%\) of nameplate remaining – a 42.5% self-discharge loss over the claimed life, before the QoS0 radio current has drawn one extra milliamp-hour for reconnects. The naive \(C/I\) division and this chapter’s own Wi-Fi-overhead caveat are both right to distrust “833 days,” for two separate and additive reasons.
  • Voltage sag on this chapter’s own 120 mA radio pulse (catalog-typical 2000 mAh pack, \(R_{int}\approx0.10\,\Omega\) fresh, \(\approx0.40\,\Omega\) aged): 12.0 mV fresh, 48.0 mV aged – both small against 3.7 V, so for this pack size the sag risk stays minor even as self-discharge does not.

11.1 Start With One Working Pipeline

Every lab is a version of the same review path: a device senses something, publishes it to a topic, the broker applies a delivery rule, and another client proves it received the right state. Build that small path first, then make it secure, reliable, battery-aware, and observable.

Chapter Roadmap

This is a long lab chapter, so use it as a sequence of acceptance records:

  1. First build one ESP32 publisher path and prove the topic, retained status, and publish interval behave as expected.
  2. Then expand the path into a dashboard and a two-device motion-to-light automation flow.
  3. Next harden the broker with TLS, authentication, and topic permissions before testing QoS and Last Will behavior.
  4. Finally use the calculators and quizzes to turn lab observations into capacity, cost, battery, and review evidence.

Checkpoints recap what you have already proved, and sections titled “Deep dive” or calculator notes can be treated as optional support on a first pass.

11.2 Learning Objectives

By the end of this chapter, you will be able to:

  • Construct ESP32 MQTT Publishers: Implement complete sensor-to-broker data pipelines with DHT22 temperature sensors using the PubSubClient library
  • Design Multi-Device Automation Systems: Configure motion sensors and lighting actuators to communicate through an MQTT broker backbone
  • Select and Justify QoS Levels: Evaluate QoS 0, 1, and 2 trade-offs and select the appropriate level based on reliability requirements and power constraints
  • Configure Secure MQTT Brokers: Deploy Mosquitto with TLS certificates, username/password authentication, and ACLs for production-ready operation
  • Implement Last Will Testament: Configure LWT messages to automatically detect and report unexpected device disconnections
  • Calculate Battery Life Impact: Analyze the energy cost of different QoS levels and compare always-on versus deep-sleep publishing strategies
Key Concepts
  • MQTT: Message Queuing Telemetry Transport — pub/sub protocol optimized for constrained IoT devices over unreliable networks
  • Broker: Central server routing messages from publishers to all matching subscribers by topic pattern
  • Topic: Hierarchical string (e.g., home/bedroom/temperature) used to route messages to interested subscribers
  • QoS Level: Quality of Service 0/1/2 trading delivery guarantee for message overhead
  • Retained Message: Last message on a topic stored by broker for immediate delivery to new subscribers
  • Last Will and Testament: Pre-configured message published by broker when a client disconnects ungracefully
  • Persistent Session: Broker stores subscriptions and pending messages allowing clients to resume after disconnection

11.3 For Beginners: MQTT Hands-On Labs

These labs guide you through building real MQTT systems step by step. You will set up a broker, connect devices, publish sensor data, and subscribe to updates. It is like assembling furniture with instructions – each step builds on the previous one until you have a complete, working MQTT application.

11.4 Prerequisites

Before diving into this chapter, you should be familiar with:

11.5 Lab 1: ESP32 DHT22 MQTT Publisher with QoS Levels

Objective: Build a temperature/humidity sensor that publishes to MQTT with different QoS levels.

Materials:

  • ESP32 development board
  • DHT22 temperature/humidity sensor
  • 10k ohm pull-up resistor
  • Breadboard and jumper wires
  • Wi-Fi connection

Circuit Diagram:

DHT22          ESP32
-----          -----
VCC   ------>  3.3V
DATA  ------>  GPIO 4 (with 10k ohm pull-up to 3.3V)
GND   ------>  GND

Complete Code:

#include <WiFi.h>
#include <PubSubClient.h>
#include <DHT.h>

// Wi-Fi credentials
const char* ssid = "YOUR_WIFI_SSID";
const char* password = "YOUR_WIFI_PASSWORD";

// MQTT Broker settings
const char* mqtt_server = "test.mosquitto.org";
const int mqtt_port = 1883;
const char* mqtt_client_id = "ESP32_DHT22_001";

// DHT22 sensor
#define DHTPIN 4
#define DHTTYPE DHT22
DHT dht(DHTPIN, DHTTYPE);

// MQTT topics
const char* topic_temp = "iotclass/lab1/temperature";
const char* topic_humidity = "iotclass/lab1/humidity";
const char* topic_status = "iotclass/lab1/status";

WiFiClient espClient;
PubSubClient client(espClient);

// Forward declaration (defined below loop)
void mqtt_callback(char* topic, byte* payload, unsigned int length);

unsigned long lastPublish = 0;
const long publishInterval = 5000; // 5 seconds

void setup_wifi() {
  Serial.println("\nConnecting to Wi-Fi...");
  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("\nWi-Fi connected");
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());
}

void reconnect_mqtt() {
  while (!client.connected()) {
    Serial.print("Connecting to MQTT broker...");

    if (client.connect(mqtt_client_id)) {
      Serial.println(" Connected");

      // Publish online status as retained message
      client.publish(topic_status, "online", true);

      // Subscribe to commands (optional)
      client.subscribe("iotclass/lab1/command");
    } else {
      Serial.print(" Failed, rc=");
      Serial.println(client.state());
      delay(5000);
    }
  }
}

void setup() {
  Serial.begin(115200);
  dht.begin();

  setup_wifi();

  client.setServer(mqtt_server, mqtt_port);
  client.setCallback(mqtt_callback);
}

void mqtt_callback(char* topic, byte* payload, unsigned int length) {
  Serial.print("Message received on ");
  Serial.print(topic);
  Serial.print(": ");

  String message = "";
  for (int i = 0; i < length; i++) {
    message += (char)payload[i];
  }
  Serial.println(message);
}

void loop() {
  if (!client.connected()) {
    reconnect_mqtt();
  }
  client.loop();

  unsigned long now = millis();
  if (now - lastPublish >= publishInterval) {
    lastPublish = now;

    // Read sensor
    float humidity = dht.readHumidity();
    float temperature = dht.readTemperature();

    if (isnan(humidity) || isnan(temperature)) {
      Serial.println("Failed to read from DHT sensor!");
      return;
    }

    // Publish temperature with QoS 0
    char tempStr[8];
    dtostrf(temperature, 6, 2, tempStr);
    bool temp_success = client.publish(topic_temp, tempStr, false);
    Serial.print("Temperature: ");
    Serial.print(tempStr);
    Serial.print("C ");
    Serial.println(temp_success ? "OK" : "FAIL");

    // Publish humidity with QoS 0
    char humStr[8];
    dtostrf(humidity, 6, 2, humStr);
    bool hum_success = client.publish(topic_humidity, humStr, false);
    Serial.print("Humidity: ");
    Serial.print(humStr);
    Serial.print("% ");
    Serial.println(hum_success ? "OK" : "FAIL");

    Serial.println("---");
  }
}

Expected Output (Serial Monitor):

Connecting to Wi-Fi...
Wi-Fi connected
IP address: 192.168.1.100
Connecting to MQTT broker... Connected
Temperature: 22.50C OK
Humidity: 45.30% OK
***
Temperature: 22.48C OK
Humidity: 45.35% OK
***
Interactive Simulator: MQTT Publisher (ESP32 + DHT22)

Try it yourself! See a complete IoT system publishing sensor data to an MQTT broker.

What This Simulates: An ESP32 with DHT22 sensor connecting to Wi-Fi, then publishing temperature and humidity data to an MQTT broker every 5 seconds.

How to Use:

  1. Click the Play button to start simulation
  2. Watch the Serial Monitor show Wi-Fi connection
  3. Observe MQTT broker connection with client ID
  4. See temperature/humidity published to topics
  5. Notice QoS 0 delivery with confirmation
  6. Open MQTT Explorer or subscriber to see messages in real-time!

Learning Points

Observe:

  • PubSubClient Library: Arduino MQTT client for ESP32
  • client.connect(): Establishes connection to broker with unique client ID
  • client.publish(): Sends message to topic (returns true/false)
  • Topic Hierarchy: iotclass/lab1/temperature uses / separators
  • QoS 0 (At Most Once): Fire-and-forget, fastest but no guarantee
  • Retained Messages: client.publish(topic, msg, true) stores last value for new subscribers
  • Auto-Reconnect: if (!client.connected()) reconnect_mqtt()

MQTT Publishing Flow:

1. ESP32 connects to Wi-Fi network
2. ESP32 connects to MQTT broker (test.mosquitto.org:1883)
3. ESP32 publishes "online" status as retained message
4. Every 5 seconds:
   a. Read DHT22 sensor (temperature, humidity)
   b. Convert float to string
   c. Publish to "iotclass/lab1/temperature" topic
   d. Publish to "iotclass/lab1/humidity" topic
   e. Print success/failure indicators
5. Maintain connection with client.loop()

MQTT Topic Structure:

MQTT topic hierarchy diagram showing rooms, sensor metrics, single-level plus wildcards, multi-level hash wildcards, and wildcard placement rules for lab topic subscriptions.
Figure 11.1: MQTT topic wildcard hierarchy for lab sensor data

Quality of Service Levels:

QoS 0 (At Most Once): Fast, no acknowledgment - used here
QoS 1 (At Least Once): Acknowledged, may deliver duplicates
QoS 2 (Exactly Once): Slowest, guaranteed single delivery

Real-World Applications:

  • Smart Agriculture: Soil moisture sensors publishing to cloud dashboard
  • Industrial Monitoring: Temperature/pressure sensors in manufacturing
  • Home Automation: Smart thermostats publishing temperature readings
  • Environmental Monitoring: Weather stations sending data to aggregation service
  • Asset Tracking: GPS devices publishing location updates

Experiment:

  • Add more sensors (motion, light) and publish to separate topics
  • Implement QoS 1 or 2 to see acknowledgments
  • Add Last Will and Testament message for disconnect detection
  • Publish JSON payload with multiple values: {"temp":22.5,"hum":45.3}
  • Add timestamp to messages for data logging

Learning Outcomes:

  • Configure ESP32 Wi-Fi connection
  • Integrate DHT22 sensor with MQTT
  • Implement publish with QoS levels
  • Handle MQTT reconnection
  • Use retained messages for status

Challenges:

  1. Modify to publish only when temperature changes by +/-0.5C (reduce traffic)
  2. Add battery voltage monitoring and publish with QoS 2
  3. Implement Last Will and Testament to detect unexpected disconnections
  4. Add JSON payload with multiple sensor readings
Broker BexCheckpoint: First Publisher Evidence

You now know:

  • The Lab 1 evidence path starts with a DHT22 on GPIO 4, a 10k ohm pull-up, and iotclass/lab1/... topics.
  • The starter publisher sends temperature and humidity every 5 seconds and publishes retained online status for new subscribers.
  • The first optimization question is whether a +/-0.5C change threshold plus a 5-minute heartbeat is better than timer-only publishing.
Try It: MQTT Topic Hierarchy Builder

Design and visualize your own MQTT topic structure. Choose a domain and configure the hierarchy depth to see how topics scale.

Run it: Before you settle on the hierarchy in the builder below, stress-test a topic design in the Topic Designer Workbench. Enter a Publish Topic Name template and a Subscriber Topic Filter, set the fleet-scale assumptions (Tenants, Sites, Devices, Signals), and press Play to watch which levels the filter matches. Switch the scenario presets to Flat anti-pattern and Wildcard mistake to see the designs that break, then bring a filter that cleanly separates telemetry, commands, and status back into the builder here.

11.5.1 Worked Example: Optimizing Sensor Publish Frequency

Problem: Your DHT22 ESP32 sensors are publishing every 5 seconds (17,280 messages/day per sensor). With 50 sensors, that’s 864,000 messages/day. Assume a cloud message price of $1/million messages, and assume the broker is also hitting CPU limits.

Analysis:

Temperature in a room rarely changes more than 0.5°C per minute. Publishing every 5 seconds means 99% of messages report identical values.

Step 1: Calculate Current Costs

  • Messages/month: 864,000 × 30 = 25.9M messages
  • AWS IoT Core: 25.9 × $1 = $25.90/month
  • Broker CPU: Mosquitto on m5.large = $70/month
  • Total: $95.90/month

Step 2: Design Optimization

Publish only when temperature changes by ≥0.5°C OR 5 minutes elapse (heartbeat):

float lastTemp = 0.0;
unsigned long lastPublish = 0;
const float THRESHOLD = 0.5;  // 0.5°C change triggers publish
const unsigned long HEARTBEAT = 300000;  // 5 min max interval

void loop() {
    float temp = dht.readTemperature();
    unsigned long now = millis();

    // Publish if temp changed OR heartbeat timeout
    if (abs(temp - lastTemp) >= THRESHOLD ||
        (now - lastPublish >= HEARTBEAT)) {

        client.publish(topic_temp, String(temp).c_str());
        lastTemp = temp;
        lastPublish = now;
    }

    delay(5000);  // Still sample every 5s, but don't always publish
}

Step 3: Calculate New Costs

For scenario estimation, assume an office temperature signal changes enough to report about 2-3 times/hour, plus a 5-minute heartbeat:

  • Meaningful changes: ~3/hour = 72/day
  • Heartbeats: (1440 min/day) / 5 = 288/day
  • Total: 360 messages/day per sensor (was 17,280)

For 50 sensors:

  • Messages/month: 360 × 50 × 30 = 540,000 messages
  • Message charge estimate: 0.54 × $1 = $0.54/month
  • Broker headroom may allow a smaller instance, depending on CPU, TLS, retained messages, and monitoring load.
  • Scenario result: scheduled publishing creates roughly 83% lower message-driven cost in this example.

Step 4: Additional Benefits

  • Bandwidth saved: 98% reduction (25.9M → 540K messages/month)
  • Battery life: Publishing less often reduces radio transmit time; the actual gain depends on sleep, Wi-Fi association, and TLS setup overhead
  • Storage costs: Time-series DB costs drop proportionally

Key Insight: Don’t blindly publish on a timer. Publish on change with a heartbeat fallback. This pattern works for most slowly-changing telemetry (temperature, humidity, pressure, battery voltage).

11.5.2 Pitfall: Broker Connection Limits Causing Silent Failures

The Mistake: Developers deploy IoT systems without considering broker connection limits. They test with 10-20 devices successfully, then deploy 500+ devices to production. New devices fail to connect with cryptic errors like “connection refused” or timeout, while existing connections work fine.

Why It Happens: MQTT brokers have configurable maximum connection limits, often defaulting to 1,024 (OS file descriptor limit) or lower. Each MQTT connection consumes a file descriptor, memory for session state (~2-10KB), and a TCP socket. When limits are reached, new connections are silently rejected without clear error messages.

The Fix: Calculate connection requirements before deployment. Configure broker limits explicitly. Implement connection health monitoring and alerting when approaching 80% capacity. Use connection pooling or MQTT bridge patterns for high-scale deployments.

Capacity Planning Formula: Required connections = (devices x 1.2) + (backend_services x 2) + (monitoring x 3). For 1,000 devices with 5 backend services and 2 monitoring tools, plan for: (1000 x 1.2) + (5 x 2) + (2 x 3) = 1,216 connections minimum.

11.6 Lab 2: Python MQTT Dashboard with Multiple Sensors

Objective: Create a real-time dashboard that subscribes to multiple sensor topics and displays data.

Materials:

  • Python 3.7+
  • paho-mqtt library
  • (Optional) Running ESP32 from Lab 1

Expected Output:

IoT MQTT Dashboard - Real-Time Sensor Data

LAB1
Temperature      22.50C              [14:32:15] QoS:0
Humidity         45.30%               [14:32:15] QoS:0
Status           Status: online       [14:32:10] QoS:0

Total messages received: 127
Last update: 14:32:15

Press Ctrl+C to exit

Learning Outcomes:

  • Subscribe to multiple MQTT topics with wildcards
  • Build real-time data visualization
  • Implement alert systems based on sensor data
  • Handle MQTT callbacks and data processing
  • Create user-friendly console interfaces

11.7 Lab 3: MQTT Home Automation - Lights and Motion

Objective: Build a complete home automation system with motion detection and automated lighting control.

Materials:

  • 2x ESP32 boards (one for motion sensor, one for light control)
  • PIR motion sensor (HC-SR501)
  • LED (or relay module for real lights)
  • 220 ohm resistor
  • Breadboard and wires

Circuit 1 - Motion Sensor (ESP32 #1):

PIR Sensor     ESP32
----------     -----
VCC    ------>  5V
OUT    ------>  GPIO 13
GND    ------>  GND

Circuit 2 - Light Control (ESP32 #2):

ESP32         LED
-----         ---
GPIO 2  ----> LED Anode (through 220 ohm resistor)
GND     ----> LED Cathode

Code for Motion Sensor (ESP32 #1):

#include <WiFi.h>
#include <PubSubClient.h>

const char* ssid = "YOUR_WIFI_SSID";
const char* password = "YOUR_WIFI_PASSWORD";
const char* mqtt_server = "test.mosquitto.org";

WiFiClient espClient;
PubSubClient client(espClient);

#define PIR_PIN 13
#define ROOM_ID "living_room"

char motion_topic[50];
char light_command_topic[50];

bool last_motion_state = false;
unsigned long motion_start_time = 0;
const unsigned long AUTO_OFF_DELAY = 30000; // 30 seconds

void setup() {
  Serial.begin(115200);
  pinMode(PIR_PIN, INPUT);

  sprintf(motion_topic, "home/%s/motion", ROOM_ID);
  sprintf(light_command_topic, "home/%s/light/command", ROOM_ID);

  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("\nWi-Fi connected");

  client.setServer(mqtt_server, 1883);
  reconnect();
}

void reconnect() {
  while (!client.connected()) {
    Serial.print("Connecting to MQTT...");
    if (client.connect("ESP32_MotionSensor")) {
      Serial.println(" Connected");
    } else {
      delay(5000);
    }
  }
}

void loop() {
  if (!client.connected()) {
    reconnect();
  }
  client.loop();

  bool motion_detected = digitalRead(PIR_PIN) == HIGH;

  // Motion started
  if (motion_detected && !last_motion_state) {
    Serial.println("Motion detected!");
    client.publish(motion_topic, "true", true);

    // Turn on light
    client.publish(light_command_topic, "ON");
    motion_start_time = millis();
    last_motion_state = true;
  }

  // Motion stopped
  if (!motion_detected && last_motion_state) {
    Serial.println("   Motion cleared");
    client.publish(motion_topic, "false", true);
    last_motion_state = false;
  }

  // Auto turn off light after delay
  if (!motion_detected && (millis() - motion_start_time > AUTO_OFF_DELAY)) {
    client.publish(light_command_topic, "OFF");
    Serial.println("Auto turning off light");
    motion_start_time = millis() + 1000000; // Prevent repeated commands
  }

  delay(200);
}

Code for Light Control (ESP32 #2):

#include <WiFi.h>
#include <PubSubClient.h>

const char* ssid = "YOUR_WIFI_SSID";
const char* password = "YOUR_WIFI_PASSWORD";
const char* mqtt_server = "test.mosquitto.org";

WiFiClient espClient;
PubSubClient client(espClient);

#define LED_PIN 2
#define ROOM_ID "living_room"

char light_command_topic[50];
char light_state_topic[50];

void mqtt_callback(char* topic, byte* payload, unsigned int length) {
  String message = "";
  for (int i = 0; i < length; i++) {
    message += (char)payload[i];
  }

  Serial.print("Received command: ");
  Serial.println(message);

  if (message == "ON") {
    digitalWrite(LED_PIN, HIGH);
    client.publish(light_state_topic, "ON", true);
    Serial.println("Light turned ON");
  } else if (message == "OFF") {
    digitalWrite(LED_PIN, LOW);
    client.publish(light_state_topic, "OFF", true);
    Serial.println("Light turned OFF");
  }
}

void setup() {
  Serial.begin(115200);
  pinMode(LED_PIN, OUTPUT);
  digitalWrite(LED_PIN, LOW);

  sprintf(light_command_topic, "home/%s/light/command", ROOM_ID);
  sprintf(light_state_topic, "home/%s/light/state", ROOM_ID);

  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("\nWi-Fi connected");

  client.setServer(mqtt_server, 1883);
  client.setCallback(mqtt_callback);

  reconnect();
}

void reconnect() {
  while (!client.connected()) {
    Serial.print("Connecting to MQTT...");
    if (client.connect("ESP32_LightControl")) {
      Serial.println(" Connected");

      // Subscribe to light commands
      client.subscribe(light_command_topic);
      Serial.print("Subscribed to: ");
      Serial.println(light_command_topic);

      // Publish initial state
      client.publish(light_state_topic, "OFF", true);
    } else {
      delay(5000);
    }
  }
}

void loop() {
  if (!client.connected()) {
    reconnect();
  }
  client.loop();
}

Expected Serial Output (Motion Sensor):

Wi-Fi connected
Connecting to MQTT... Connected
Motion detected!
   Motion cleared
Auto turning off light

Expected Serial Output (Light Control):

Wi-Fi connected
Connecting to MQTT... Connected
Subscribed to: home/living_room/light/command
Received command: ON
Light turned ON
Received command: OFF
Light turned OFF
Interactive Simulator: MQTT Subscriber (Light Control)

What This Simulates: ESP32 subscribing to MQTT commands and controlling an LED based on received messages - the other half of publish-subscribe.

Key Points

MQTT Subscription Flow:

  1. ESP32 subscribes to home/living_room/light/command
  2. Broker forwards matching messages to this client
  3. mqtt_callback() function executes when message arrives
  4. LED state changes based on “ON”/“OFF” payload
  5. ESP32 publishes new state to home/living_room/light/state (retained)

Real-World: Smart home lights, automated curtains, door locks, HVAC controls

Experiment: Add dimming levels (0-100), multiple rooms, motion sensor integration

Learning Outcomes:

  • Build multi-device MQTT communication
  • Implement automation logic with sensors and actuators
  • Use retained messages for state synchronization
  • Create topic naming conventions for home automation
  • Handle timing and auto-off functionality

Challenges:

  1. Add manual control via MQTT (smartphone app or Node-RED)
  2. Implement brightness control with PWM
  3. Add multiple rooms with independent control
  4. Create schedules (morning/evening modes)
Broker BexCheckpoint: Multi-Device Automation

You now know:

  • The home automation lab splits motion sensing on GPIO 13 from light control on GPIO 2, so publish and subscribe roles are visible on separate ESP32 boards.
  • The topic plan keeps motion state, light commands, and light state separate instead of mixing sensor evidence with actuator commands.
  • The default AUTO_OFF_DELAY is 30 seconds, so PIR hold time, polling rate, and command timing all become part of the acceptance record.
Try It: Motion-to-Light Automation Timing

Experiment with motion sensor timing parameters to understand how AUTO_OFF_DELAY, PIR hold time, and sensor polling rate affect automation behavior and power consumption.

11.8 Lab 4: Secure MQTT with TLS and Authentication

Objective: Implement MQTT security using TLS encryption and username/password authentication.

Materials:

  • ESP32 or Python client
  • Private MQTT broker (Mosquitto installed locally or on Raspberry Pi)
  • SSL/TLS certificates

Step 1: Install and Configure Mosquitto Broker

Install Mosquitto on Linux/Raspberry Pi:

sudo apt update
sudo apt install mosquitto mosquitto-clients

Step 2: Generate SSL/TLS Certificates

# Create certificate directory
sudo mkdir -p /etc/mosquitto/certs
cd /etc/mosquitto/certs

# Generate CA certificate
sudo openssl req -new -x509 -days 365 -extensions v3_ca \
  -keyout ca.key -out ca.crt \
  -subj "/C=US/ST=State/L=City/O=IoTClass/CN=CA"

# Generate server key and certificate
sudo openssl genrsa -out server.key 2048
sudo openssl req -new -out server.csr -key server.key \
  -subj "/C=US/ST=State/L=City/O=IoTClass/CN=mqtt.local"
sudo openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out server.crt -days 365

# Set permissions
sudo chmod 644 /etc/mosquitto/certs/*.crt
sudo chmod 600 /etc/mosquitto/certs/*.key

Step 3: Configure Mosquitto with Security

Edit /etc/mosquitto/mosquitto.conf:

# Default listener (disabled)
listener 1883
allow_anonymous false

# TLS listener
listener 8883
cafile /etc/mosquitto/certs/ca.crt
certfile /etc/mosquitto/certs/server.crt
keyfile /etc/mosquitto/certs/server.key
require_certificate false

# Password file
password_file /etc/mosquitto/passwd

# Logging
log_dest file /var/log/mosquitto/mosquitto.log
log_type all

Step 4: Create User Accounts

# Create password file with first user
sudo mosquitto_passwd -c /etc/mosquitto/passwd iotuser

# Add more users (without -c flag)
sudo mosquitto_passwd /etc/mosquitto/passwd admin

Step 5: Restart Mosquitto

sudo systemctl restart mosquitto
sudo systemctl status mosquitto

MQTT publish/subscribe diagram with an IoT sensor publisher, secured broker, dashboard and mobile subscribers, topic paths, and security layers for TLS transport, authentication, and authorization.

MQTT publish/subscribe flow with broker security layers
Figure 11.2: MQTT security architecture layers TLS transport, authentication, and topic-level authorization around the broker that routes messages from publishers to subscribers.

Testing Connection Security:

# Test with mosquitto_pub (with authentication)
mosquitto_pub -h mqtt.local -p 8883 \
  -u iotuser -P your_password \
  --cafile /etc/mosquitto/certs/ca.crt \
  -t "secure/test" -m "Hello from terminal" -q 1

# Test without authentication (should fail)
mosquitto_pub -h mqtt.local -p 8883 \
  --cafile /etc/mosquitto/certs/ca.crt \
  -t "secure/test" -m "This will fail"

Learning Outcomes:

  • Generate SSL/TLS certificates for MQTT
  • Configure Mosquitto broker with security
  • Implement username/password authentication
  • Use encrypted MQTT connections
  • Understand certificate-based security
  • Test and troubleshoot secure MQTT

Security Best Practices:

  1. Always use TLS in production (port 8883)
  2. Disable anonymous access
  3. Use strong passwords (12+ characters)
  4. Implement topic-level ACLs (Access Control Lists)
  5. Keep certificates updated (renew before expiration)
  6. Monitor broker logs for suspicious activity
  7. Use client certificates for enhanced security (mutual TLS)
Broker BexCheckpoint: Secure Broker Baseline

You now know:

  • The secure lab moves MQTT traffic to TLS port 8883 with CA, server certificate, and server key files.
  • The minimum broker baseline disables anonymous access and adds a password file before production testing.
  • The negative test matters: an unauthenticated mosquitto_pub command should fail before you trust the broker configuration.
Try It: MQTT Security Configuration Advisor

Select your deployment scenario and security requirements to get a tailored Mosquitto configuration with an overall security score.

11.9 Interactive Simulator: MQTT QoS Levels and Last Will Testament

What This Simulates: ESP32 demonstrating MQTT Quality of Service levels and Last Will and Testament for reliable communication

QoS Levels Explained:

QoS 0 (At Most Once):          QoS 1 (At Least Once):      QoS 2 (Exactly Once):
Publisher -> Message ->         Publisher -> Message ->      Publisher -> Message ->
Broker -> Subscriber            Broker -> Subscriber         Broker -> PUBREC <-
                                Broker <- PUBACK             Publisher -> PUBREL ->
Fast, no guarantee              May duplicate                Broker -> PUBCOMP <-
                                Acknowledged                 Guaranteed once

Use for: sensor readings        Use for: commands            Use for: billing data
Power: Lowest                   Power: Medium                Power: Highest
Latency: ~5ms                   Latency: ~15ms               Latency: ~30ms

Run it: Rather than tracing the handshake diagrams above, operate the QoS Delivery Workbench to see delivery guarantees play out. Set the Publish QoS to 0, 1, and 2 in turn, then change the Network event to drop the first PUBLISH, the acknowledgement, or the QoS 2 PUBREL and press Play to watch which messages are lost, retried, or duplicated. Lower the Subscriber maximum QoS to see a downgrade, adjust Messages per minute and Round-trip time, and toggle the idempotent option, then use those observations to answer the QoS choice in the check below.

11.10 Quick Check: QoS Levels

Last Will and Testament (LWT):

Device connects with LWT set:
{
  topic: "devices/sensor01/status",
  message: "offline",
  qos: 1,
  retain: true
}

Normal flow:                    Unexpected disconnect:
1. Device connects              1. Device crashes
2. Publishes "online"           2. Network timeout (60s)
3. Sends data periodically      3. Broker detects disconnect
4. Publishes "offline"          4. Broker publishes LWT:
5. Disconnects gracefully          "devices/sensor01/status" -> "offline"
                                5. Monitoring system alerted

Result: Device status always known, even after crash

How to Use:

  1. Click the Play button to start simulation
  2. Watch LWT being set during connection
  3. Observe messages sent with different QoS levels
  4. See PUBACK responses for QoS 1
  5. Monitor retained status messages

Learning Points

QoS Level Comparison:

  • QoS 0 Guarantee: None Duplicates: No ACK overhead: 0 bytes Use case: Sensors
  • QoS 1 Guarantee: At least once Duplicates: Possible ACK overhead: 4 bytes (PUBACK) Use case: Commands
  • QoS 2 Guarantee: Exactly once Duplicates: Never ACK overhead: 12 bytes (PUBREC + PUBREL + PUBCOMP) Use case: Billing

Last Will and Testament Flow:

  1. CONNECT packet with LWT fields, for example:
    • Client ID: sensor01
    • LWT topic: devices/sensor01/lwt
    • LWT message: "offline"
    • LWT QoS: 1
    • LWT retain flag: true
    • Keep-alive: 60 seconds
  2. The broker stores the LWT but does not publish it yet.
  3. If the client disconnects gracefully, the LWT is discarded. If the client times out or crashes, the broker publishes the LWT message.
  4. Monitoring clients subscribed to devices/+/lwt receive the notification and can alert an operator or trigger remediation.

Real-World Applications:

  1. Healthcare - Patient monitor disconnection alerts (QoS 2)
  2. Industrial - Equipment status tracking with LWT
  3. Smart Home - Door/window sensor states (QoS 1, retained)
  4. Fleet Management - Vehicle tracking with offline detection
  5. Agriculture - Irrigation system failure notification

Experiments to Try:

  1. Compare QoS - Send 100 messages with each QoS, measure time
  2. Network Interruption - Disconnect Wi-Fi, observe LWT trigger
  3. Retained Status - New subscriber receives last status immediately
  4. Duplicate Detection - QoS 1 may deliver twice, handle with message IDs
  5. Battery Impact - Calculate battery life with different QoS levels

Choosing QoS Level (Rule of Thumb):

  • Telemetry (temperature, heartbeat): QoS 0 – frequent, replaceable
  • Commands (door sensor, firmware): QoS 1 – important, idempotent
  • Transactions (payment, config change): QoS 2 – critical, non-idempotent

LWT Best Practices:

DO:
- Set LWT on critical devices
- Use retain=true for status topics
- Set QoS 1 for LWT
- Include timestamp in LWT message
- Subscribe to +/status for monitoring

DON'T:
- Use large LWT messages (keep <100 bytes)
- Set very short keep-alive (<30s)
- Forget to publish "online" after connect
- Use QoS 0 for LWT (unreliable)

Example LWT message:
{
  "status": "offline",
  "reason": "unexpected",
  "timestamp": 1698765432,
  "last_seen": "2024-10-28T10:23:52Z"
}
Try It: MQTT Keep-Alive and LWT Timeout Planner

Configure keep-alive and LWT settings to balance between fast disconnect detection and network overhead. The broker detects a dead client after 1.5x the keep-alive interval (per MQTT spec).

Run it: Before you fix keep-alive and LWT values in the planner below, watch how the broker actually detects a lost client in the Last Will Workbench. Set the Keep Alive interval and a Will Delay, choose the Keep-alive timeout scenario, and press Play or Trigger event to see the broker publish the Will after 1.5x keep-alive; then compare the Graceful disconnect and Reconnect before Will Delay scenarios, which suppress the Will. Carry the detection-time and false-trigger behavior you observe into the keep-alive and QoS choices you record in the planner.

11.10.1 Pitfall: Stale Retained Messages After Device Removal

The Mistake: Developers use retained messages for device status (e.g., devices/sensor42/status with retain=true), but when devices are decommissioned or replaced, the old retained messages remain on the broker indefinitely. New subscribers receive stale “online” status for devices that no longer exist.

Why It Happens: Retained messages persist until explicitly cleared with an empty payload. Most developers focus on the “publish” side and forget that retained messages require lifecycle management. Brokers like Mosquitto will keep retained messages forever unless configured otherwise.

The Fix: Implement device decommissioning that clears retained messages. Publish an empty payload (zero-length) with retain=true to delete the retained message. For MQTT 5.0, use Message Expiry Interval to auto-expire stale status.

# MQTT 3.1.1: Clear retained message on device removal
def decommission_device(client, device_id):
    # Publish empty payload with retain=true to clear
    client.publish(f"devices/{device_id}/status", payload="", qos=1, retain=True)
    client.publish(f"devices/{device_id}/config", payload="", qos=1, retain=True)
    client.publish(f"devices/{device_id}/lwt", payload="", qos=1, retain=True)
    print(f"Cleared all retained messages for {device_id}")

# MQTT 5.0: Auto-expire status messages
from paho.mqtt.properties import Properties
from paho.mqtt.packettypes import PacketTypes

props = Properties(PacketTypes.PUBLISH)
props.MessageExpiryInterval = 300  # Expire after 5 minutes of no update

client.publish(
    "devices/sensor01/status",
    payload='{"status": "online", "timestamp": 1699123456}',
    qos=1,
    retain=True,
    properties=props
)
# If device stops publishing, status auto-clears after 5 minutes

Broker Configuration (Mosquitto): Set retain_available false to disable retained messages entirely if your application doesn’t need them, or use $SYS/broker/retained messages/count to monitor accumulation.

11.10.2 Pitfall: Last Will Triggered on Graceful Disconnect

The Mistake: Developers configure Last Will and Testament (LWT) to publish “offline” status, expecting it only fires on crashes or network failures. But they observe LWT messages being published even during normal shutdown sequences, flooding monitoring systems with false “offline” alerts.

Why It Happens: The MQTT spec states LWT is published when the broker closes the connection “without receiving a DISCONNECT packet.” Many developers call client.disconnect() but don’t wait for completion, or the TCP connection closes before the DISCONNECT packet is sent. Network issues during graceful shutdown can also prevent DISCONNECT delivery.

The Fix: Always explicitly publish “offline” status before disconnecting, then send DISCONNECT. Use blocking disconnect or wait for confirmation. For Python paho-mqtt, use disconnect() followed by loop_stop() in the correct order.

// ESP32: Graceful shutdown with explicit status
void gracefulShutdown() {
    // Step 1: Publish explicit offline status
    mqttClient.publish("devices/sensor01/status", "offline", true);  // retain=true
    delay(100);  // Allow time for publish to complete

    // Step 2: Send DISCONNECT packet (prevents LWT)
    mqttClient.disconnect();
    delay(100);  // Allow TCP to close cleanly

    // Step 3: Now safe to power down
    WiFi.disconnect(true);
    esp_deep_sleep_start();
}

// If LWT fires anyway, it's redundant (already sent "offline")
// Better than missing offline status if crash occurs before explicit publish
# Python paho-mqtt 2.0+: Proper disconnect sequence
import time

def graceful_disconnect(client):
    # Publish explicit status first
    result = client.publish("devices/sensor01/status", "offline", qos=1, retain=True)
    result.wait_for_publish(timeout=5.0)  # Block until published

    # Now disconnect - this prevents LWT from firing
    client.disconnect()
    client.loop_stop()  # Stop network thread AFTER disconnect

# Common mistake: loop_stop() before disconnect()
# This kills the network thread before DISCONNECT packet is sent
# Result: Broker never receives DISCONNECT, triggers LWT

MQTT 5.0 Enhancement: Use Reason Code in DISCONNECT to tell broker why you’re disconnecting. Reason Code 0x00 (Normal disconnection) explicitly signals “don’t publish LWT.”

Battery Life Calculation:

Scenario: Send temperature every 60 seconds

QoS 0:
- Assumptions: ~120 mA radio current, ~50 ms radio-on time per publish
- Per message: 120 mA x 50 ms / 3,600,000 = 0.0017 mAh
- Per day: 1440 x 0.0017 = 2.4 mAh/day
- Battery (2000 mAh): 2000 / 2.4 = ~833 days (~2.3 years)

QoS 1:
- Assumptions: ~120 mA radio current, ~150 ms radio-on time (PUBLISH + PUBACK)
- Per message: 120 mA x 150 ms / 3,600,000 = 0.0050 mAh
- Per day: 1440 x 0.0050 = 7.2 mAh/day
- Battery (2000 mAh): 2000 / 7.2 = ~278 days (~9 months)

QoS 2:
- Assumptions: ~120 mA radio current, ~300 ms radio-on time (4-way handshake)
- Per message: 120 mA x 300 ms / 3,600,000 = 0.0100 mAh
- Per day: 1440 x 0.0100 = 14.4 mAh/day
- Battery (2000 mAh): 2000 / 14.4 = ~139 days (~4.6 months)

Conclusion: QoS 0 uses ~67% less TX energy than QoS 1; QoS 2 has the highest overhead.

Note: These estimates isolate MQTT exchange time. If the device reconnects Wi-Fi/TLS each minute (common with deep sleep), connection overhead will dominate and battery life will be much lower.

11.10.3 Putting Numbers to It

The QoS battery calculations above are a simplified “always-on” radio model. With ESP32 deep sleep, connection setup often dominates:

Deep sleep scenario (ESP32 wakes every 60 s, sends 1 reading, sleeps):

Wi-Fi connection overhead:

  • Scan channels: 200 ms @ 80 mA = 4.44 µAh
  • Associate + DHCP: 1,500 ms @ 100 mA = 41.67 µAh
  • Total connection: 46.11 µAh per wake

MQTT connection (clean session):

  • TCP handshake: 150 ms @ 80 mA = 3.33 µAh
  • CONNECT + CONNACK: 100 ms @ 80 mA = 2.22 µAh
  • Total MQTT setup: 5.55 µAh

Message transmission (QoS 0):

  • PUBLISH: 50 ms @ 120 mA = 1.67 µAh
  • Total message: 1.67 µAh

Per-wake cycle: \[E_{total} = 46.11 + 5.55 + 1.67 = 53.33 \text{ µAh}\]

Daily energy (1,440 wakes): \[E_{day} = 1{,}440 \times 53.33 = 76{,}795 \text{ µAh} = 76.8 \text{ mAh}\]

Battery life (2000 mAh): \[\frac{2{,}000}{76.8} = 26 \text{ days}\]

Key insight: In this example, Wi-Fi setup is much larger than the MQTT publish. Treat persistent sessions, sleep interval, retained status, and TLS setup as scenario variables instead of assuming a fixed battery-life multiplier.

Broker BexCheckpoint: Reliability and Energy Evidence

You now know:

  • QoS choices change both delivery guarantees and packet exchanges: QoS 0 has no acknowledgement, QoS 1 adds PUBACK, and QoS 2 adds PUBREC, PUBREL, and PUBCOMP.
  • Last Will evidence depends on the broker not receiving DISCONNECT; keep-alive detection uses 1.5x the configured interval.
  • In the deep sleep example, Wi-Fi setup, MQTT setup, and one publish add to 53.33 uAh per wake, which is the number to challenge before promising battery life.

11.11 Interactive Design Tools

Use these calculators as scenario estimators for MQTT deployments. Confirm final sizing with measurements from your broker, radio, TLS settings, and cloud pricing.

11.11.1 MQTT QoS Battery Life Calculator

Estimate how MQTT publish exchange time changes battery demand at each QoS level.

11.11.2 MQTT Broker Capacity Planner

Plan your MQTT broker resources based on device count and connection patterns.

11.11.3 MQTT Message Cost Estimator

Estimate monthly MQTT message charges and potential savings from change-based publishing. Update the unit price to match the provider and region you are using.

11.11.4 MQTT Deep Sleep Energy Calculator

Model the full wake cycle energy for ESP32 with deep sleep, including Wi-Fi and MQTT connection overhead.

Broker BexCheckpoint: Calculator Results to Review Evidence

You now know:

  • The battery calculator isolates MQTT publish exchange time, so it should not be treated as a full product battery-life estimate by itself.
  • The broker capacity planner uses the same connection formula as the pitfall section: devices x 1.2, backend services x 2, and monitoring x 3.
  • The cost and deep sleep calculators turn the lab trade-offs into reviewable claims about message volume, heartbeat interval, Wi-Fi setup, and MQTT setup overhead.

11.12 Knowledge Check

Test your understanding of MQTT implementations with these questions:

11.13 Matching Quiz: MQTT Concepts and Definitions

11.14 Ordering Quiz: Secure MQTT Broker Setup

11.15 Auto-Gradable Quick Check

11.16 Label the Diagram

11.17 Code Challenge

11.18 Deep-Dive Note: Presence Evidence Boundaries

Last Will and Testament is strongest when the lab treats presence as evidence, not as a dashboard badge. A useful transcript shows the subscriber already listening, the will-enabled client connecting, the exact ungraceful stop, and the broker publication on the status topic. Record the broker, TLS mode, username or client-certificate method, will topic, retain flag, QoS, keep-alive, and termination method so another engineer can reproduce why the will did or did not fire.

Run this test on a private or local broker when the topic carries device status. Public brokers are useful for syntax checks, but they do not provide stable authorization boundaries, broker log access, or protection from another user publishing to the same topic. In production, scope the status topic by site and device id, protect it with ACLs so only the device or trusted service can write it, and include timestamps so operators can distinguish a fresh offline event from stale retained state.

The under-the-hood failure to rehearse is split-brain presence. A device can publish retained online before its sensors are ready, a long Will Delay Interval can make alarms late, and an aggressive keep-alive can make mobile or cellular devices flap offline during normal roaming. MQTT 5 Will Delay helps short network blips by waiting before publishing the will; retained-message cleanup prevents a decommissioned device or reprovisioned replacement from inheriting another device’s stale marker.

Keep one presence acceptance record: clean DISCONNECT suppresses the will, killed process or network drop fires it after keep-alive detection, late subscribers see the retained state, and decommissioning clears retained status topics.

11.19 Summary

This chapter provided complete hands-on MQTT implementation labs:

  • Lab 1 - ESP32 Publisher: Built a DHT22 temperature sensor system with QoS 0 publishing, retained status messages, and automatic reconnection
  • Lab 2 - Python Dashboard: Created multi-topic subscription with wildcards, real-time display, and alert thresholds
  • Lab 3 - Home Automation: Implemented motion-controlled lighting using two ESP32 devices communicating via MQTT
  • Lab 4 - Secure MQTT: Configured TLS certificates, username/password authentication, and ACLs for production deployment
  • QoS and LWT: Demonstrated Quality of Service levels (0/1/2) with battery impact calculations and Last Will Testament for disconnect detection
  • Common Pitfalls: Identified stale retained messages, LWT timing issues, and TLS timeout problems with solutions

11.20 Concept Relationships

MQTT Hands-On Labs connect to:

Lab progression: Lab 1 (basic publisher) → Lab 2 (Python dashboard) → Lab 3 (multi-device automation) → Lab 4 (secure deployment). Each builds on previous concepts while adding complexity.

11.21 See Also

11.22 What’s Next

  • MQTT Production Review: Scenario-based questions, MQTT vs CoAP vs HTTP comparisons, broker clustering Why read it: Synthesizes all MQTT concepts from hands-on labs into production-ready knowledge.
  • MQTT QoS and MQTT Sessions: QoS 0/1/2 mechanics, clean vs persistent sessions, message queuing Why read it: Deepens understanding of the QoS trade-offs applied throughout these labs.
  • MQTT Security Fundamentals: TLS mutual authentication, certificate management, threat modelling Why read it: Expands on the Lab 4 security configuration with production security design.
  • MQTT Python Patterns: Reconnection logic, async callbacks, error handling, message batching Why read it: Builds production-quality Python clients beyond the dashboard pattern in Lab 2.
  • Sensor Lab Implementation: ESP32 wiring, sensor setup, measurement checks, and lab review Why read it: Supports hardware setup for Labs 1 and 3 and extends to battery-powered designs.
  • MQTT Production Review: Load testing, network simulation, capacity validation Why read it: Validates MQTT broker capacity planning before production deployment.

11.23 Key Takeaway

Hands-on MQTT labs should verify observable broker behavior, not only code execution. Check subscription filters, retained state, Last Will messages, QoS acknowledgements, and reconnection behavior in the broker logs or dashboard.