6 MQTT Session Management

The Mistake: Developers configure clean_session=false on publishing devices (sensors), expecting the broker to buffer their outbound messages when the network is down.

Why It Happens: The term “persistent session” suggests messages are persisted in both directions. Developers assume that if subscribers get queued messages, publishers should too. The MQTT spec is clear but often misread: persistent sessions only queue messages to clients, not from them.

The Fix: Implement local message buffering on the publisher side. When mqttClient.connected() returns false, store messages locally (SPIFFS, SD card, or RAM buffer) and publish them on reconnection.

// ESP32 local buffering pattern
#define BUFFER_SIZE 100
struct BufferedMessage {
  char topic[64];
  char payload[256];
  uint8_t qos;
};
BufferedMessage buffer[BUFFER_SIZE];
int bufferIndex = 0;

void publishWithBuffer(const char* topic, const char* payload, uint8_t qos) {
  if (mqttClient.connected()) {
    // Flush buffer first
    for (int i = 0; i < bufferIndex; i++) {
      mqttClient.publish(buffer[i].topic, buffer[i].payload, buffer[i].qos);
    }
    bufferIndex = 0;
    // Then publish current message
    mqttClient.publish(topic, payload, qos);
  } else if (bufferIndex < BUFFER_SIZE) {
    // Store locally when offline
    strncpy(buffer[bufferIndex].topic, topic, 63);
    strncpy(buffer[bufferIndex].payload, payload, 255);
    buffer[bufferIndex].qos = qos;
    bufferIndex++;
  }
}

MQTT 3.1.1 Spec Reference: Section 3.1.2.4 states “If CleanSession is set to 0, the Server MUST resume communications with the Client based on state from the current Session.” The “state” includes subscriptions and inflight messages to the client, not messages the client wants to send.

The Mistake: Developers enable persistent sessions (clean_session=false) but use auto-generated or random client IDs like ESP32_ + random() or allow the library to generate one.

Why It Happens: Many MQTT libraries generate unique client IDs automatically to avoid ID collisions. Developers don’t realize this breaks persistent session restoration. Each reconnection creates a new session with different ID, so queued messages and subscriptions from the previous session are orphaned and eventually expire.

The Fix: Use a stable, unique client ID derived from hardware identifiers. For ESP32, use the MAC address or chip ID. For MQTT 5.0, you can also let the broker assign a persistent ID via Assigned Client Identifier.

// MQTT 3.1.1: Derive stable ID from hardware
char clientId[32];
uint64_t chipId = ESP.getEfuseMac();  // Unique per chip
snprintf(clientId, sizeof(clientId), "ESP32_%04X%08X",
         (uint16_t)(chipId >> 32), (uint32_t)chipId);

// Connect with persistent session
if (mqttClient.connect(clientId, user, pass,
                       willTopic, willQos, willRetain, willMessage,
                       false)) {  // clean_session = false
  // Session restored - subscriptions and queued messages available
}

// MQTT 5.0: Let broker assign ID (paho-mqtt Python)
client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2,
                     client_id="",  // Empty = broker assigns
                     protocol=mqtt.MQTTv5)
client.connect(broker, port, clean_start=False,
               properties=Properties(PacketTypes.CONNECT))
# Check assigned ID in CONNACK properties

Broker Configuration (Mosquitto): Set persistent_client_expiration to control how long orphaned sessions are kept. Default is infinite, which can consume broker memory if clients use random IDs.

# mosquitto.conf - expire orphaned sessions after 7 days
persistent_client_expiration 7d

The Mistake: Developers configure QoS 2 on the publisher side, expecting guaranteed exactly-once delivery to subscribers, but subscribers connect with QoS 0 or 1. They’re confused when messages are duplicated or lost at the subscriber despite using QoS 2 for publishing.

Why It Happens: MQTT QoS is not end-to-end; it applies separately to publisher-to-broker and broker-to-subscriber segments. The effective QoS for delivery is the minimum of the two. Publishing with QoS 2 to a subscriber with QoS 0 subscription results in QoS 0 delivery (fire-and-forget) to that subscriber.

The Fix: Match QoS levels across the entire message path. If exactly-once delivery is required, both publisher and subscriber must use QoS 2. Document QoS requirements in your API specification and validate them during system integration testing.

# Publisher: QoS 2 for critical command
client.publish("factory/line1/emergency_stop", "STOP", qos=2)

# Subscriber: MUST also use QoS 2 for exactly-once delivery
def on_connect(client, userdata, flags, reason_code, properties):
    # WRONG: QoS 0 subscription downgrades all deliveries
    # client.subscribe("factory/line1/emergency_stop", qos=0)

    # CORRECT: Match publisher QoS for end-to-end guarantee
    client.subscribe("factory/line1/emergency_stop", qos=2)

# Effective QoS = min(publisher_qos, subscriber_qos)
# QoS 2 publish + QoS 0 subscribe = QoS 0 delivery (NO guarantee!)
# QoS 2 publish + QoS 2 subscribe = QoS 2 delivery (exactly-once)

Production impact: In a manufacturing system, configuring emergency-stop publishers at QoS 2 does not guarantee exactly-once delivery to PLC subscribers that use QoS 0. During a network glitch, that mismatch can still produce lost or repeated stop events because the subscriber leg has been downgraded. Document and test the QoS contract on both sides of every critical topic.

The Mistake: Developers deploy hundreds of IoT devices with persistent sessions (clean_session=false) and long keep-alive intervals (300+ seconds). When the broker restarts or fails over, all devices attempt to reconnect simultaneously, overwhelming the broker with CONNECT packets and queued message delivery.

Why It Happens: Persistent sessions are designed to survive brief disconnections, but broker restarts trigger mass reconnection. With 1,000 devices each having 50 queued messages, the broker must deliver 50,000 messages within seconds while also handling 1,000 simultaneous CONNECT handshakes. Default broker configurations often can’t handle this “thundering herd” scenario.

The Fix: Implement staggered reconnection with exponential backoff and jitter. Configure broker max_queued_messages_per_client to limit queue buildup. For MQTT 5.0, use Session Expiry Interval to automatically clean up stale sessions.

import random
import time

class MQTTClientWithBackoff:
    def __init__(self, client_id):
        self.client_id = client_id
        self.base_delay = 1.0  # 1 second base
        self.max_delay = 120.0  # 2 minute cap
        self.attempt = 0

    def connect_with_backoff(self, broker, port):
        while True:
            try:
                self.client.connect(broker, port)
                self.attempt = 0  # Reset on success
                return
            except Exception as e:
                self.attempt += 1
                # Exponential backoff: 1s, 2s, 4s, 8s... capped at 120s
                delay = min(self.base_delay * (2 ** self.attempt), self.max_delay)
                # Add jitter: random 0-50% of delay to spread reconnections
                jitter = random.uniform(0, delay * 0.5)
                total_delay = delay + jitter
                print(f"Reconnect attempt {self.attempt} in {total_delay:.1f}s")
                time.sleep(total_delay)

# Broker configuration (mosquitto.conf)
# Limit queue buildup per client (Mosquitto 2.x)
# max_queued_messages 1000
# max_inflight_messages 20
#
# MQTT 5.0: Auto-expire sessions after 1 hour of inactivity
# persistent_client_expiration 1h

Sizing Guide: For N devices with Q average queued messages and M bytes per message, broker restart requires handling NxQxM bytes immediately. Example: 1,000 devices x 100 messages x 500 bytes = 50MB burst. Ensure broker memory can handle 2-3x this peak load.