4 MQTT Packet and Broker Features

Concept Check

Concept Relationships

Advanced MQTT features connect to both protocol internals and system architecture:

MQTT Protocol Layers:

Carry the chapter forward as one connected chain. First, MQTT Publish-Subscribe Basics - Basic pub-sub concepts. Then, MQTT QoS Levels - Understanding packet acknowledgments. Then, MQTT Security - TLS encryption overhead. Finally, MQTT Architecture - Broker design patterns.

Advanced Features:

Carry the chapter forward as one connected chain. First, MQTT 5.0 Specification - Official standard. Then, Shared Subscriptions - Load balancing pattern. Then, Message Expiry - Time-to-live for stale data. Finally, Topic Aliases - Bandwidth optimization.

Broker Technologies:

Carry the chapter forward as one connected chain. First, Mosquitto - Single-node, learning deployments. Then, EMQX - Clustering for 10M+ connections. Then, HiveMQ - Enterprise with managed clustering. Finally, VerneMQ - Distributed Erlang-based broker.

System Integration:

Carry the chapter forward as one connected chain. First, Message Broker Clustering - HA patterns. Then, Load Balancing - Connection distribution. Then, Edge Gateway Design - Topic bridging. Finally, Capacity Planning - Sizing brokers.

Prerequisites You Should Know:

Carry the chapter forward as one connected chain. First, MQTT packet structure: 2-byte fixed header + variable header + payload. Then, Variable-length encoding saves bytes for small messages. Then, Broker memory: ~10-20 KB per connection + message queue storage. Finally, Topic hierarchy depth impacts wildcard matching performance.

What This Enables:

Carry the chapter forward as one connected chain. First, Optimize bandwidth usage with topic aliases (23 bytes saved per message). Then, Design scalable topic hierarchies supporting wildcard queries. Then, Plan broker capacity: connections, message throughput, memory. Finally, Select appropriate broker for deployment scale (100K vs 10M devices).

See Also

MQTT Protocol Internals:

• MQTT 3.1.1 Specification - Packet format reference
• MQTT 5.0 Specification - New features
• Wireshark MQTT Dissector - Packet analysis tool

Broker Comparison:

• Mosquitto Documentation - Lightweight broker
• EMQX Documentation - Distributed clustering
• HiveMQ Documentation - Enterprise features
• Broker Performance Benchmarks - Throughput comparison

Topic Design Patterns:

• MQTT Essentials: Topics - Best practices
• IoT Topic Naming Conventions - Hierarchy design
• Wildcard Subscription Patterns - Performance optimization

MQTT 5.0 Features:

• MQTT 5.0 Migration Guide - Upgrade considerations
• Shared Subscriptions Tutorial - Load balancing
• Topic Aliases Deep Dive - Bandwidth optimization

Implementation Guides:

• MQTT Hands-On Labs - ESP32 hands-on projects
• Broker Clustering Guide - High availability
• EMQX Cost Estimator - Workload-based planning tool

Try It Yourself

Experiment 1: MQTT Packet Structure Analysis

Capture and analyze MQTT packets with Wireshark:

# Install mosquitto broker and clients
sudo apt install mosquitto mosquitto-clients

# Start Wireshark with MQTT filter
wireshark -f "tcp port 1883" -k

# In another terminal, publish a message
mosquitto_pub -h localhost -t "test/topic" -m "Hello MQTT" -q 1

What to Observe:

• Fixed header: 2 bytes (0x32 for PUBLISH QoS 1)
• Variable header: topic length (2 bytes) + topic (10 bytes) + packet ID (2 bytes)
• Payload: “Hello MQTT” (10 bytes)
• Total: 26 bytes (minimal overhead!)

Experiment 2: Topic Hierarchy Performance

Compare wildcard matching efficiency:

import paho.mqtt.client as mqtt
import time

# Flat topics (inefficient)
flat_topics = [f"sensor_{i}_temperature" for i in range(1000)]

# Hierarchical topics (efficient)
hierarchical_topics = [f"building/floor{i//100}/room{i%100}/temp" for i in range(1000)]

# Measure subscription time
client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2)
client.connect("localhost", 1883)

start = time.time()
for topic in flat_topics:
    client.subscribe(topic, qos=0)
flat_time = time.time() - start

# Clear subscriptions
client.disconnect()
client.connect("localhost", 1883)

start = time.time()
client.subscribe("building/+/+/temp", qos=0)  # Single wildcard subscription!
hierarchical_time = time.time() - start

print(f"Flat (1000 subscriptions): {flat_time:.3f}s")
print(f"Hierarchical (1 wildcard): {hierarchical_time:.3f}s")
print(f"Speedup: {flat_time/hierarchical_time:.0f}x faster")

What to Observe:

• Flat: ~0.5-1.0 seconds for 1,000 subscriptions
• Hierarchical: ~0.001 seconds for 1 wildcard
• 500-1000x faster subscription setup!

Experiment 3: MQTT 5.0 Topic Aliases

Measure bandwidth savings with topic aliases (requires MQTT 5.0 broker):

from paho.mqtt.client import Client as MQTTClient, MQTTv5
from paho.mqtt.properties import Properties
from paho.mqtt.packettypes import PacketTypes

client = MQTTClient(callback_api_version=2, protocol=MQTTv5)
client.connect("localhost", 1883)

# First message: establish alias
long_topic = "farm/northfield/zone1/row12/plant45/soil/moisture"
props = Properties(PacketTypes.PUBLISH)
props.TopicAlias = 1

client.publish(long_topic, "25.5", properties=props)
print(f"First message: {len(long_topic)} byte topic")

# Subsequent messages: use alias (empty topic)
for i in range(100):
    props = Properties(PacketTypes.PUBLISH)
    props.TopicAlias = 1
    client.publish("", f"2{i}.{i}", properties=props)  # Empty topic, uses alias!

savings = len(long_topic) * 100  # Bytes saved over 100 messages
print(f"Saved {savings} bytes with topic alias")

What to Observe:

• Topic name: 50 bytes
• 100 messages: saves 50 × 100 = 5,000 bytes
• Critical for LoRaWAN (200 byte/day limit)

Challenge: Broker Capacity Planning

Calculate broker requirements for a smart city deployment:

Given:
- 100,000 streetlights
- Publish every 5 minutes (12 msg/hour each)
- 3 subscribers per message (dashboard, analytics, alerts)
- Average message: 120 bytes

Calculate:
1. Messages per hour
2. Broker fan-out factor
3. Required bandwidth
4. RAM for connections (assume 20 KB per connection)
5. Select appropriate broker (Mosquitto, EMQX, or HiveMQ)

Bonus: Build your own capacity planning calculator!