1186 MQTT Advanced Topics

1186.1 Learning Objectives

By the end of this chapter, you will be able to:

Understand MQTT Packet Structure: Parse and analyze MQTT binary packets
Design Topic Hierarchies: Create scalable, maintainable topic naming conventions
Optimize for Bandwidth: Reduce message overhead for constrained networks
Implement MQTT 5.0 Features: Use message expiry, topic aliases, and shared subscriptions

1186.2 Prerequisites

Before diving into this chapter, you should be familiar with:

MQTT Publish-Subscribe Basics: Topics, wildcards, and broker architecture
MQTT Quality of Service: QoS levels and session management

1186.3 MQTT Packet Structure

Understanding MQTT’s binary packet format is essential for protocol debugging and optimization.

1186.3.1 Fixed Header (All Packets)

Every MQTT packet begins with a 2-byte minimum fixed header:

Bit Position	7	6	5	4	3	2	1	0
Byte 1	MsgType[3]	MsgType[2]	MsgType[1]	MsgType[0]	DUP	QoS[1]	QoS[0]	RETAIN
Byte 2+	Remaining Length (1-4 bytes, variable-length encoded)

1186.3.2 Fixed Header Fields

Field	Size	Description	Values
Message Type	4 bits	Packet type	1=CONNECT, 3=PUBLISH, 8=SUBSCRIBE
DUP	1 bit	Duplicate flag	0=First, 1=Duplicate
QoS Level	2 bits	Quality of Service	00=QoS 0, 01=QoS 1, 10=QoS 2
RETAIN	1 bit	Retained message	0=No, 1=Yes
Remaining Length	1-4 bytes	Remaining packet size	0-268,435,455 bytes

1186.3.3 Remaining Length Encoding

MQTT uses variable-length encoding to minimize overhead:

Value Range	Bytes Needed	Example
0 - 127	1 byte	23 -> `0x17`
128 - 16,383	2 bytes	200 -> `0xC8 0x01`
16,384 - 2,097,151	3 bytes	50,000 -> `0xD0 0x86 0x03`
2,097,152 - 268,435,455	4 bytes	1,000,000 -> `0xC0 0x84 0x3D`

1186.3.4 PUBLISH Packet Example

Topic: "sensor/temp"
Payload: "25.5"
QoS: 1

Hex dump:
30 17 00 0B 73 65 6E 73 6F 72 2F 74 65 6D 70 00 01 32 35 2E 35

Breakdown:
30        -> Fixed header: PUBLISH (0011), DUP=0, QoS=01, RETAIN=0
17        -> Remaining length: 23 bytes
00 0B     -> Topic length: 11 bytes
73 65 6E 73 6F 72 2F 74 65 6D 70  -> "sensor/temp" (UTF-8)
00 01     -> Packet ID: 1 (for QoS 1 acknowledgment)
32 35 2E 35  -> "25.5" (UTF-8 payload)

1186.3.5 Control Packet Types

Type	Name	Direction	Purpose
1	CONNECT	Client->Broker	Connection request
2	CONNACK	Broker->Client	Connection acknowledgment
3	PUBLISH	Both	Publish message
4	PUBACK	Both	QoS 1 acknowledgment
5	PUBREC	Both	QoS 2 step 1
6	PUBREL	Both	QoS 2 step 2
7	PUBCOMP	Both	QoS 2 step 3
8	SUBSCRIBE	Client->Broker	Subscribe to topics
9	SUBACK	Broker->Client	Subscribe acknowledgment
12	PINGREQ	Client->Broker	Keep-alive ping
13	PINGRESP	Broker->Client	Ping response
14	DISCONNECT	Client->Broker	Graceful disconnect

1186.4 Packet Size Optimization

Tips for Battery-Powered Devices

Keep topic names short - h/l/t vs home/living_room/temperature saves 20 bytes
Use binary payloads - 0x19 (1 byte) vs "25" (2 bytes)
Choose appropriate QoS - QoS 0 uses 50% less messages than QoS 1
Limit retained messages - Only essential status topics
Increase keep-alive interval - 300s vs 60s = 80% fewer PINGREQ packets

Example savings:

Topic: h/b/t (5 bytes) vs home/bedroom/temperature (23 bytes) = 18 bytes saved
100 messages/day x 365 days = 657 KB saved per year per device
For 1000 devices = 641 MB saved annually

1186.5 Worked Example: Smart Building Topic Design

1186.6 Designing Topic Hierarchy for Smart Building

Scenario: Design the MQTT topic structure for a 10-story commercial office building. Each floor has 20 rooms with temperature sensors, occupancy detectors, and smart lighting.

1186.6.1 Step 1: Identify Requirements

Telemetry: Temperature, humidity, occupancy from 200 rooms
Commands: Control lights, blinds, HVAC per room
Status: Online/offline for 600+ devices
Alerts: Fire alarms, security events
Access patterns: Dashboard shows all temps, HVAC controls one floor

1186.6.2 Step 2: Design Base Structure

Bad approach (flat topics):

sensor_floor1_room101_temp     # No hierarchy, can't use wildcards
sensor_floor1_room101_humidity # 600+ individual subscriptions!

Good approach (hierarchical):

building/floor1/room101/sensors/temperature
building/floor1/room101/sensors/humidity
building/floor1/room101/lights/status
building/floor1/room101/lights/command

1186.6.3 Step 3: Apply Naming Conventions

{building}/{floor}/{room}/{device_type}/{measurement_or_action}

Examples:
building/floor03/room305/sensors/temperature    # Telemetry
building/floor03/room305/lights/command         # Control
building/floor03/room305/lights/status          # State
building/floor03/hvac/setpoint                  # Zone control
building/alerts/fire                            # Building-wide

Naming rules:

Use lowercase with no spaces
Use / as separator only
Pad numbers for sorting: floor03, not floor3
End with action type: /temperature, /command, /status

1186.6.4 Step 4: Plan Wildcard Subscriptions

Use Case	Subscription Pattern	Matches
All temps (dashboard)	`building/+/+/sensors/temperature`	200 topics
One floor’s sensors	`building/floor05/+/sensors/#`	40 topics
One room’s everything	`building/floor03/room305/#`	~10 topics
All alerts	`building/alerts/#`	Fire, security

1186.6.5 Step 5: Handle Edge Cases

# Shared spaces (no room number)
building/floor01/lobby/sensors/occupancy
building/stairwell-a/sensors/smoke

# Building-wide systems
building/hvac/chiller/status
building/elevator/car1/position
building/energy/meter/consumption

# System topics
$SYS/broker/clients/connected
building/$status/gateway/floor03

1186.6.6 Result: Topic Hierarchy

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#16A085', 'secondaryColor': '#E67E22', 'tertiaryColor': '#7F8C8D'}}}%%
flowchart TD
    ROOT["building/"]

    ROOT --> FLOORS["floor{01-10}/"]
    ROOT --> ALERTS["alerts/"]
    ROOT --> ENERGY["energy/"]

    FLOORS --> ROOMS["room{01-20}/"]
    FLOORS --> LOBBY["lobby/"]
    FLOORS --> HVAC["hvac/"]

    ROOMS --> R_SENSORS["sensors/"]
    ROOMS --> R_LIGHTS["lights/"]

    R_SENSORS --> RS_TEMP["temperature"]
    R_SENSORS --> RS_HUM["humidity"]
    R_SENSORS --> RS_OCC["occupancy"]

    R_LIGHTS --> RL_STATUS["status"]
    R_LIGHTS --> RL_CMD["command"]

    ALERTS --> A_FIRE["fire"]
    ALERTS --> A_SEC["security"]

    style ROOT fill:#2C3E50,stroke:#16A085,color:#fff
    style FLOORS fill:#16A085,stroke:#2C3E50,color:#fff
    style ALERTS fill:#E67E22,stroke:#2C3E50,color:#fff

Figure 1186.1: Smart building MQTT topic hierarchy

Key design decisions:

Physical hierarchy (building/floor/room) enables location-based queries
Device type grouping (sensors, lights) separates telemetry from control
Action suffixes (status, command) distinguish read vs write
Padded numbers (floor03) ensure correct sorting

1186.7 Worked Example: Fleet Tracking Topics

Designing Topics for Delivery Truck Fleet

Scenario: 500 delivery trucks with GPS, fuel level, and engine temperature sensors. Dispatch needs individual truck queries and aggregate fleet data.

Step 1: Define topic structure

fleet/{truck_id}/{sensor_type}

Examples:
fleet/truck-001/gps
fleet/truck-001/fuel
fleet/truck-001/temp

Step 2: Enable efficient queries

Query	Subscription	Why It Works
All data from truck-001	`fleet/truck-001/#`	Single subscription
All GPS data	`fleet/+/gps`	Cross-fleet GPS
All data	`fleet/#`	Fleet dashboard

Step 3: Add metadata topics

fleet/truck-001/status        # online/offline (retained)
fleet/truck-001/location/city # Current city (retained)
fleet/alerts/breakdown        # Fleet-wide alerts

Show code

{
  const container = document.getElementById('kc-mqtt-advanced-1');
  if (container && typeof InlineKnowledgeCheck !== 'undefined') {
    container.innerHTML = '';
    container.appendChild(InlineKnowledgeCheck.create({
      question: "You're designing an MQTT topic structure for a fleet of 500 delivery trucks. Each truck has GPS, fuel level, and engine temperature sensors. Which topic hierarchy is most appropriate?",
      options: [
        {text: "fleet/truck-001/gps, fleet/truck-001/fuel, fleet/truck-001/temp", correct: true, feedback: "Correct! This hierarchical structure enables: (1) individual truck queries with fleet/truck-001/#, (2) all GPS data with fleet/+/gps, (3) all data with fleet/#."},
        {text: "gps/truck-001, fuel/truck-001, temp/truck-001", correct: false, feedback: "This measurement-first hierarchy makes it hard to query all data from one truck. You'd need 3 subscriptions instead of just fleet/truck-001/#."},
        {text: "truck-001-gps, truck-001-fuel, truck-001-temp (flat topics)", correct: false, feedback: "Flat topics can't use wildcards. With 500 trucks x 3 sensors = 1,500 individual subscriptions!"},
        {text: "fleet/sensors/truck-001/gps with JSON payload containing all data", correct: false, feedback: "Putting all sensor data in one message means subscribers can't filter by sensor type."}
      ],
      difficulty: "medium",
      topic: "mqtt-topics"
    }));
  }
}

1186.8 MQTT 5.0 Features

MQTT 5.0 introduced significant enhancements for enterprise IoT:

1186.8.1 Message Expiry (TTL)

# MQTT 5.0: Message expires after 60 seconds
publish_properties = Properties(PacketTypes.PUBLISH)
publish_properties.MessageExpiryInterval = 60  # seconds

client.publish(
    "sensors/temperature",
    "25.5",
    qos=1,
    properties=publish_properties
)
# If subscriber is offline > 60 seconds, message is discarded

Use case: Sensor readings that become stale quickly (GPS, real-time status).

1186.8.2 Topic Aliases (Bandwidth Optimization)

# First message: Establish alias
publish_properties = Properties(PacketTypes.PUBLISH)
publish_properties.TopicAlias = 1

client.publish(
    "building/floor03/room305/sensors/temperature",  # 42 bytes
    "25.5",
    properties=publish_properties
)

# Subsequent messages: Use alias only
publish_properties.TopicAlias = 1
client.publish(
    "",  # Empty topic, use alias (saves 42 bytes!)
    "25.6",
    properties=publish_properties
)

Savings: 1000 messages/hour with 40-byte topics = 40 KB/hour saved per device.

1186.8.3 Shared Subscriptions (Load Balancing)

# Three workers share subscription to same topic
Worker-1: SUBSCRIBE "$share/workers/sensors/temperature"
Worker-2: SUBSCRIBE "$share/workers/sensors/temperature"
Worker-3: SUBSCRIBE "$share/workers/sensors/temperature"

# Broker distributes messages round-robin:
Message 1 -> Worker-1
Message 2 -> Worker-2
Message 3 -> Worker-3
Message 4 -> Worker-1 (cycles)

1186.8.4 Request/Response Pattern

# Requester: Send command with response topic
request_props = Properties(PacketTypes.PUBLISH)
request_props.ResponseTopic = "devices/sensor001/response"
request_props.CorrelationData = b"request-123"

client.publish(
    "devices/sensor001/command",
    '{"cmd": "get_config"}',
    properties=request_props
)

# Responder: Reply to specified topic
def on_message(client, userdata, msg):
    cmd = json.loads(msg.payload)
    if cmd["cmd"] == "get_config":
        response_props = Properties(PacketTypes.PUBLISH)
        response_props.CorrelationData = msg.properties.CorrelationData

        client.publish(
            msg.properties.ResponseTopic,
            '{"interval": 60, "qos": 1}',
            properties=response_props
        )

1186.8.5 Feature Comparison

Feature	MQTT 3.1.1	MQTT 5.0
Message expiry	Not supported	Built-in TTL
Reason codes	1 (success/fail)	256 detailed codes
User properties	Encode in payload	Native support
Topic aliases	Not supported	Up to 65535 aliases
Shared subscriptions	Broker-specific	Standardized
Flow control	Not supported	Built-in

Recommendation: Use MQTT 5.0 for new projects. Fall back to 3.1.1 only for legacy compatibility.

1186.9 Broker Selection

1186.9.1 Popular MQTT Brokers

Broker	Type	Best For	Connections
Mosquitto	Free, open-source	DIY, Raspberry Pi	~100K
EMQX	Open-source	Large scale	10M+
HiveMQ	Commercial	Enterprise, clustering	10M+
AWS IoT Core	Cloud	AWS integration	Unlimited
test.mosquitto.org	Public test	Learning only	N/A

1186.9.2 Selection Criteria

Device Count	Recommended Broker
< 1,000 devices	Mosquitto (simple, free)
1,000 - 100,000	EMQX or VerneMQ
> 100,000	HiveMQ, AWS IoT Core
Multi-cloud	Self-hosted cluster

1186.10 Summary

Key takeaways:

MQTT packets have minimal overhead (2-byte header minimum)
Topic hierarchy enables powerful wildcard queries
MQTT 5.0 adds message expiry, topic aliases, and shared subscriptions
Choose broker based on scale and feature requirements

Topic design principles:

Use hierarchical structure for wildcard queries
Physical-then-logical organization
Consistent naming conventions
Plan for future scalability

1186.11 What’s Next

Now that you understand MQTT’s advanced features:

Next Chapter: MQTT Practice and Exercises - Hands-on exercises and common pitfalls
Deep Dive: MQTT Comprehensive Review - Broker internals and message flow
Hands-on: MQTT Labs - Build real MQTT applications