27  IoT Protocol Security

27.1 Learning Objectives

By the end of this chapter, you will be able to:

  • Secure MQTT Communications: Configure TLS encryption, authentication, and topic-level access control for MQTT brokers
  • Diagnose MQTT Vulnerabilities: Identify and mitigate UTF-8 validation DoS attacks and QoS abuse vectors
  • Harden CoAP Endpoints: Implement DTLS encryption and rate limiting for constrained devices
  • Defend Against Amplification Attacks: Detect and mitigate CoAP UDP amplification vulnerabilities
  • Select Protocol Security Measures: Justify appropriate security controls based on protocol characteristics and device constraints

Key Concepts

  • MQTT security: The security features of the MQTT protocol: TLS 1.3 for transport encryption, client certificate authentication, username/password authentication, and topic-level access control through broker ACLs.
  • CoAP DTLS: Datagram TLS — the UDP-based equivalent of TLS used to secure CoAP communications between constrained devices and servers, required when CoAP is used for sensitive data transmission.
  • Broker ACL: An Access Control List configured on the MQTT broker restricting which clients can publish or subscribe to specific topics, preventing one device from eavesdropping on or publishing to another device’s topic.
  • Certificate-based authentication: Using X.509 certificates to authenticate IoT devices to brokers and servers rather than passwords, providing stronger authentication that cannot be guessed and can be uniquely revoked per device.
  • Protocol fuzzing: Testing IoT protocol implementations by sending malformed messages to discover input validation failures that could be exploited to crash or compromise devices or brokers.
  • Message integrity: Cryptographic verification that a received protocol message has not been modified in transit — provided by TLS/DTLS MAC fields, or optionally by application-layer HMAC signing for end-to-end integrity.
In 60 Seconds

IoT protocol security encompasses the configuration, hardening, and monitoring of MQTT, CoAP, AMQP, and other IoT-specific protocols to ensure that data transmitted between devices and cloud systems maintains confidentiality, integrity, and authenticity. The key challenge is that many IoT protocols were designed for constrained environments where security features are optional, making it easy to accidentally deploy them without any security enabled.

What is IoT Protocol Security? IoT devices communicate using specialized protocols designed for constrained environments. MQTT (Message Queuing Telemetry Transport) uses a publish-subscribe pattern where devices send messages through a central broker. CoAP (Constrained Application Protocol) provides REST-like communication over UDP for resource-limited devices. Both protocols require specific security measures beyond standard web security.

Why does it matter? In 2022, security researchers found that 47% of publicly exposed MQTT brokers had no authentication—anyone could connect and control devices. Unsecured IoT protocols enable attackers to intercept sensor data, send malicious commands to actuators, or use devices as DDoS amplifiers.

Key terms: | Term | Definition | |——|————| | MQTT | Message Queuing Telemetry Transport—lightweight pub/sub messaging protocol for IoT | | CoAP | Constrained Application Protocol—UDP-based REST protocol for constrained devices | | DTLS | Datagram TLS—security protocol for UDP-based communications | | QoS | Quality of Service—message delivery guarantees (0: at-most-once, 1: at-least-once, 2: exactly-once) | | Topic ACL | Access Control List defining which clients can publish/subscribe to which topics |

“I send hundreds of messages every day using MQTT!” Sammy the Sensor said. “I publish my temperature readings to a topic like ‘building/floor3/temperature’, and anyone subscribed to that topic gets my data. But here is the scary part – if the MQTT broker has no password, ANYONE can connect and listen to my readings or even send fake commands!”

Max the Microcontroller nodded seriously. “In 2022, researchers found that almost half of all public MQTT brokers had no authentication at all. That is like having a mailbox with no lock on a busy street – anyone can read your mail or stuff fake letters inside!”

“So how do we fix it?” Lila the LED asked.

“Three things!” Max counted on his fingers. “First, TLS encryption – wrap every message in a secret code so eavesdroppers see only gibberish. Second, authentication – every device needs a username and password or a certificate to connect. Third, topic ACLs – access control lists that say WHO can publish or subscribe to WHICH topics. The temperature sensor can publish to temperature topics but should never be able to send commands to the door lock!”

“CoAP is similar but uses UDP instead of TCP,” Bella the Battery explained. “It is lighter weight, which is great for tiny devices like me who need to save energy. But it needs DTLS for security – that is like TLS but designed for UDP. And we need rate limiting too, because attackers can use CoAP devices as amplifiers to flood other targets with traffic!”

27.2 Prerequisites

Before diving into this chapter, you should be familiar with:

27.3 MQTT (Message Queuing Telemetry Transport)

MQTT publish-subscribe messaging protocol architecture showing publishers sending sensor data to central message broker which routes messages to subscribers based on topic hierarchies, enabling lightweight IoT device communication
Figure 27.1: MQTT protocol overview
MQTT broker serving as central message routing hub connecting multiple IoT sensor publishers to application subscribers, managing topic subscriptions, Quality of Service levels, and message delivery guarantees
Figure 27.2: MQTT broker architecture
MQTT network topology diagram showing distributed IoT sensors, edge gateways, central message broker, and cloud applications connected through hierarchical topic-based message routing enabling scalable publish-subscribe communication
Figure 27.3: MQTT network topology

Architecture: Publish-Subscribe pattern with message broker.

MQTT architecture diagram showing sensor publishers sending messages to central broker which routes to multiple subscribers based on topic hierarchy, illustrating the decoupled publish-subscribe messaging pattern with Quality of Service guarantees
Figure 27.4: MQTT Publish-Subscribe Architecture: Sensors, Broker, and Subscriber Routing

Topic Hierarchy: adt/lora/adeunis/0018B2000000023A - Hierarchical organization with / separators - Wildcard subscriptions: # (multi-level), + (single-level)

27.3.1 MQTT Security Risks

MQTT security vulnerabilities showing malicious client sending invalid UTF-8 characters to exploit broker validation weaknesses, causing subscriber disconnections and enabling Denial of Service attacks through Quality of Service retry mechanisms
Figure 27.5: MQTT attack scenarios
MQTT UTF-8 validation attack sequence showing malicious client publishing invalid UTF-8 payload that passes through non-validating broker to subscriber, which rejects message and disconnects, triggering broker retry loop with QoS 2 guarantees creating Denial of Service condition
Figure 27.6: MQTT UTF-8 Validation DoS Attack: Invalid Message Causing Infinite Retry Loop

UTF-8 Validation DoS Attack:

  1. Malicious client sends invalid UTF-8 characters
  2. Broker doesn’t validate (passes through)
  3. Subscriber validates and disconnects
  4. Broker retries delivery (QoS=2)
  5. Continuous retry = Denial of Service

Mitigation:

  • Always validate UTF-8 at broker
  • Disable QoS=2 if not needed
  • Implement rate limiting

27.3.2 MQTT Security Implementation

MQTT broker performing UTF-8 validation scenario where broker inspects incoming messages for invalid characters and rejects malformed data before forwarding to subscribers, protecting clients from malicious payloads
Figure 27.7: MQTT: Broker validates
MQTT subscriber validation scenario where client application detects invalid UTF-8 in received message and disconnects from broker, creating vulnerability to Denial of Service if broker retries delivery with Quality of Service guarantees
Figure 27.8: MQTT: Subscriber validates
MQTT security risk scenario where neither broker nor subscriber validate message encoding, allowing invalid UTF-8 to propagate through system potentially causing application crashes, data corruption, or unpredictable behavior in client applications
Figure 27.9: MQTT: Neither validates
Fog computing three-tier architecture showing IoT devices at edge layer, fog nodes providing local processing and security filtering at middle layer, and cloud services at top layer, enabling distributed MQTT message validation and encryption
Figure 27.10: Fog computing architecture for IoT security

Secure MQTT implementations require TLS encryption, authentication, topic-level access control, and UTF-8 validation. Production systems should use libraries like paho-mqtt with proper certificate management and QoS configuration.

27.3.3 MQTT Security Configuration Checklist

Security Control Implementation Notes
Transport Encryption TLS 1.3 on port 8883 Disable TLS 1.0/1.1
Authentication Username/password or X.509 certificates No anonymous connections
Authorization Topic-level ACLs Restrict pub/sub per client
Message Validation UTF-8 validation at broker Prevent encoding attacks
Rate Limiting Message rate limits per client Prevent flooding
Last Will Testament Configure for ungraceful disconnection Detect compromised clients

27.4 CoAP (Constrained Application Protocol)

Constrained Application Protocol (CoAP) RESTful architecture for resource-limited IoT devices, showing UDP-based message exchange with GET, POST, PUT, DELETE methods and confirmable/non-confirmable message types for efficient constrained device communication
Figure 27.11: CoAP protocol overview
Datagram Transport Layer Security (DTLS) handshake sequence showing client and server exchanging HelloVerifyRequest with cookie to prevent denial-of-service, followed by certificate exchange, key agreement, and establishment of encrypted UDP communication channel for CoAP security
Figure 27.12: DTLS two-way handshake

Characteristics:

  • UDP-based (smaller packets than MQTT)
  • No central broker required
  • RESTful API (GET, POST, PUT, DELETE)
  • Multicast support
  • Resource observation

Security: DTLS (Datagram TLS) for CoAP = CoAPs

CoAP protocol stack comparison showing CoAP running directly over UDP for unsecured communication versus CoAPS with DTLS security layer inserted between CoAP application layer and UDP transport, providing encryption, authentication, and integrity protection
Figure 27.13: CoAP vs CoAPS Protocol Stack: Adding DTLS Security Layer

Vulnerabilities:

  • IP address spoofing (IPv4)
  • Amplification attacks (small request triggers large response)
  • Amplification factor: 10-50x

The Misconception: If I use HTTPS, my IoT data is fully protected.

Why It’s Wrong:

  • HTTPS only protects data IN TRANSIT (network)
  • Data at rest (storage) needs separate encryption
  • Data in use (memory) can be extracted
  • Endpoints may be compromised (device or server)
  • Metadata (who talks to whom) still visible

Real-World Example:

  • Smart thermostat sends data over HTTPS
  • Data encrypted during transmission
  • Data stored in cloud database: Unencrypted!
  • Database breach: All temperature history exposed
  • HTTPS didn’t protect the stored data

The Correct Understanding: | Data State | HTTPS Protection | Additional Needed | |————|——————|——————-| | In transit | Protected | - | | At rest (storage) | Not protected | Database encryption, KMS | | In use (memory) | Not protected | Secure enclaves (TEE) | | Metadata | Not protected | VPN, anonymization |

HTTPS is necessary but not sufficient. Encrypt data at every state: transit, rest, and use.

27.5 IoT Protocol Security Comparison

This view compares security features across major IoT application protocols, helping select the right protocol for your security requirements:

IoT protocol security comparison table showing MQTT, CoAP, and HTTP/REST protocols with security features including transport encryption (TLS/DTLS), authentication methods, message size overhead, Quality of Service levels, and suitability for different device classes and network types

Protocol security comparison

Protocol Selection Guide:

  • MQTT: Best for pub/sub patterns, many devices, real-time data (smart home, industrial sensors)
  • CoAP: Best for request/response with constrained devices (battery-powered sensors, LPWAN)
  • HTTP/REST: Best for cloud integration, web apps, non-constrained gateways

27.6 Knowledge Check

### Choosing the Right Security Level: A Decision Framework

Not every IoT device can afford full TLS 1.3. Security choices involve trade-offs between protection strength, computational cost, and device capability:

If your device/scenario is… Choose… Because…
Gateway/SBC with Linux, sensitive data TLS 1.3 (mutual) Full certificate chain, forward secrecy, strongest protection
Constrained MCU (>64 KB RAM) over UDP DTLS 1.2 with PSK No TCP overhead, pre-shared keys avoid certificate parsing
Severely constrained (<32 KB RAM), private network PSK-only (no TLS) Application-layer encryption with pre-provisioned symmetric keys
High-value asset, compliance required (HIPAA, GDPR) TLS 1.3 + client certs Regulatory audit trail, mutual authentication, hardware key storage
Local mesh network (Zigbee/Thread), trusted perimeter Network-layer encryption AES-128-CCM at link layer, no application TLS needed
Prototyping/development only No encryption (plaintext) Acceptable ONLY in isolated lab environments, never in production

Quick Decision Flowchart:

  1. Does the data cross the public internet? Yes –> TLS 1.3 (MQTT) or DTLS 1.2 (CoAP), minimum
  2. Does the device have <64 KB RAM? Yes –> DTLS with PSK or application-layer AES
  3. Is the data personally identifiable or safety-critical? Yes –> Mutual TLS with client certificates
  4. Is the network physically isolated (no internet access)? Yes –> Network-layer encryption may suffice
  5. Default: TLS 1.3 for MQTT, DTLS 1.2 for CoAP. When in doubt, encrypt.

“No encryption because the device is too small” is the most common IoT security anti-pattern. If TLS/DTLS truly cannot fit, use application-layer AES-CCM with pre-shared keys. Never deploy unencrypted devices on any network connected to the internet.

27.7 Case Study: Exposed MQTT Brokers in Industrial IoT

In 2017, security researcher Lucas Lundgren presented at Black Hat Europe the results of scanning the internet for publicly accessible MQTT brokers. The findings revealed the scale of unsecured IoT protocol deployments and their real-world consequences.

The Discovery: Using Shodan (an internet-connected device search engine) and custom scanning tools, Lundgren identified over 67,000 MQTT brokers accessible from the public internet. Of these, approximately 30,000 (45%) required no authentication – anyone could connect, subscribe to all topics using the # wildcard, and publish arbitrary messages.

What Was Exposed:

Sector Exposed Data Risk Level
Home automation Smart lock states, alarm system status, occupancy patterns Physical safety – attackers could unlock doors or determine when homes were empty
Healthcare Patient monitoring telemetry (heart rate, blood oxygen) HIPAA violation, patient safety if data tampered
Industrial PLC control messages, sensor readings from oil/gas facilities Physical damage to equipment, environmental disaster
Transportation GPS coordinates of fleet vehicles, fuel levels, speed Cargo theft, vehicle tracking, driver privacy
Energy Smart meter readings, grid sensor data Infrastructure mapping for targeted attacks

A Specific Incident: In one documented case, an oil and gas company’s MQTT broker was publicly accessible on port 1883 (unencrypted) with no authentication. The broker handled telemetry from 150+ wellhead sensors transmitting pressure, temperature, and flow rate data. An attacker could subscribe to topic wellhead/+/pressure and monitor all wellhead pressures in real time. More critically, the broker also accepted publish commands on control topics – meaning an attacker could send wellhead/014/valve/close to shut a valve remotely. A pressure buildup from a closed valve on an active well could cause a blowout.

Root Causes:

  1. Default configuration shipped without authentication: Mosquitto (the most popular open-source MQTT broker, used by 65% of deployments) ships with allow_anonymous true in its default configuration. Operators deployed the broker without changing this setting.

  2. Port 1883 exposed to internet: MQTT’s standard port (1883 for plaintext, 8883 for TLS) was opened on firewalls for “remote monitoring” without VPN or access restrictions. Many operators did not realise their broker was internet-facing.

  3. No topic ACLs configured: Even brokers with basic username/password authentication rarely implemented topic-level access control. An authenticated temperature sensor could subscribe to # and read all topics, including control commands for other devices.

Cost of Remediation: After Lundgren’s disclosure, the MQTT community and cloud providers responded:

  • Eclipse Foundation updated Mosquitto’s default configuration to allow_anonymous false (version 2.0, December 2020 – three years after the disclosure)
  • AWS IoT Core and Azure IoT Hub require certificate-based authentication by default, eliminating the anonymous access problem for cloud-hosted brokers
  • OWASP added MQTT-specific guidance to the IoT Security Verification Standard (ISVS)

Preventive Checklist for IoT Protocol Deployment:

Step Action Verification
1 Disable anonymous access allow_anonymous false in mosquitto.conf
2 Enable TLS on port 8883 openssl s_client -connect broker:8883 succeeds
3 Configure topic ACLs Each client can only pub/sub to its own topics
4 Block port 1883 from internet nmap -p 1883 <public-IP> shows filtered
5 Enable broker-side UTF-8 validation Send invalid UTF-8 test payload, verify rejection
6 Set up connection rate limiting Broker rejects > 10 new connections/second from single IP

Key Takeaway: The 67,000 exposed MQTT brokers demonstrate that protocol security failures are rarely about cryptographic weakness. They are about deployment defaults. A broker with TLS 1.3, mutual certificate authentication, and topic ACLs is effectively impenetrable – but these features must be explicitly configured. The most dangerous MQTT broker is the one deployed with mosquitto -d and no further configuration.

Scenario: A logistics company deploys GPS tracking devices in 200 delivery trucks. Each device (Quectel BG96 cellular modem + STM32 microcontroller) publishes location updates every 30 seconds to an Eclipse Mosquitto broker running on AWS EC2. The mobile app shows real-time truck locations, ETAs, and driver status.

Initial (Insecure) Architecture:

  • MQTT broker on port 1883 (unencrypted)
  • No authentication (allow_anonymous true)
  • All devices publish to topic trucks/+/location with wildcard read access
  • API keys hardcoded in mobile app APK

Security Implementation Steps:

Step 1: Enable TLS Encryption (Transport Security)

Generate CA certificate and broker certificate:

# CA certificate (valid 10 years)
openssl req -new -x509 -days 3650 -extensions v3_ca -keyout ca.key -out ca.crt -subj "/CN=FleetCA"

# Broker certificate signed by CA
openssl genrsa -out broker.key 2048
openssl req -new -key broker.key -out broker.csr -subj "/CN=fleet.example.com"
openssl x509 -req -in broker.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out broker.crt -days 3650

Update mosquitto.conf:

listener 8883
cafile /etc/mosquitto/certs/ca.crt
certfile /etc/mosquitto/certs/broker.crt
keyfile /etc/mosquitto/certs/broker.key
require_certificate false  # Start with server-side TLS only
tls_version tlsv1.3

Device firmware change (MQTT client):

// ESP32 Arduino MQTT client with TLS
#include <WiFiClientSecure.h>
#include <PubSubClient.h>

const char* ca_cert = \
"-----BEGIN CERTIFICATE-----\n" \
"MIIDXTCCAkWgAwIBAgIJAKL...\n" \  // CA certificate
"-----END CERTIFICATE-----\n";

WiFiClientSecure espClient;
PubSubClient client(espClient);

void setup() {
  espClient.setCACert(ca_cert);  // Verify broker certificate
  client.setServer("fleet.example.com", 8883);
}

Verification: Port 1883 blocked, port 8883 requires TLS. Wireshark shows encrypted MQTT payload.

Step 2: Implement Client Authentication (Mutual TLS)

Generate per-device certificates (script for 200 devices):

for i in {1..200}; do
  DEVICE_ID="truck-$(printf "%03d" $i)"
  openssl genrsa -out "$DEVICE_ID.key" 2048
  openssl req -new -key "$DEVICE_ID.key" -out "$DEVICE_ID.csr" -subj "/CN=$DEVICE_ID"
  openssl x509 -req -in "$DEVICE_ID.csr" -CA ca.crt -CAkey ca.key -CAcreateserial -out "$DEVICE_ID.crt" -days 3650
done

Update mosquitto.conf:

require_certificate true  # Mutual TLS (clients must present certificate)
use_identity_as_username true  # Use certificate CN as MQTT username

Device firmware: Flash device-specific certificate and private key to STM32 flash memory (encrypted partition).

Step 3: Implement Topic-Level Access Control (ACLs)

Create ACL file /etc/mosquitto/acls:

# Truck 001 can only publish to its own topic
user truck-001
topic write trucks/001/location
topic write trucks/001/status

# Truck 002
user truck-002
topic write trucks/002/location
topic write trucks/002/status

# ... (200 devices)

# Mobile app backend (AWS Lambda) can subscribe to all topics
user fleet-backend
topic read trucks/+/location
topic read trucks/+/status

Update mosquitto.conf:

acl_file /etc/mosquitto/acls

Verification: Device truck-001 attempting to publish to trucks/002/location receives MQTT PUBACK with error code 0x87 (Not Authorized).

Step 4: Implement UTF-8 Validation (DoS Prevention)

Update mosquitto.conf:

validate_utf8 true  # Broker validates all UTF-8 strings

Test: Send malformed UTF-8 payload:

import paho.mqtt.client as mqtt

client = mqtt.Client()
client.tls_set(ca_certs="ca.crt", certfile="truck-001.crt", keyfile="truck-001.key")
client.connect("fleet.example.com", 8883)

# Malformed UTF-8: 0xC0 0x80 (overlong encoding of NULL)
malformed_payload = b'\xC0\x80'
client.publish("trucks/001/location", malformed_payload)

Result: Broker rejects message, logs warning: Invalid UTF-8 in PUBLISH topic. Client receives MQTT DISCONNECT.

Step 5: Implement Rate Limiting (QoS Abuse Prevention)

Update mosquitto.conf:

max_inflight_messages 20  # Limit unacknowledged messages per client
max_queued_messages 100   # Limit queued messages for offline clients

Step 6: Configure Last Will and Testament (Detect Compromised Devices)

Device firmware:

void connectMQTT() {
  client.connect("truck-001", NULL, NULL,  // clientID, username, password (NULL = use cert)
                 "trucks/001/status",      // LWT topic
                 1,                        // LWT QoS
                 true,                     // LWT retain
                 "{\"status\":\"OFFLINE\",\"timestamp\":\"2024-02-08T14:30:00Z\"}");  // LWT message
}

Backend monitoring: If trucks/001/status receives OFFLINE message unexpectedly (not during scheduled maintenance), trigger alert for potential device compromise or network attack.

Security Audit Results:

Control Before After Risk Reduction
Transport Encryption Plaintext MQTT TLS 1.3 on port 8883 Prevents eavesdropping on GPS coordinates
Authentication Anonymous Mutual TLS (200 device certs) Prevents unauthorized devices joining fleet
Authorization All devices read/write all topics Per-device topic ACLs Prevents truck-001 from spoofing truck-002 location
Input Validation None UTF-8 validation enabled Prevents DoS via malformed messages
Rate Limiting Unlimited 20 inflight messages/client Prevents QoS flooding attacks
Anomaly Detection None Last Will monitoring Detects unexpected device disconnections

Cost Analysis:

  • Certificate generation (scripted): 2 hours engineering time
  • Mosquitto TLS configuration: 1 hour
  • ACL file generation: 2 hours (scripted from device inventory)
  • Firmware update for TLS: 4 hours development + testing
  • Deployment (OTA firmware update to 200 trucks): 1 week (28 trucks/day @ 5 min/truck)
  • Total: ~40 engineering hours (~$8,000 at $200/hour)
  • Per-device cost: $40 security overhead (amortized)

Prevented Attack: In 2022, a similar unsecured fleet system had GPS coordinates intercepted by competitors who tracked delivery routes and undercut pricing. The breach cost $500,000 in lost contracts. The $8,000 security investment prevents this class of attack entirely.

Choosing the right application protocol depends on device constraints, network type, and communication pattern.

Decision Criteria MQTT (Message Queue Telemetry Transport) CoAP (Constrained Application Protocol) HTTP/REST Best For
Transport TCP (reliable, connection-oriented) UDP (unreliable, connectionless) TCP (reliable) MQTT/HTTP: reliable delivery needed; CoAP: low overhead, packet loss acceptable
Message Size ~2 bytes header + topic + payload ~4 bytes header + options + payload ~200+ bytes headers (HTTP/1.1) CoAP: smallest packets for LPWAN (LoRa, NB-IoT); MQTT: medium; HTTP: largest
Communication Pattern Publish-Subscribe (many-to-many) Request-Response (one-to-one) Request-Response (one-to-one) MQTT: sensor data to multiple subscribers; CoAP/HTTP: command-response
Device Constraints Moderate (32 KB RAM minimum) Severe (8 KB RAM possible) High (128 KB+ RAM) CoAP: Class 1 devices (<10 KB RAM); MQTT: Class 2 (50+ KB); HTTP: unconstrained
Network Type Wi-Fi, Ethernet, Cellular LPWAN (LoRa, NB-IoT), 6LoWPAN, Zigbee Wi-Fi, Ethernet, Cellular CoAP over UDP suits LPWAN; MQTT suits broadband
Security TLS 1.3 on TCP (port 8883) DTLS 1.3 on UDP (port 5684) TLS 1.3 on TCP (port 443) All support strong encryption; DTLS adds handshake overhead for UDP
QoS Levels 0 (at-most-once), 1 (at-least-once), 2 (exactly-once) Confirmable (at-least-once), Non-confirmable (at-most-once) None (TCP handles reliability) MQTT QoS 2 for critical commands; CoAP confirmable for important data
Retained Messages Yes (broker stores last message per topic) No (stateless request-response) No (stateless) MQTT: retrieve last-known sensor state without polling
Broker/Server Requires MQTT broker (Mosquitto, HiveMQ) No broker (peer-to-peer or with CoAP server) Requires HTTP server (nginx, Apache) MQTT: centralized management; CoAP: distributed architecture
Firewall Friendliness Port 8883 often blocked by corporate firewalls Port 5684 often blocked Port 443 almost always open HTTP: best for enterprise networks; MQTT/CoAP may need VPN
Latency Low (persistent TCP connection, no handshake per message) Very low (no connection setup) High (TCP handshake + TLS handshake per request) CoAP: lowest latency for sporadic messages; MQTT: lowest for frequent messages
Battery Life Good (persistent connection reduces overhead) Excellent (no connection state, sleep between messages) Poor (repeated TLS handshakes) CoAP: best for battery-powered sensors (Class 1 devices)
Existing Ecosystem Strong (AWS IoT Core, Azure IoT Hub, Google Cloud IoT) Growing (Eclipse Californium, libcoap) Universal (every cloud platform) HTTP: easiest cloud integration; MQTT: IoT-native cloud services

Quick Selection Guide:

  1. What’s your device class?
    • Class 0 (< 10 KB RAM): CoAP (only option)
    • Class 1 (10-50 KB RAM): CoAP or lightweight MQTT-SN
    • Class 2 (50+ KB RAM): MQTT (best balance)
    • Unconstrained (128 KB+ RAM): HTTP/REST or MQTT
  2. What’s your network?
    • LPWAN (LoRa, NB-IoT, Sigfox): CoAP over UDP
    • 6LoWPAN, Zigbee, Thread: CoAP (native support)
    • Wi-Fi, Ethernet, Cellular: MQTT or HTTP
    • Behind restrictive firewall: HTTP on port 443
  3. What’s your communication pattern?
    • Sensor → Multiple consumers: MQTT (pub-sub)
    • Command → Device → Response: CoAP or HTTP (request-response)
    • Frequent periodic updates: MQTT (persistent connection)
    • Sporadic infrequent messages: CoAP (no connection overhead)
  4. What’s your power budget?
    • Battery (coin cell, AA): CoAP (sleep between messages)
    • Battery (rechargeable, solar): MQTT (acceptable overhead)
    • Mains powered: HTTP or MQTT (power not a constraint)
  5. What’s your security requirement?
    • Strong encryption mandatory: All three support TLS/DTLS
    • Mutual authentication needed: MQTT with client certificates or CoAP with DTLS PSK
    • Firewall traversal critical: HTTP (port 443 universally open)

Hybrid Architectures:

Many production systems combine protocols: - Edge devices → CoAP → Gateway → MQTT → Cloud: Constrained sensors use CoAP locally; gateway translates to MQTT for cloud - Sensors → MQTT → Broker → HTTP → Web App: MQTT for device communication; HTTP REST API for web/mobile clients - Field devices → CoAP → Fog Node → HTTP → Central Server: CoAP in field networks; HTTP for internet backbone

Best For (Summary): - MQTT: Smart home hubs, industrial sensors (50+ devices), building automation, fleet tracking (200+ devices), real-time dashboards - CoAP: Environmental sensors (battery-powered), LPWAN deployments (LoRa, NB-IoT), wearables, agricultural IoT (soil moisture, weather stations) - HTTP: Cloud API integration, web dashboards, mobile apps, enterprise systems, unconstrained gateways

Common Mistake: Using MQTT Without Authentication Because “It’s Just Internal Network Traffic”

The Mistake: Many organizations deploy MQTT brokers with allow_anonymous true and no TLS encryption, reasoning that the broker is on an internal network behind a firewall, so authentication is unnecessary. After all, only authorized employees can access the internal network, right?

Why This Fails:

  1. Lateral Movement After Initial Breach: When an attacker compromises any device on the internal network (via phishing, malware, compromised laptop, vulnerable web server), they gain access to the entire network. An unauthenticated MQTT broker becomes an immediate high-value target. The attacker can:

    • Subscribe to # (wildcard) and capture ALL sensor data across the organization
    • Publish malicious commands to control actuators (door locks, HVAC, industrial equipment)
    • Use the broker as a C2 (command and control) channel for malware, hiding in legitimate MQTT traffic

  2. Insider Threats: “Internal network” includes contractors, temporary employees, visitors on guest Wi-Fi (if misconfigured), and disgruntled employees. Without authentication, anyone with network access can connect to the broker and exfiltrate data or sabotage operations.

  3. Supply Chain and IoT Device Vulnerabilities: If a single IoT device on the network is compromised (via firmware vulnerability, default credentials, physical access), that device can connect to the unauthenticated MQTT broker and subscribe to all topics. The broker doesn’t distinguish between legitimate sensors and compromised attackers.

  4. Topic Enumeration: Without authentication, there’s no access control. An attacker can discover the entire topic hierarchy:

    mosquitto_sub -h internal-broker.local -p 1883 -t '#' -v
# Reveals: company/floor2/conference-room-A/occupancy: true
#          company/server-room/temperature: 28.5
#          company/ceo-office/door-lock/status: unlocked

    This exposes physical security information (which rooms are empty, when executives are present) and operational data (server room overheating, network closet access).

Real-World Example: 2019 Smart Office MQTT Breach

A European co-working space deployed 300 sensors (occupancy, temperature, door locks) connected to an internal MQTT broker with no authentication. When a visitor’s laptop was compromised via a malicious USB drive, the attacker used the laptop’s network access to connect to the MQTT broker (port 1883, no password). Within 2 hours, the attacker had: - Subscribed to building/# and logged 3 weeks of occupancy data showing when private offices were empty - Published to locks/executive-suite/unlock to open the CEO’s office door - Exfiltrated 50,000 records of occupancy patterns (GDPR violation - occupancy = personal data) - Cost to company: €120,000 fine (GDPR breach) + €50,000 security remediation + reputational damage

The Correct Approach (Defense in Depth for Internal Networks):

Even on internal networks, MQTT brokers require authentication and authorization:

Minimum Security (Internal Network):

  1. Username/Password Authentication:

    # mosquitto.conf
allow_anonymous false
password_file /etc/mosquitto/passwd

    Generate passwords:

    mosquitto_passwd -c /etc/mosquitto/passwd sensor-001
mosquitto_passwd /etc/mosquitto/passwd mobile-app-backend

  2. Topic ACLs:

    # /etc/mosquitto/acls
user sensor-001
topic write sensors/temperature/sensor-001

user mobile-app-backend
topic read sensors/#

  3. TLS Encryption (prevents eavesdropping by compromised devices on same network):

    listener 8883
cafile /etc/mosquitto/certs/ca.crt
certfile /etc/mosquitto/certs/broker.crt
keyfile /etc/mosquitto/certs/broker.key

Recommended Security (Production Internal Network):

  • Mutual TLS (client certificates) instead of passwords
  • Network segmentation (IoT VLAN separate from corporate VLAN)
  • Firewall rules allowing only authorized devices to reach broker port
  • SIEM logging of all MQTT connections and failed authentication attempts

Why “Internal Network” Is Not a Security Boundary:

According to Verizon’s 2023 Data Breach Investigations Report: - 74% of breaches involve a human element (phishing, stolen credentials, social engineering) - Once inside the network, attackers spend an average of 21 days before detection - 82% of breaches involve lateral movement from initial compromise to target systems

The lesson: “Internal network” means “accessible to anyone who has compromised any device on the network.” MQTT brokers handling sensitive data (occupancy, access control, industrial telemetry) must be authenticated and encrypted regardless of network location. The additional overhead of enabling authentication is minimal (2-hour configuration task) compared to the cost of a breach (six-figure fines, incident response, reputation damage).

Saying “we don’t need MQTT authentication on the internal network” is equivalent to saying “we don’t need door locks because our building has a fence.” Perimeter defenses fail. Defense in depth requires security at every layer, including authentication for internal services.

Message delivery probability with Quality of Service levels

\[P_{\text{delivery}}^{\text{QoS-n}} = \begin{cases} P_{\text{network}} & \text{QoS 0 (at-most-once)} \\ 1 - (1-P_{\text{network}})^k & \text{QoS 1 (at-least-once, k retries)} \\ 1 & \text{QoS 2 (exactly-once)} \end{cases}\]

Interactive QoS Calculator:

Working through an example:

Given: MQTT broker with 5% packet loss, 3 retry attempts for QoS 1

Step 1: QoS 0 delivery probability \(P_{\text{QoS-0}} = 1 - 0.05 = 0.95\) (95% single-attempt success)

Step 2: QoS 1 with retries \(P_{\text{QoS-1}} = 1 - (0.05)^3 = 1 - 0.000125 = 0.999875\) (99.99%)

Step 3: Bandwidth overhead Messages per QoS 1: \(1 + (1-P_{\text{network}}) \times k = 1 + 0.05 \times 3 = 1.15\) (15% overhead)

Step 4: QoS 2 four-way handshake Message count: \(4\) (PUBLISH + PUBREC + PUBREL + PUBCOMP) Overhead vs QoS 0: \(400\%\) bandwidth increase

Step 5: Latency impact QoS 0: \(T_{\text{latency}} = RTT\) (single round-trip) QoS 2: \(T_{\text{latency}} = 2 \times RTT\) (two round-trips)

Result: QoS 1 provides 99.99% delivery with 15% bandwidth overhead. QoS 2 guarantees 100% delivery but costs 4× bandwidth and 2× latency. Use QoS 2 only for critical non-idempotent commands (billing, door unlock).

In practice: Temperature readings tolerate 5% loss (QoS 0). Alarms need reliability (QoS 1). Financial transactions require exactly-once (QoS 2). Choosing appropriate QoS balances reliability vs efficiency for constrained networks.

Common Pitfalls

MQTT without TLS transmits device credentials, sensor data, and control commands in plaintext that any network observer can read. The misconception that IoT networks are physically isolated and therefore do not need encryption is dangerously wrong — apply TLS to all MQTT connections.

Granting a client subscription rights with a wildcard (#) allows it to receive messages from all topics, not just its own. Define granular per-device topic ACLs so each device can only publish to and subscribe from its designated topics.

A fleet-wide shared certificate cannot be revoked for a single compromised device without revoking all devices simultaneously. Issue unique per-device certificates that can be individually revoked through the CA’s CRL or OCSP.

CoAP devices on ‘internal’ networks are still reachable by attackers who compromise any device on that network. Treat all device-to-device communication as potentially interceptable and apply DTLS even for internal CoAP traffic.

27.8 Summary

This chapter covered IoT protocol security for MQTT and CoAP:

MQTT Security:

  • Publish-subscribe architecture with central broker
  • Requires TLS encryption (port 8883), authentication, and topic ACLs
  • UTF-8 validation DoS attack mitigated by broker-side validation
  • QoS levels: Use QoS 2 only for critical non-idempotent operations

CoAP Security:

  • UDP-based REST protocol for constrained devices
  • DTLS provides encryption and authentication
  • Vulnerable to amplification attacks (10-50x factor)
  • Rate limiting and response caching for mitigation

Protocol Selection:

  • MQTT: Real-time pub/sub with many devices
  • CoAP: Request/response for constrained devices
  • HTTP: Cloud integration with non-constrained gateways

27.9 What’s Next

If you want to… Read this
Understand firmware update security over these protocols Secure Data Firmware Updates
Study software vulnerabilities in protocol implementations Secure Data Software Vulnerabilities
Explore IoT device and network security IoT Device and Network Security
Apply network segmentation to contain protocol traffic Network Segmentation
Return to the security module overview IoT Security Fundamentals