22  DTLS and Transport Security

In 60 Seconds

DTLS (Datagram Transport Layer Security) brings TLS-grade encryption and authentication to UDP-based IoT protocols like CoAP, enabling secure real-time sensor communication without TCP’s latency overhead. This overview chapter maps four sub-chapters covering DTLS fundamentals, handshake mechanics, attack defenses, and performance optimization.

Key Concepts
  • DTLS (Datagram Transport Layer Security): TLS adapted for UDP-based transports; adds an explicit record sequence number and retransmission timer to handle datagram loss and reordering
  • TLS vs DTLS: TLS requires TCP (ordered, reliable); DTLS runs over UDP (unordered, lossy); DTLS adds handshake message sequence numbers and cookie exchange to prevent DoS
  • DTLS Cookie Exchange: Anti-DoS mechanism in DTLS handshake: client sends ClientHello → server sends HelloVerifyRequest with stateless cookie → client resends ClientHello with cookie → server verifies before allocating state
  • DTLS Epoch: Counter incremented each time the key material changes (new session or renegotiation); prevents replay attacks across session key changes
  • Record Sequence Number: 48-bit counter in each DTLS record; used for replay attack detection via anti-replay window (default 64-record sliding window)
  • DTLS 1.2 vs DTLS 1.3: DTLS 1.2 (RFC 6347) based on TLS 1.2; DTLS 1.3 (RFC 9147, 2022) based on TLS 1.3 with improved handshake, 0-RTT support, and unified record header
  • CoAP + DTLS: Standard security mechanism for CoAP (RFC 7252); DTLS mode selection via CoAP URI scheme: coap:// (no security), coaps:// (DTLS), coap+tcp:// (TLS)
  • PSK Mode: Pre-Shared Key DTLS where a symmetric key is pre-configured on both client and server; simpler than certificate-based PKI; suitable for constrained IoT devices

22.1 Learning Objectives

By the end of this chapter, you will be able to:

  • Explain DTLS architecture and how it adapts TLS security for unreliable UDP transport
  • Compare DTLS vs TLS trade-offs for IoT applications based on latency, reliability, and power requirements
  • Configure DTLS for CoAP on port 5684 for secure constrained device communication
  • Implement session resumption to reduce handshake overhead and conserve battery on IoT devices
MVU — Minimum Viable Understanding

DTLS provides TLS-equivalent encryption, authentication, and integrity for UDP-based IoT protocols such as CoAP. Use DTLS on port 5684 for real-time sensor communication, VoIP, and video streaming where low latency matters. Use TLS over TCP for firmware updates, REST APIs, and MQTT connections where reliability matters more than speed. For deployments under 100 devices use pre-shared keys (PSK, ~1.5 KB RAM); for 100+ devices use certificate-based authentication (~5 KB RAM) with per-device revocation. Always enable session resumption to cut reconnection overhead by ~60%.

22.2 Overview

This section covers Datagram Transport Layer Security (DTLS), the protocol that brings TLS-equivalent encryption to UDP-based IoT applications like CoAP, video streaming, and real-time telemetry.

Characters: Sammy the Sensor, Lila the LED, Max the Microcontroller, Bella the Battery

Sammy wants to send a secret temperature reading to the cloud, but there are sneaky eavesdroppers on the network!

Max says: “Don’t worry, Sammy! I’ll put your message in a secret lockbox using DTLS. It’s like writing your message in invisible ink that only the cloud can read.”

Sammy: “But what about TLS? My friend uses that!”

Max: “TLS is great, but it needs TCP – that’s like sending a registered letter where you wait for a delivery confirmation every time. DTLS works with UDP – it’s like sending a sealed postcard. Super fast, still secret, and perfect for when you need to send lots of readings quickly!”

Bella adds: “And the best part? With session resumption, I only need to power the lockbox mechanism once. After that, re-locking is quick and saves me 60% of my energy!”

Lila blinks green: “I light up when the handshake is done – that means the secret channel is ready!”

Key idea: DTLS is like a secret code that protects fast messages (UDP) the same way TLS protects slower, reliable messages (TCP). It keeps IoT sensor data safe from spies!

If you’re new to IoT security, here’s what you need to know:

The problem: Many IoT devices use UDP (User Datagram Protocol) to send data quickly – for example, a temperature sensor reporting every 10 seconds. But UDP has no built-in security. Anyone listening on the network can read the data.

The solution: DTLS (Datagram Transport Layer Security) adds encryption and authentication to UDP, just like TLS does for TCP (which is what makes HTTPS secure for websites).

Think of it this way:

  • TCP + TLS = HTTPS (secure web browsing)
  • UDP + DTLS = Secure CoAP (secure IoT sensor communication)

Why not just use TLS? TLS requires TCP, which has a three-way handshake and guarantees packet delivery. This adds latency (100-300ms) that real-time IoT applications can’t afford. DTLS gives you security without that overhead.

When to use DTLS:

  • CoAP-based sensor networks (most common)
  • Real-time video or audio streaming from IoT cameras
  • VoIP and WebRTC on IoT devices
  • Any UDP-based IoT protocol needing encryption

When to use TLS instead:

  • Firmware updates (reliability matters more than speed)
  • REST API calls from IoT gateways
  • MQTT over TCP connections

22.3 Sub-Chapter Map

This topic is covered in four focused sub-chapters. Read them in the order shown in the Learning Path below, or jump to any sub-chapter that addresses your immediate need:

22.3.1 DTLS Fundamentals and Architecture

Learn the basics of DTLS, including:

  • Why DTLS exists (TLS adapted for UDP)
  • DTLS architecture and protocol stack
  • Security properties: encryption, authentication, integrity
  • Use cases: CoAP, VoIP, WebRTC, gaming
  • Challenges for constrained IoT devices
  • DTLS vs IPsec comparison

22.3.2 DTLS Handshake Protocols

Understand the handshake process in detail:

  • DTLS 1.2 complete handshake (3-RTT with cookie)
  • DTLS 1.3 improvements (1-RTT)
  • TLS 1.3 handshake step-by-step
  • 0-RTT session resumption (benefits and risks)
  • Version comparison: TLS 1.2 vs 1.3, DTLS 1.2 vs 1.3
  • Battery impact of handshake latency

22.3.3 DTLS Attack Scenarios and Authentication

Learn about security threats and defenses:

  • Amplification DoS attacks and cookie defense
  • Session hijacking and MAC protection
  • Downgrade attacks and finished message verification
  • PSK vs certificate authentication comparison
  • Smart home PSK example (10 devices)
  • Industrial IoT certificate example (1000 sensors)

22.3.4 DTLS Performance and Implementation

Master practical implementation:

  • Performance benchmarks (ESP32, handshake time, memory)
  • Session resumption optimization (60% battery savings)
  • Connection ID for mobile devices (RFC 9146)
  • Replay window tuning for lossy networks
  • Worked examples: gateway memory calculation, battery life analysis
  • Cross-hub connections to related content

22.4 Learning Path

Recommended order:

  1. Fundamentals - Start here to understand why DTLS exists
  2. Handshake - Learn how secure connections are established
  3. Attacks & Auth - Understand threats and authentication options
  4. Performance - Optimize for production deployments

Prerequisites: Before starting, review:

Next steps: After completing this section, continue to:

22.5 DTLS Protocol Architecture

The following diagram shows how DTLS fits into the IoT protocol stack alongside TLS, and illustrates the key security services it provides over UDP transport.

DTLS Security Architecture diagram showing layered protocol stack: application layer with CoAP, VoIP, and WebRTC protocols at top in orange; DTLS record layer in navy in the middle providing AES-GCM encryption, certificate or PSK authentication, HMAC integrity protection, and sequence-number replay protection; UDP transport and IP network layers in teal at the bottom. TLS over TCP is shown in parallel to illustrate that DTLS provides equivalent security to TLS but operates over UDP rather than TCP.
Figure 22.1: DTLS Security Architecture showing the protocol stack with application layer (CoAP, VoIP, WebRTC) at top, DTLS record layer in the middle providing encryption, authentication, and integrity, and UDP transport at the bottom. TLS over TCP is shown in parallel for comparison.

22.6 DTLS Handshake Overview

When establishing a secure connection, DTLS follows a specific handshake process that adds a cookie exchange step compared to TLS, protecting against UDP-based amplification attacks. The choice between DTLS versions and authentication modes significantly impacts performance on constrained devices.

DTLS handshake sequence diagram showing the complete connection establishment process: Client sends ClientHello to Server; Server responds with HelloVerifyRequest containing a stateless cookie for DoS protection; Client retransmits ClientHello with the cookie included; Server sends ServerHello, Certificate, ServerKeyExchange, and ServerHelloDone; Client sends ClientKeyExchange, ChangeCipherSpec, and Finished; Server responds with ChangeCipherSpec and Finished; secure DTLS session is now established with shared encryption keys. The cookie exchange adds one round trip compared to TLS but prevents UDP-based amplification attacks.
Figure 22.2: DTLS Handshake flow showing the cookie exchange for DoS protection followed by key exchange and cipher negotiation to establish a secure DTLS session.

22.7 Quick Reference

Feature TLS (TCP) DTLS (UDP)
Use Case Web, downloads CoAP, VoIP, video
Latency 100-300ms 10-50ms
Packet Loss Retransmits Tolerates
IoT Fit APIs, firmware Real-time sensors
Mode Memory Scalability Best For
PSK ~1.5 KB Poor < 100 devices
Certificates ~5 KB Excellent Enterprise

Common Pitfalls
  1. Using TLS instead of DTLS for CoAP: CoAP runs over UDP, so standard TLS (which requires TCP) simply will not work. You must use DTLS (port 5684) for securing CoAP traffic. Attempting to tunnel CoAP over TCP+TLS negates CoAP’s low-latency advantages.

  2. Neglecting session resumption: Every full DTLS handshake costs significant energy on battery-powered devices (e.g., ~270ms measured end-to-end on an ESP32 including crypto processing, plus substantial radio-on time). Without session resumption, a device reconnecting every 10 minutes will devote over 23% of its radio energy budget to handshakes alone. Always implement session tickets or cached sessions.

  3. Choosing PSK for large-scale deployments: Pre-shared keys are simple to set up for a handful of devices, but they become a security and management nightmare at scale. If one key is compromised in a shared-PSK setup, all devices using that key are vulnerable. Use certificate-based authentication (PKI) for deployments exceeding ~100 devices.

  4. Ignoring DTLS record size vs. UDP MTU: DTLS adds overhead (13-byte record header + MAC + possible padding). If your application data plus DTLS overhead exceeds the UDP MTU (typically 1280 bytes for IPv6), fragmentation occurs at the IP layer, dramatically increasing packet loss probability in lossy IoT networks. Design your payloads to fit within a single DTLS record.

22.8 Knowledge Check

Test your understanding of DTLS and transport security concepts:

22.9 Worked Example: Securing a Hospital Patient Monitoring Network

Scenario: A hospital deploys 800 wireless patient monitors (heart rate, SpO2, blood pressure) across 12 floors. Each monitor sends a 48-byte vital signs reading every 10 seconds to a floor gateway via CoAP/UDP. The security team must decide between DTLS with PSK and DTLS with certificates, and evaluate the impact of session resumption on battery life.

Step 1: Assess authentication mode based on scale

Factor PSK Mode Certificate Mode
RAM per session ~1.5 KB ~5 KB
Device count 800 800
Total gateway RAM for sessions 1.2 MB 4.0 MB
Key compromise impact All devices sharing that key exposed Only compromised device affected
Provisioning Manual key distribution PKI infrastructure required

Decision: With 800 devices, certificates are required. A single compromised PSK would expose patient data across an entire floor. HIPAA compliance also mandates per-device identity and revocation capability.

Step 2: Calculate handshake cost with and without session resumption

Full DTLS 1.2 handshake (3-RTT with cookie):

Phase Packets Bytes TX Bytes RX Time
ClientHello 1 95 0 -
HelloVerifyRequest (cookie) 1 0 60 50 ms RTT
ClientHello + cookie 1 155 0 -
ServerHello + Certificate + KeyExchange + Done 1 0 1,200 50 ms RTT
ClientKeyExchange + ChangeCipherSpec + Finished 1 350 0 -
ChangeCipherSpec + Finished 1 0 80 50 ms RTT
Total 6 600 bytes 1,340 bytes ~150 ms

Session resumption (abbreviated handshake):

Phase Packets Bytes TX Bytes RX Time
ClientHello + session ticket 1 130 0 -
ServerHello + ChangeCipherSpec + Finished 1 0 100 50 ms RTT
ChangeCipherSpec + Finished 1 0 40 -
Total 3 130 bytes 140 bytes ~50 ms

Battery Life Comparison: Full Handshake vs. Session Resumption

Let’s quantify the battery impact over 1 year for a hospital monitor reconnecting every 10 minutes:

\[ \text{Reconnections per year} = \frac{365 \times 24 \times 60}{10} = 52{,}560 \text{ sessions} \]

Energy per full DTLS 1.2 handshake (TX: 600 bytes at 80 mA, RX: 1,340 bytes at 40 mA, 50 kbps). The formula first computes time in seconds, then multiplies by current to get milliampere-seconds (mAs), then divides by 3,600 to convert to mAh:

\[ E_{\text{full}} = \frac{\frac{600 \times 8}{50{,}000} \times 80 + \frac{1{,}340 \times 8}{50{,}000} \times 40}{3{,}600} = \frac{7.68 + 8.58}{3{,}600} \approx 0.0045 \text{ mAh} \]

With session resumption (TX: 130 bytes, RX: 140 bytes):

\[ E_{\text{resume}} = \frac{\frac{130 \times 8}{50{,}000} \times 80 + \frac{140 \times 8}{50{,}000} \times 40}{3{,}600} = \frac{1.66 + 0.90}{3{,}600} \approx 0.00071 \text{ mAh} \]

Over 1 year, cumulative handshake energy savings:

\[ \Delta E = 52{,}560 \times (0.0045 - 0.00071) \approx 199 \text{ mAh} \approx 0.20 \text{ Ah per device per year} \]

With a 3,000 mAh battery recharged daily, session resumption reduces security overhead from 23.5% to 4.5% of total radio budget, delivering a 20% reduction in total radio energy consumption.

Step 3: Calculate battery impact over 24 hours

Assume each monitor uses a 3,000 mAh battery (recharged daily), radio TX at 80 mA, RX at 40 mA, NB-IoT at 50 kbps:

Metric Full Handshake Each Time Session Resumption
Handshake TX bytes 600 130
Handshake RX bytes 1,340 140
Handshake TX time 96 ms 21 ms
Handshake RX time 214 ms 22 ms
Handshake energy 0.0045 mAh 0.0007 mAh
Data TX per reading (48 + 13 DTLS + 8 UDP) 69 bytes 69 bytes
Readings per day 8,640 8,640
Handshakes per day (reconnect every 10 min) 144 144
Daily handshake energy 0.65 mAh 0.10 mAh
Daily data energy 2.12 mAh 2.12 mAh
Total daily energy 2.77 mAh 2.22 mAh
Battery % used for security 23.5% of radio budget 4.5% of radio budget

Savings from session resumption: 0.55 mAh/day = 20% reduction in total radio energy consumption.

Try It: DTLS Session Resumption Energy Calculator

Adjust the parameters below to see how session resumption affects daily handshake energy for your IoT deployment.

Step 4: Evaluate DTLS record size vs UDP MTU

Component Size
Vital signs payload 48 bytes
DTLS record header 13 bytes
DTLS MAC (SHA-256) 32 bytes
DTLS IV (AES-CCM) 8 bytes
CoAP header 4 bytes
UDP header 8 bytes
IPv6 header 40 bytes
Total on wire 153 bytes
IPv6 minimum MTU 1,280 bytes

153 bytes is well within the MTU. No fragmentation risk.

Try It: DTLS Record Size vs MTU Checker

Enter your application payload size to see whether DTLS overhead causes IP fragmentation.

Key insight: For this 800-device hospital deployment, the combination of certificate-based DTLS (per-device identity for HIPAA compliance) with session resumption (20% total radio energy savings) provides the right balance. The full handshake costs only 150 ms – acceptable for the initial connection – while resumed sessions at 50 ms keep ongoing overhead minimal. The critical lesson is that session resumption is not optional for devices reconnecting frequently: without it, security handshakes would consume over 23% of the radio energy budget.


22.10 Summary

DTLS enables secure real-time IoT communication by adapting TLS for UDP transport. Key points:

  • Cookie mechanism prevents amplification DoS attacks
  • Sequence numbers provide replay protection
  • Session resumption reduces handshake energy by up to ~84% (full handshake vs. abbreviated resumption)
  • DTLS 1.3 offers 44% faster handshakes than 1.2
  • PSK mode uses 70% less memory than certificates

Choose DTLS for latency-sensitive applications; choose TLS/TCP for reliability-critical data.

22.10.1 Key Takeaways

Concept What to Remember
DTLS purpose TLS-equivalent security for UDP-based protocols (CoAP, VoIP, WebRTC)
Port number CoAP over DTLS uses port 5684 (vs. 5683 for plain CoAP)
Cookie exchange Prevents amplification DoS – server sends stateless cookie before committing resources
PSK vs. Certificates PSK for < 100 devices (1.5 KB RAM); Certificates for enterprise scale (5 KB RAM)
DTLS 1.2 vs. 1.3 1.2 = 3-RTT handshake (~270ms); 1.3 = 1-RTT (~150ms), 44% faster
Session resumption Reuses negotiated parameters, reduces handshake energy by ~84%; saves ~20% of total radio budget
Record size Keep payload + DTLS overhead within UDP MTU to avoid IP fragmentation
Replay protection Sequence numbers + sliding window prevent replayed packets

22.10.2 Decision Guide

  • Use DTLS when: Sending frequent sensor readings, real-time video/audio, any CoAP traffic, latency-sensitive data
  • Use TLS/TCP when: Performing firmware updates, sending critical commands, using MQTT or HTTP APIs, reliability matters more than speed
  • Use PSK when: Small deployment (< 100 devices), constrained memory, simple provisioning acceptable
  • Use Certificates when: Enterprise scale (100+ devices), need per-device revocation, regulatory compliance required

22.11 What’s Next

Topic Chapter Description
DTLS Fundamentals DTLS Fundamentals and Architecture Start here to understand why DTLS exists, its protocol stack, and its core security properties
DTLS Handshake DTLS Handshake Protocols Deep dive into the cookie exchange, 3-RTT vs 1-RTT flows, and 0-RTT session resumption
Attacks and Auth DTLS Attack Scenarios and Authentication Understand amplification DoS, session hijacking, downgrade attacks, and PSK vs certificate trade-offs
Performance DTLS Performance and Implementation Benchmarks, Connection ID for mobile devices, replay window tuning, and battery life analysis
Transport Layer Transport Fundamentals TCP, UDP, and QUIC transport-layer trade-offs and when to choose each
Cryptography Encryption Architecture AES-GCM, ECDHE, and the cryptographic primitives that DTLS relies on