31  DTLS Security Architecture

In 60 Seconds

DTLS secures UDP-based IoT communication through a 6-step handshake with cookie-based DoS protection (preventing resource exhaustion on 64 KB RAM gateways), AES-GCM authenticated encryption, and replay protection via sequence numbers. The overhead is 29 bytes per record (13 B header + 16 B auth tag), and session resumption reduces handshake cost from 620 bytes to 200 bytes (68% reduction, 3 RTT to 1 RTT), enabling secure CoAP deployments with 5-10 year battery life.

Key Concepts
  • DTLS vs TLS Architecture Review: Both provide authentication + encryption; DTLS adds: epoch/sequence numbers, anti-replay window, retransmission timer, cookie exchange, message fragmentation for handshake records
  • IoT-Specific DTLS Optimization Flags: mbedTLS: MBEDTLS_AES_ALT (hardware AES), MBEDTLS_ENTROPY_HARDWARE_ALT (TRNG), MBEDTLS_SSL_MAX_CONTENT_LEN=1024 (memory reduction); WolfSSL: –enable-armasm –enable-aesni
  • DTLS PSK Identity: String identifying the pre-shared key in use; sent in ClientKeyExchange; format: “device_id:batch_id:region” allows server-side key lookup; maximum 65535 bytes but keep <64 bytes for constrained devices
  • DTLS Alert Protocol: 2-byte messages (level: warning=1, fatal=2; description: 0–120); fatal alerts terminate the session; common: 20=bad_record_mac, 42=bad_certificate, 48=unknown_ca, 80=handshake_failure
  • DTLS Renegotiation: DTLS 1.2 supports in-session renegotiation (TLS_EMPTY_RENEGOTIATION_INFO_SCSV); DTLS 1.3 replaces with KeyUpdate message; renegotiation adds 6 RTTs overhead; minimize renegotiation frequency
  • DTLS Performance Metrics: Handshake RTTs: DTLS 1.2 PSK=4, DTLS 1.2 cert=6, DTLS 1.3 PSK=2, DTLS 1.3 1-RTT=2, DTLS 1.3 0-RTT=1; prioritize PSK for constrained devices
  • IANA Cipher Suite Registry: Official registry of TLS/DTLS cipher suites with assigned numeric codes; recommended IoT suites: 0x1301 (TLS_AES_128_GCM_SHA256), 0xC0A8 (TLS_PSK_WITH_AES_128_CCM_8) for constrained devices
  • Certificate Pinning in DTLS: Device stores expected server certificate fingerprint (SHA-256 hash) at manufacturing; rejects DTLS handshake if server presents different certificate; eliminates CA trust dependency

31.1 Learning Objectives

By the end of this chapter, you will be able to:

  • Explain DTLS architecture: Describe how DTLS adapts TLS for UDP and secures IoT communication at the transport layer
  • Analyze the handshake process: Identify each DTLS handshake stage and justify its role in establishing a secure session
  • Evaluate DoS protection mechanisms: Assess how cookie-based verification prevents resource exhaustion on constrained gateways
  • Configure session resumption: Apply abbreviated handshake techniques to calculate and reduce energy overhead for battery-powered IoT deployments

This review covers the security architecture of DTLS, which protects IoT data in transit. Think of security as layers of protection: authentication verifies identity, encryption hides the content, and integrity checks detect tampering. DTLS provides all three for IoT devices communicating over unreliable networks.

“Let us review how DTLS protects our data,” said Max the Microcontroller. “The handshake has 6 steps with cookie-based DoS protection. On a gateway with only 64 KB of RAM, without cookies, an attacker could exhaust all memory with fake handshake requests.”

“AES-GCM is the encryption algorithm used by DTLS,” explained Sammy the Sensor. “It provides both confidentiality – nobody can read the data – and authenticity – nobody can modify it without being detected. The 16-byte authentication tag is like a tamper-evident seal on a package.”

“The overhead is manageable,” added Lila the LED. “Each DTLS record adds 29 bytes – 13 bytes for the header and 16 bytes for the authentication tag. For a 10-byte sensor reading, that means 75 percent overhead. But session resumption reduces the handshake from 620 bytes to 200 bytes.”

“For CoAP deployments targeting 5 to 10 year battery life, session resumption is not optional – it is essential,” emphasized Bella the Battery. “Without it, every reconnection after sleep costs a full handshake. With it, we resume encrypted communication almost instantly.”

31.2 Prerequisites

Required Chapters:

Estimated Time: 20 minutes

31.3 DTLS Security Architecture

DTLS (Datagram Transport Layer Security) adapts TLS for UDP by adding datagram-aware features: message sequencing, retransmission timers, and cookie-based DoS protection. The architecture covers two distinct concerns — the handshake protocol that establishes a shared session key, and the record protocol that encrypts and authenticates every subsequent datagram.

DTLS security architecture diagram for IoT communication showing three-component system: IoT device with application, DTLS, and UDP layers; a central DTLS protocol stack handling handshake and record protection; and a cloud server with UDP, DTLS, and application layers. The handshake performs six steps with cookie-based DoS protection and PSK or ECC key exchange. Record protection applies AES-GCM encryption with a 16-byte authentication tag and a 13-byte DTLS header, totalling 29 bytes overhead per record.
Figure 31.1: DTLS Security Architecture for IoT Communication

DTLS (Datagram Transport Layer Security) architecture for securing UDP-based IoT communication. System spans three components: IoT device client (application→DTLS→UDP layers), DTLS handshake protocol, record protection mechanism, and cloud server (UDP→DTLS→application). Handshake proceeds in 6 steps including cookie-based DoS protection (HelloVerifyRequest prevents resource exhaustion from spoofed ClientHello messages) and key exchange supporting PSK or ECC. Record protection applies 5-stage pipeline: plaintext data → sequence number (replay protection) → AES-GCM encryption (confidentiality) → 16-byte authentication tag (integrity) → 13-byte DTLS record header. Key features enable constrained IoT deployment: cookie verification protects limited-RAM gateways (64 KB) from DoS attacks, session resumption reduces handshake bytes by 68% (200 B vs 620 B), AES-GCM provides authenticated encryption, PSK mode avoids expensive certificate validation. Typical overhead: 29 bytes per record (13 B header + 16 B auth tag), amortized handshake cost with resumption. DTLS enables secure CoAP (CoAPS) achieving 5-10 year battery life for sensors reporting every minute.

31.4 DTLS Key Features

Feature Purpose Benefit for IoT
Cookie-based DoS protection Prevents resource exhaustion Protects constrained gateways
Session resumption Abbreviated handshake ~67% energy savings
Sequence numbers Replay protection Prevents repeated commands
AES-GCM encryption Confidentiality Protects sensor data
Authentication tag Integrity Detects tampering
PSK mode Pre-shared keys Avoids certificate overhead

31.5 DTLS Handshake Process

The DTLS handshake establishes a secure session:

  1. ClientHello: Client initiates with supported cipher suites
  2. HelloVerifyRequest: Server sends cookie for DoS protection
  3. ClientHello + Cookie: Client proves address reachability
  4. ServerHello + Certificate: Server responds with chosen parameters
  5. Key Exchange: PSK or ECC-based key agreement
  6. Finished: Both sides confirm successful handshake

31.5.1 Handshake Overhead

Handshake Type Messages Bytes Round Trips
Full handshake 6 620 3 RTT
Session resumption 3 200 1 RTT

DTLS handshake overhead significantly impacts battery life for IoT devices. Let’s quantify the energy savings from session resumption:

Full DTLS handshake: \[\begin{align} \text{Bytes} &= 620\text{ B (6 messages across 3 RTT)} \\ \text{Time at 250 kbps} &= \frac{620}{31{,}250} + 3 \times \text{RTT} = 20\text{ ms} + 3 \times 100\text{ ms} = 320\text{ ms} \end{align}\]

Session resumption: \[\begin{align} \text{Bytes} &= 200\text{ B (3 messages, 1 RTT)} \\ \text{Time} &= \frac{200}{31{,}250} + 100\text{ ms} = 6\text{ ms} + 100\text{ ms} = 106\text{ ms} \end{align}\]

Energy savings calculation (50 mA radio TX/RX current at 3.3 V): \[\begin{align} E_{\text{full}} &= 3.3\text{ V} \times 50\text{ mA} \times 320\text{ ms} = 52.8\text{ mJ per handshake} \\ E_{\text{resume}} &= 3.3\text{ V} \times 50\text{ mA} \times 106\text{ ms} = 17.5\text{ mJ per handshake} \end{align}\]

For a sensor reporting every minute with handshakes every hour (60 messages per session): \[\begin{align} \text{Daily handshakes} &= \frac{1440\text{ min/day}}{60\text{ min/session}} = 24\text{ sessions} \\ E_{\text{full/day}} &= 24 \times 52.8\text{ mJ} = 1{,}267\text{ mJ/day} \\ E_{\text{resume/day}} &= 24 \times 17.5\text{ mJ} = 420\text{ mJ/day} \\ \text{Savings} &= \frac{1{,}267 - 420}{1{,}267} \approx 67\% \end{align}\]

Converting to capacity consumed (at 3.3 V): 1,267 mJ/day ÷ (3.3 V × 3600 s/h) = 0.107 mAh/day for full handshakes versus 0.035 mAh/day with resumption. With a 2000 mAh battery and other loads factored in, session resumption extends battery life significantly—purely by caching session state.

Session resumption saves 68% bytes (620 B → 200 B) and 2 round trips (3 RTT → 1 RTT).

Try It: DTLS Handshake Energy Calculator

Adjust the network parameters to see how handshake time and energy change for your deployment.

31.6 DoS Protection with Cookies

DTLS cookies prevent denial-of-service attacks:

Without cookies (vulnerable):

Attacker → Server: ClientHello (spoofed source IP)
Server: Allocates state, generates keys, sends ServerHello
Result: Server wasted RAM/CPU for fake client
DoS attack: Send 1000s of ClientHellos → Server runs out of memory

With cookies (protected):

1. Client → Server: ClientHello
2. Server: Generates cookie = HMAC(client_IP, secret)
3. Server → Client: HelloVerifyRequest(cookie)
   - No state allocated yet!
4. Client → Server: ClientHello + cookie
5. Server: Verifies cookie, then allocates state
6. Continue normal handshake

Cookie generation:

cookie = HMAC-SHA256(
    key = server_secret,
    message = client_IP + client_port + server_IP + server_port
)

Why critical for constrained devices:

  • Limited RAM: IoT gateway with 64 KB RAM can’t handle 1000s of fake handshakes
  • Limited CPU: Expensive key generation (ECC, RSA in full-PKI configurations) for fake clients wastes power
  • Stateless verification: Cookie HMAC is fast, doesn’t allocate state

31.7 Session Resumption for Battery Savings

Session resumption dramatically reduces energy consumption:

Without session resumption:

Every message: Full handshake (120 mJ) + data transmission (10 mJ)
Energy per message: 130 mJ
Per day (2,880 messages): 2,880 × 130 mJ = 374,400 mJ ≈ 374 J
At 3.3 V: 374 J ÷ (3.3 V × 3,600 s/h) ≈ 31.5 mAh/day

With session resumption:

First message: Full handshake (120 mJ) + data (10 mJ) = 130 mJ
Subsequent: Abbreviated handshake (40 mJ) + data (10 mJ) = 50 mJ
Per day: 1 × 130 mJ + 2,879 × 50 mJ = 144,080 mJ ≈ 144 J
At 3.3 V: 144 J ÷ (3.3 V × 3,600 s/h) ≈ 12.1 mAh/day
Savings: (31.5 − 12.1) / 31.5 ≈ 62% reduction in energy per day

How session resumption works:

  1. Initial handshake creates session ID and master secret
  2. Both parties cache session state
  3. Subsequent connections reference session ID
  4. Skip expensive key exchange, reuse master secret
  5. Session lifetime: hours to days (configurable)

Best practices:

  • Enable session resumption (IETF RFC 6347)
  • Session lifetime: 8-24 hours (balance security vs efficiency)
  • Periodic full handshake (daily) for key rotation

31.10 Worked Example: Selecting DTLS Configuration for a Smart Meter Network

Scenario: A utility company is deploying 10,000 smart electricity meters across a city. Each meter sends a 32-byte reading every 15 minutes over CoAP/UDP to a regional gateway. The gateway has 256 KB RAM. Meters run on mains power but have constrained 32-bit MCUs with 64 KB RAM. The network uses cellular NB-IoT with 150 ms average RTT. Choose the optimal DTLS configuration.

Step 1: Evaluate Authentication Method

Method Handshake Size RAM per Session Computation Provisioning
PSK (Pre-Shared Key) ~200 bytes ~100 bytes Minimal (HMAC only) Factory-programmed 16-byte key
RPK (Raw Public Key) ~350 bytes ~200 bytes 1 ECC operation Keypair + server public key
Certificate (X.509) ~2,000+ bytes ~1,000 bytes 2+ ECC operations + ASN.1 parsing Full PKI infrastructure

Decision: PSK. With 10,000 meters, each key is factory-programmed during manufacturing. Certificate infrastructure is unnecessary overhead for devices communicating with a single known gateway. PSK handshake fits comfortably in NB-IoT’s MTU.

Step 2: Calculate Gateway Memory Requirements

Concurrent sessions: 10,000 meters / (15 min / ~5s active) ≈ 56 concurrent
Session state per meter: ~100 bytes (PSK) + 48 bytes (sequence numbers, IV)
Active session memory: 56 × 148 = 8,288 bytes (~8 KB)
Cached session tickets (all): 10,000 × 64 bytes = 640 KB → exceeds 256 KB!

Problem: Caching session tickets for all 10,000 meters exceeds gateway RAM.

Solution: LRU session cache for 5,000 most-recent meters (320 KB fits). Remaining meters perform full handshake on reconnection (once per ~30 minutes worst case).

Try It: DTLS Gateway Session Cache Calculator

Explore how device count, reporting interval, and session ticket size affect gateway RAM requirements.

Step 3: Estimate Energy and Bandwidth Impact

Configuration Handshake Energy Per-Record Overhead Daily Energy (96 readings)
No DTLS 0 mJ 0 bytes 96 × 10 mJ = 960 mJ
DTLS (full handshake every time) 120 mJ 29 bytes/record 96 × (120 + 10) = 12,480 mJ
DTLS (session resumption) 40 mJ (first) + 0 (subsequent) 29 bytes/record 40 + 95 × 10.3 ≈ 1,019 mJa

a 10.3 mJ = 10 mJ data transmission + 0.3 mJ AES-GCM processing overhead for the 29-byte DTLS record header and authentication tag.

Session resumption reduces DTLS overhead to 6% of total energy versus 92% without it.

Step 4: Configuration Summary

Parameter Value Rationale
Authentication PSK-AES128-GCM-SHA256 Minimal handshake, 128-bit security, 16-byte GCM tag (29 B overhead)
Session lifetime 4 hours Covers 16 readings before re-handshake
Session cache 5,000 entries (LRU) Fits gateway 256 KB RAM
Cookie verification Enabled Protects gateway from spoofed ClientHello
Replay window 32 packets Covers 8 hours of readings with margin
Key rotation Monthly via secure firmware update Balances security with operational simplicity

Key Insight: For large-scale metering deployments, PSK with session resumption achieves near-zero security overhead (6% energy increase) while providing authenticated encryption. The gateway’s limited RAM – not the meters’ capabilities – is the binding constraint that determines cache sizing.

31.11 Summary

DTLS provides essential security for UDP-based IoT:

  • Cookie-based DoS protection prevents resource exhaustion attacks on constrained devices
  • Session resumption reduces handshake bytes by 68% (620 B to 200 B) and cuts round trips from 3 to 1, enabling long battery life
  • AES-GCM encryption provides confidentiality and integrity with minimal overhead
  • PSK mode avoids expensive certificate operations for constrained devices

31.12 Concept Relationships

Builds on:

Security Architecture:

Application-Layer Integration:

Contrasts with:

  • TLS over TCP - Connection-oriented security (no cookie mechanism needed)
  • IPsec - Network-layer security (different trust model)
  • Application-layer encryption - Protocol-agnostic but more complex

Common Misconceptions:

  • “DTLS is less secure than TLS” - Wrong: Same cryptographic strength, different transport adaptation
  • “Session resumption weakens security” - Wrong: Master secret rotation maintains forward secrecy
  • “PSK is insecure” - Wrong: PSK with strong keys (128-bit) provides adequate security for constrained devices

Key Insight: DTLS’s cookie mechanism is the key innovation enabling UDP security—without it, constrained gateways are vulnerable to resource exhaustion from spoofed handshake floods.

31.13 See Also

Related Transport Reviews:

Deep Dives:

Specifications:

  • RFC 6347: DTLS 1.2 - Core protocol specification
  • RFC 9147: DTLS 1.3 - Latest version with improved performance
  • RFC 7925: TLS/DTLS Profiles for IoT - Constrained device recommendations

Implementation Resources:

  • Mbed TLS DTLS - Lightweight DTLS library
  • tinydtls - Minimal DTLS for severely constrained devices
  • wolfSSL - Commercial DTLS with PSK support

External Guides:

Common Pitfalls

Self-signed certificates are not inherently insecure, but they must be explicitly trusted by the client as the root CA. IoT devices that accept any self-signed certificate without prior trust establishment are trivially MITM-attacked. At device manufacturing, provision the expected server certificate (or CA public key) into protected storage. Never configure devices to skip certificate validation (mbedtls_ssl_conf_authmode(MBEDTLS_SSL_VERIFY_NONE)) in production.

PSK keys shared across all devices in a fleet create a single-point-of-failure: if one device is compromised and the PSK extracted, all fleet devices are compromised. Implement per-device unique PSKs (device ID + random 32-byte key generated at manufacturing) rather than a single fleet-wide PSK. Per-device PSK rotation can be performed via DTLS connection using the old PSK to deliver the new PSK, then renegotiating with the new key.

DTLS version negotiation allows downgrade attacks if the client accepts DTLS 1.0 when only DTLS 1.2 is intended. DTLS 1.0 has known vulnerabilities (POODLE, BEAST for CBC mode). Configure mbedTLS minimum version: mbedtls_ssl_conf_min_version(&conf, MBEDTLS_SSL_MAJOR_VERSION_3, MBEDTLS_SSL_MINOR_VERSION_3) (= DTLS 1.2). Never deploy IoT devices accepting DTLS 1.0 connections in 2024+.

Default mbedTLS and WolfSSL builds include many cipher suites including weak legacy ones (RC4, 3DES, MD5-based suites). Allowing weak cipher suites risks downgrade attacks where an attacker manipulates the handshake to force a weak cipher. Explicitly configure the cipher suite allowlist using mbedtls_ssl_conf_ciphersuites() to include only strong, modern suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 and TLS_PSK_WITH_AES_128_GCM_SHA256 for IoT deployments.

31.14 What’s Next

Continue building your knowledge of transport security and protocol selection:

Chapter Topic Why Read It
Transport Comprehensive Review Full protocol comparison Compare DTLS against all IoT transport options with assessment questions
DTLS and Security DTLS implementation deep dive Implement DTLS with PSK, RPK, and certificate modes on constrained devices
DTLS Handshake Protocols DTLS 1.2 vs 1.3 handshake mechanics Analyze 0-RTT session resumption and calculate its battery impact for your deployment
DTLS Attack Scenarios and Authentication Attack defenses and auth modes Evaluate PSK vs certificate authentication trade-offs for fleet sizes from 10 to 10,000
CoAP Fundamentals CoAP over DTLS (CoAPS) Configure CoAPS on port 5684 for secure RESTful IoT communication
Transport Review: Overhead Analysis Per-protocol overhead quantification Calculate total overhead budget for your constrained device and select the lowest-cost option