757 Transport Review: DTLS Security Architecture

757.1 Learning Objectives

By the end of this chapter, you will be able to:

Understand DTLS architecture: Explain how DTLS secures UDP-based IoT communication
Analyze handshake process: Describe the DTLS handshake stages and their purpose
Apply DoS protection: Explain cookie-based protection for constrained devices
Optimize for IoT: Use session resumption to reduce handshake overhead

757.2 Prerequisites

Required Chapters:

Transport Review: Protocol Selection - Protocol trade-offs
Transport Fundamentals - TCP/UDP basics

Estimated Time: 20 minutes

757.3 DTLS Security Architecture

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#16A085', 'secondaryColor': '#E67E22', 'tertiaryColor': '#7F8C8D', 'fontSize': '14px'}}}%%
flowchart TB
    subgraph CLIENT["IoT Device (Client)"]
        C1["Application<br/>(CoAP)"]
        C2["DTLS Layer<br/>(Encryption)"]
        C3["UDP Layer<br/>(Transport)"]
        C1 --> C2 --> C3
    end

    subgraph HANDSHAKE["DTLS Handshake"]
        H1["1. ClientHello"]
        H2["2. HelloVerifyRequest<br/>(Cookie for DoS protection)"]
        H3["3. ClientHello + Cookie"]
        H4["4. ServerHello + Certificate"]
        H5["5. Key Exchange<br/>(PSK or ECC)"]
        H6["6. Finished"]
        H1 --> H2 --> H3 --> H4 --> H5 --> H6
    end

    subgraph RECORD["DTLS Record Protection"]
        R1["Plaintext Data"]
        R2["Add Sequence Number<br/>(Replay protection)"]
        R3["Encrypt with AES-GCM<br/>(Confidentiality)"]
        R4["Add Auth Tag 16B<br/>(Integrity)"]
        R5["DTLS Record Header 13B<br/>(Type, Version, Length)"]
        R1 --> R2 --> R3 --> R4 --> R5
    end

    subgraph SERVER["Cloud Server"]
        S1["UDP Layer<br/>(Transport)"]
        S2["DTLS Layer<br/>(Decryption)"]
        S3["Application<br/>(CoAP)"]
        S1 --> S2 --> S3
    end

    CLIENT --> HANDSHAKE
    HANDSHAKE --> RECORD
    RECORD --> SERVER

    FEATURES["DTLS Features<br/>━━━━━━━━━━━━<br/>✓ Cookie-based DoS protection<br/>✓ Session resumption (68% savings)<br/>✓ Replay protection (sequence numbers)<br/>✓ AES-GCM encryption (confidentiality)<br/>✓ HMAC authentication (integrity)<br/>✓ PSK or certificate-based auth<br/>━━━━━━━━━━━━<br/>Overhead: 13B header + 16B tag<br/>Handshake: 620B full, 200B resumed"]

    HANDSHAKE --> FEATURES
    RECORD --> FEATURES

    style CLIENT fill:#2C3E50,stroke:#16A085,stroke-width:3px,color:#fff
    style SERVER fill:#2C3E50,stroke:#16A085,stroke-width:3px,color:#fff
    style HANDSHAKE fill:#E67E22,stroke:#2C3E50,stroke-width:3px,color:#fff
    style RECORD fill:#16A085,stroke:#2C3E50,stroke-width:3px,color:#fff
    style FEATURES fill:#7F8C8D,stroke:#2C3E50,stroke-width:2px,color:#fff
    style C1 fill:#34495E,stroke:#16A085,stroke-width:1px,color:#fff
    style C2 fill:#34495E,stroke:#16A085,stroke-width:1px,color:#fff
    style C3 fill:#34495E,stroke:#16A085,stroke-width:1px,color:#fff
    style S1 fill:#34495E,stroke:#16A085,stroke-width:1px,color:#fff
    style S2 fill:#34495E,stroke:#16A085,stroke-width:1px,color:#fff
    style S3 fill:#34495E,stroke:#16A085,stroke-width:1px,color:#fff
    style H1 fill:#F39C12,stroke:#2C3E50,stroke-width:1px,color:#000
    style H2 fill:#F39C12,stroke:#2C3E50,stroke-width:1px,color:#000
    style H3 fill:#F39C12,stroke:#2C3E50,stroke-width:1px,color:#000
    style H4 fill:#F39C12,stroke:#2C3E50,stroke-width:1px,color:#000
    style H5 fill:#F39C12,stroke:#2C3E50,stroke-width:1px,color:#000
    style H6 fill:#F39C12,stroke:#2C3E50,stroke-width:1px,color:#000
    style R1 fill:#1ABC9C,stroke:#2C3E50,stroke-width:1px,color:#fff
    style R2 fill:#1ABC9C,stroke:#2C3E50,stroke-width:1px,color:#fff
    style R3 fill:#1ABC9C,stroke:#2C3E50,stroke-width:1px,color:#fff
    style R4 fill:#1ABC9C,stroke:#2C3E50,stroke-width:1px,color:#fff
    style R5 fill:#1ABC9C,stroke:#2C3E50,stroke-width:1px,color:#fff

Figure 757.1: DTLS Security Architecture for IoT Communication

DTLS (Datagram Transport Layer Security) architecture for securing UDP-based IoT communication. System spans three components: IoT device client (application→DTLS→UDP layers), DTLS handshake protocol, record protection mechanism, and cloud server (UDP→DTLS→application). Handshake proceeds in 6 steps including cookie-based DoS protection (HelloVerifyRequest prevents resource exhaustion from spoofed ClientHello messages) and key exchange supporting PSK or ECC. Record protection applies 5-stage pipeline: plaintext data → sequence number (replay protection) → AES-GCM encryption (confidentiality) → 16-byte authentication tag (integrity) → 13-byte DTLS record header. Key features enable constrained IoT deployment: cookie verification protects limited-RAM gateways (64KB) from DoS attacks, session resumption reduces handshake energy by 68% (200B vs 620B), AES-GCM provides authenticated encryption, PSK mode avoids expensive certificate validation. Typical overhead: 29 bytes per record (13B header + 16B auth tag), amortized handshake cost with resumption. DTLS enables secure CoAP (CoAPS) achieving 5-10 year battery life for sensors reporting every minute.

757.4 DTLS Key Features

Feature	Purpose	Benefit for IoT
Cookie-based DoS protection	Prevents resource exhaustion	Protects constrained gateways
Session resumption	Abbreviated handshake	68% energy savings
Sequence numbers	Replay protection	Prevents repeated commands
AES-GCM encryption	Confidentiality	Protects sensor data
Authentication tag	Integrity	Detects tampering
PSK mode	Pre-shared keys	Avoids certificate overhead

757.5 DTLS Handshake Process

The DTLS handshake establishes a secure session:

ClientHello: Client initiates with supported cipher suites
HelloVerifyRequest: Server sends cookie for DoS protection
ClientHello + Cookie: Client proves address reachability
ServerHello + Certificate: Server responds with chosen parameters
Key Exchange: PSK or ECC-based key agreement
Finished: Both sides confirm successful handshake

757.5.1 Handshake Overhead

Handshake Type	Messages	Bytes	Round Trips
Full handshake	6	620	3 RTT
Session resumption	3	200	1 RTT

Session resumption saves 68% bytes and 2 round trips.

757.6 DoS Protection with Cookies

DTLS cookies prevent denial-of-service attacks:

Without cookies (vulnerable):

Attacker → Server: ClientHello (spoofed source IP)
Server: Allocates state, generates keys, sends ServerHello
Result: Server wasted RAM/CPU for fake client
DoS attack: Send 1000s of ClientHellos → Server runs out of memory

With cookies (protected):

1. Client → Server: ClientHello
2. Server: Generates cookie = HMAC(client_IP, secret)
3. Server → Client: HelloVerifyRequest(cookie)
   - No state allocated yet!
4. Client → Server: ClientHello + cookie
5. Server: Verifies cookie, then allocates state
6. Continue normal handshake

Cookie generation:

cookie = HMAC-SHA256(
    key = server_secret,
    message = client_IP + client_port + server_IP + server_port
)

Why critical for constrained devices:

Limited RAM: IoT gateway with 64 KB RAM can’t handle 1000s of fake handshakes
Limited CPU: Expensive key generation (RSA, ECC) for fake clients wastes power
Stateless verification: Cookie HMAC is fast, doesn’t allocate state

757.7 Session Resumption for Battery Savings

Session resumption dramatically reduces energy consumption:

Without session resumption:

Every message: Full handshake (120 mJ) + data transmission (10 mJ)
Energy per message: 130 mJ
Per day (2,880 messages): 374 J = 104 mAh

With session resumption:

First message: Full handshake (120 mJ) + data (10 mJ) = 130 mJ
Subsequent: Abbreviated handshake (40 mJ) + data (10 mJ) = 50 mJ
Per day: 1 × 130 mJ + 2,879 × 50 mJ = 144 J = 40 mAh
Savings: 64% reduction!

How session resumption works:

Initial handshake creates session ID and master secret
Both parties cache session state
Subsequent connections reference session ID
Skip expensive key exchange, reuse master secret
Session lifetime: hours to days (configurable)

Best practices:

Enable session resumption (IETF RFC 6347)
Session lifetime: 8-24 hours (balance security vs efficiency)
Periodic full handshake (daily) for key rotation

757.8 Visual Reference Gallery

Visual: DTLS Handshake Process

Modern visualization of DTLS handshake process showing ClientHello, ServerHello, certificate exchange, and key establishment over UDP with retransmission handling using IEEE color palette

DTLS Handshake

DTLS provides TLS-level security for UDP-based IoT protocols like CoAP, with modifications for datagram transport.

AI-Generated Figure Variants for DTLS

DTLS handshake sequence diagram showing ClientHello, HelloVerifyRequest with cookie, ServerHello, certificate exchange, key exchange, and Finished messages for secure UDP communication — DTLS Handshake

DTLS protocol architecture diagram showing relationship between DTLS record layer, handshake protocol, and underlying UDP transport for securing constrained device communication — DTLS Architecture

Diagram showing DTLS stateless cookie mechanism for preventing denial-of-service attacks by requiring clients to prove their address before allocating server resources — DTLS Cookie DoS Protection

DTLS session resumption flow diagram showing abbreviated handshake using session tickets to reduce connection overhead for battery-powered IoT devices — DTLS Session Resumption

757.9 Summary

DTLS provides essential security for UDP-based IoT:

Cookie-based DoS protection prevents resource exhaustion attacks on constrained devices
Session resumption reduces handshake overhead by 68%, enabling long battery life
AES-GCM encryption provides confidentiality and integrity with minimal overhead
PSK mode avoids expensive certificate operations for constrained devices

757.10 What’s Next

Continue exploring transport protocol analysis:

Transport Review: Knowledge Check: Quiz questions to test your understanding
DTLS and Security: Deep dive into DTLS implementation
CoAP Protocol: RESTful communication secured by DTLS