748  DTLS Handshake Protocols

748.1 Learning Objectives

By the end of this chapter, you will be able to:

  • Analyze DTLS 1.2 Handshake: Trace the complete handshake including HelloVerifyRequest for DoS protection
  • Compare DTLS Versions: Understand improvements from DTLS 1.2 to DTLS 1.3
  • Understand TLS 1.3: Explain how TLS 1.3’s 1-RTT handshake influenced DTLS 1.3
  • Evaluate Session Resumption: Analyze 0-RTT resumption benefits and security trade-offs
TipMVU: Minimum Viable Understanding

Core concept: DTLS handshakes establish encrypted sessions over UDP, with the cookie mechanism preventing DoS amplification attacks that plague UDP-based protocols.

Why it matters: The handshake is the most expensive part of DTLS - understanding it helps you optimize battery life and connection latency.

Key takeaway: DTLS 1.3 reduces handshake from 3-RTT to 2-RTT (or 1-RTT for resumption), providing 50% faster connection establishment compared to DTLS 1.2.

748.2 Prerequisites

Before diving into this chapter, you should be familiar with:

748.3 DTLS 1.2 Handshake

Mermaid diagram

Mermaid diagram
Figure 748.1: DTLS 1.2 Handshake Protocol showing complete 3-RTT (round-trip time) connection establishment: phase 1 initial handshake where client sends ClientHello with supported versions, cipher suites, and random nonce, server responds with HelloVerifyRequest containing stateless cookie for DoS protection; phase 2 cookie exchange where client retransmits ClientHello including cookie, server sends ServerHello selecting cipher suite plus Certificate, ServerKeyExchange with ECDHE parameters, and ServerHelloDone; phase 3 key exchange where client optionally sends Certificate, then ClientKeyExchange with key material, CertificateVerify signature, ChangeCipherSpec to activate encryption, and Finished message with handshake hash, server responds with ChangeCipherSpec and Finished; phase 4 encrypted application data exchange with all records protected by negotiated cipher suite; all handshake messages include explicit sequence numbers and are retransmitted if lost due to UDP unreliability, contrast with TLS 1-2 RTT handshake.

This variant shows the same handshake concept through a side-by-side comparison - useful for understanding why DTLS requires extra round trips and how the cookie mechanism adds overhead for DoS protection.

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#E67E22', 'secondaryColor': '#16A085', 'tertiaryColor': '#E67E22', 'fontSize': '11px'}}}%%
graph LR
    subgraph TLS["TLS 1.3 (over TCP)<br/>1-RTT Handshake"]
        direction TB
        T1["ClientHello<br/>+ KeyShare"] --> T2["ServerHello<br/>+ KeyShare<br/>+ Encrypted Extensions<br/>+ Certificate<br/>+ Finished"]
        T2 --> T3["Client Finished<br/>+ Application Data"]
        T3 --> T4["Secure Channel<br/>Total: ~100ms"]
    end

    subgraph DTLS["DTLS 1.2 (over UDP)<br/>3-RTT Handshake"]
        direction TB
        D1["ClientHello"] --> D2["HelloVerifyRequest<br/>Cookie"]
        D2 --> D3["ClientHello<br/>+ Cookie"]
        D3 --> D4["ServerHello<br/>+ Certificate<br/>+ ServerKeyExchange"]
        D4 --> D5["ClientKeyExchange<br/>+ ChangeCipherSpec<br/>+ Finished"]
        D5 --> D6["ChangeCipherSpec<br/>+ Finished"]
        D6 --> D7["Secure Channel<br/>Total: ~250ms"]
    end

    WHY["Why Extra Steps?<br/>----------------<br/>DoS Protection<br/>Cookie validates<br/>client IP address<br/>----------------<br/>Packet Loss<br/>UDP may drop<br/>handshake msgs<br/>----------------<br/>Sequencing<br/>No TCP ordering<br/>explicit seq#"]

    TLS -.->|"TCP guarantees<br/>delivery"| WHY
    DTLS -.->|"UDP needs<br/>extra protection"| WHY

    style TLS fill:#16A085,stroke:#2C3E50,color:#fff
    style DTLS fill:#E67E22,stroke:#2C3E50,color:#fff
    style WHY fill:#2C3E50,stroke:#16A085,color:#fff

Figure 748.2: Side-by-side comparison of TLS 1.3 (1-RTT) versus DTLS 1.2 (3-RTT) handshakes showing why UDP requires additional round trips

{fig-alt=“Comparison diagram showing TLS 1.3 with 1-RTT handshake taking ~100ms versus DTLS 1.2 with 3-RTT handshake taking ~250ms. TLS 1.3 sends ClientHello with KeyShare, receives ServerHello bundle, completes with Finished. DTLS 1.2 adds HelloVerifyRequest cookie exchange step for DoS protection because UDP cannot rely on TCP’s delivery guarantees. Center box explains three reasons for extra overhead: DoS protection via cookie validation, packet loss handling since UDP may drop handshake messages, and explicit sequence numbers since TCP ordering not available.”}

748.3.1 DTLS 1.2 Handshake Steps

DTLS Handshake Steps:

  1. ClientHello: Client proposes cipher suites
  2. HelloVerifyRequest: Server sends cookie (DoS protection)
  3. ClientHello (with cookie): Client proves legitimacy
  4. Server responds: Certificate, key exchange parameters
  5. Client key exchange: Client sends key material
  6. Finished messages: Both sides confirm (encrypted)
  7. Secure communication: Application data flows encrypted

Overhead: 6-10 round trips (vs 1.5-2 for TLS over TCP)

Why more overhead? - Unreliable transport: Must handle lost handshake packets - Retransmission: DTLS retransmits handshake messages - Cookie mechanism: Extra round trip for DoS protection

748.4 DTLS 1.2 vs DTLS 1.3

DTLS 1.3 (RFC 9147, 2022) improves upon DTLS 1.2:

Feature DTLS 1.2 DTLS 1.3
Handshake Latency 2 round trips 1 round trip (1-RTT)
Cipher Suites Many (some weak) Only AEAD (authenticated encryption)
Perfect Forward Secrecy Optional (ECDHE) Mandatory
0-RTT Resumption No Yes (risky, opt-in)
Legacy Crypto RSA, CBC, RC4 Removed (only modern ciphers)
Complexity Higher Lower (simplified)

748.4.1 DTLS 1.3 Handshake (1-RTT)

Mermaid diagram

Mermaid diagram
Figure 748.3: DTLS 1.3 Fast 1-RTT Handshake Protocol showing streamlined connection establishment: client sends ClientHello in first message already including KeyShare with ECDHE public key, SupportedGroups indicating elliptic curves, and SignatureAlgorithms for certificate verification, enabling server to derive encryption keys immediately; server responds with ServerHello containing its KeyShare for key agreement, then sends EncryptedExtensions, Certificate, CertificateVerify signature, and Finished message all encrypted under derived handshake keys; client verifies server certificate, sends its own Certificate (if mutual auth), CertificateVerify, and Finished message encrypted; both parties can immediately exchange application data encrypted under session keys; messages in curly braces are encrypted; 1-RTT handshake is 50-150ms faster than DTLS 1.2’s 3-RTT handshake providing significant battery life improvement for IoT devices, mandatory perfect forward secrecy via ECDHE, eliminates legacy vulnerable ciphers.

Benefit: DTLS 1.3 shaves off 100-200ms from handshake (critical for mobile IoT).

748.5 TLS 1.3 Handshake: Step-by-Step

TLS 1.3 reduced the handshake from 2 round-trips to 1 (or 0 for resumption). Understanding the TLS 1.3 handshake is essential because DTLS 1.3 follows the same security model.

748.5.1 Full Handshake (1-RTT)

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#E67E22', 'secondaryColor': '#16A085', 'tertiaryColor': '#E67E22'}}}%%
sequenceDiagram
    participant C as Client
    participant S as Server

    Note over C,S: Step 1: ClientHello
    C->>S: ClientHello + KeyShare + Supported Ciphers

    Note over C,S: Step 2: ServerHello
    S->>C: ServerHello + KeyShare + Certificate + Finished

    Note over C,S: Step 3: Client Finished
    C->>S: Finished + [Application Data]

    Note over C,S: Encrypted Communication
    C->>S: Application Data (encrypted)
    S->>C: Application Data (encrypted)

Figure 748.4

748.5.2 What Happens at Each Step

Step Client Action Server Action Security Achieved
1 Generate ephemeral key pair, send public - Forward secrecy initiated
2 - Verify client, send cert + key Server authenticated
3 Verify server cert, derive session key Derive session key Mutual key agreement
4+ Encrypt with session key Encrypt with session key Confidentiality + integrity

Step-by-Step Breakdown:

  1. ClientHello (Step 1):
    • Client generates ephemeral ECDHE key pair
    • Sends public key in key_share extension
    • Lists supported cipher suites (e.g., TLS_AES_256_GCM_SHA384)
    • Includes random nonce for freshness
  2. ServerHello (Step 2):
    • Server selects cipher suite
    • Generates its own ephemeral key pair
    • Derives handshake keys from ECDHE shared secret
    • Sends Certificate (encrypted with handshake keys)
    • Sends CertificateVerify (signature proving possession of private key)
    • Sends Finished (MAC over entire handshake transcript)
  3. Client Finished (Step 3):
    • Client verifies server certificate chain
    • Verifies CertificateVerify signature
    • Derives session keys
    • Sends Finished message
    • Can immediately send application data!
  4. Encrypted Data (Step 4+):
    • All subsequent data encrypted with AEAD cipher
    • Sequence numbers prevent replay
    • Session keys provide forward secrecy

748.5.3 0-RTT Resumption (Pre-Shared Key)

When clients have previously connected and cached session data, they can resume with zero round-trip latency:

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#E67E22', 'secondaryColor': '#16A085', 'tertiaryColor': '#E67E22'}}}%%
sequenceDiagram
    participant C as Client
    participant S as Server

    rect rgb(22, 160, 133)
        Note over C: Client has cached<br/>session ticket (PSK)
    end

    C->>S: ClientHello + PSK + Early Data
    S->>C: ServerHello + Finished

    rect rgb(230, 126, 34)
        Note over C,S: Application data sent<br/>in first message!
    end

Figure 748.5

0-RTT Benefits: - Latency: Application data sent immediately (no waiting for handshake) - Battery: Fewer packets = less radio-on time - User experience: Faster page loads, quicker sensor responses

Warning0-RTT Security Warning

0-RTT is vulnerable to replay attacks. An attacker can capture the first message and retransmit it, causing the server to process the early data multiple times.

Safe for: - GET requests (idempotent) - Cache lookups - Read-only queries

NOT safe for: - POST/PUT requests - Database writes - Financial transactions - IoT actuator commands (door unlock, motor start)

Mitigation: Servers should mark 0-RTT data as potentially replayed and only process idempotent operations.

748.5.4 TLS 1.3 vs TLS 1.2 Handshake Comparison

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#E67E22', 'secondaryColor': '#16A085', 'tertiaryColor': '#E67E22'}}}%%
flowchart TB
    subgraph TLS12["TLS 1.2 (2-RTT)"]
        A1[ClientHello] --> B1[ServerHello]
        B1 --> C1[Certificate]
        C1 --> D1[ServerKeyExchange]
        D1 --> E1[ServerHelloDone]
        E1 --> F1[ClientKeyExchange]
        F1 --> G1[ChangeCipherSpec]
        G1 --> H1[Finished]
        H1 --> I1[Encrypted Data]
    end

    subgraph TLS13["TLS 1.3 (1-RTT)"]
        A2[ClientHello + KeyShare] --> B2[ServerHello + KeyShare]
        B2 --> C2[Certificate + Finished]
        C2 --> D2[Client Finished]
        D2 --> E2[Encrypted Data]
    end

    TLS12 --> Decision{IoT Device?}
    TLS13 --> Decision
    Decision -->|Yes| Recommend["Prefer TLS 1.3<br/>50% less latency<br/>Mandatory forward secrecy"]

    style A1 fill:#2C3E50,color:#fff
    style A2 fill:#16A085,color:#fff
    style Recommend fill:#E67E22,color:#fff

Figure 748.6
Feature TLS 1.2 TLS 1.3
Round Trips 2-RTT 1-RTT (or 0-RTT resumption)
Messages 9+ messages 3-4 messages
Forward Secrecy Optional (DHE/ECDHE) Mandatory
Cipher Suites 300+ (many weak) 5 (all AEAD)
RSA Key Exchange Allowed Removed
CBC Mode Allowed Removed
Compression Allowed Removed (CRIME attack)

IoT Recommendation: Always use TLS 1.3 (or DTLS 1.3) when available for: - 50% faster handshake (critical for battery life) - Mandatory forward secrecy (protects past sessions if key compromised) - Simplified cipher suite selection (fewer configuration errors)

748.6 Summary

DTLS 1.2 Handshake: - 3-RTT with cookie mechanism for DoS protection - 6-10 messages total - ~250ms on typical networks

DTLS 1.3 Improvements: - 1-RTT handshake (2-RTT with cookie) - Mandatory forward secrecy - AEAD-only ciphers (no weak options) - 0-RTT resumption available (use cautiously)

TLS 1.3 Influence: - DTLS 1.3 mirrors TLS 1.3 security model - Same cipher suites, same key exchange - Adapted for UDP’s unreliability

Battery Impact: - DTLS 1.3 saves 100-200ms per connection - Session resumption reduces overhead by 60% - Critical for battery-powered IoT devices

748.7 What’s Next?

Continue to DTLS Attack Scenarios and Authentication to learn about common attacks against DTLS, defense mechanisms, and how to choose between PSK and certificate-based authentication for your IoT deployment.