12 DTLS: Handshake and UDP Protection

12.5.1 Start With the First Flight

Picture a small device that must set up a safe link over a lossy radio. A final success line does not show what was checked or retried.

First, save each flight, peer name, check, wait, and result. Keep a full new link apart from a short return to old state.

More retries can help on a weak link, but they spend time, power, and radio space. A short return is cheap, yet it rests on state that may be old or gone.

DTLS protection depends on the observed handshake flights, key state, and tested attack boundary for this UDP exchange.

Use the Practitioner section to review flights and retries. Use Under the Hood to study full setup, return state, loss, and security limits in more depth.

A protocol is a set of rules for an exchange. Transport layer security is a set of checks that protects a link. Datagram Transport Layer Security (DTLS) protects datagrams. User Datagram Protocol (UDP) sends those datagrams without a built-in session.

Plain check

• Save each flight. Save each wait. Save each retry. Save the final app result.
• Use Practitioner to review. Use deeper key checks. Test old return state. State each bound.

A DTLS handshake is a sequence of evidence, not a single success flag. Begin with the first flight: who sent it, what route validation is needed, what retransmission behavior is expected, and what identity proof will be accepted. Resumption and optimization make sense only after the full handshake story is clear.

12.5.2 Overview: A DTLS Handshake Is Evidence, Not Just Success

Datagram Transport Layer Security, or DTLS, establishes a protected session for traffic that uses UDP-style datagrams. A completed handshake is useful, but it is not the whole review. The evidence needs to show which peer identity was accepted, how the message flights behaved, whether route validation applied, and where the protected session boundary begins.

That matters for IoT because the handshake often crosses lossy links, constrained gateways, NAT or firewall boundaries, and devices with long-lived credentials. A reviewable record keeps the cryptographic claim and the datagram behavior together.

For example, a field bridge might connect to gateway gw-03 using a provisioned PSK identity bridge-12. A useful handshake record does not stop at "connected." It says the first client flight was observed, the gateway required the expected reachability proof before expensive state was committed, the accepted server flight matched the configured policy, the bridge proved possession of the expected identity material, transcript confirmation completed, and only then did protected telemetry begin. If the trace also shows one lost flight and a retransmission that converged without changing the selected policy, the reviewer can separate normal datagram recovery from a security exception. The result is a bounded claim: this peer and path established a DTLS session under this policy, with these limits and retest triggers.

To see why a successful connection flag is too weak, trace the actual flight sequence in Figure 12.1 before judging the session. The diagram turns the handshake into observable checkpoints rather than treating it as one opaque event.

+

Read Figure 12.1 as a DTLS 1.2 example: from the client’s first ClientHello to the optional HelloVerifyRequest cookie challenge, then follow the repeated hello into the server and client key-exchange flights. DTLS 1.3 carries its optional cookie challenge in HelloRetryRequest instead. When configured, the cookie exchange limits expensive server work until return reachability is shown; the later flights establish the negotiated session. In either version, protected application data must satisfy the selected handshake and peer-identity policy.

12.5.3 Practitioner: Review Flights, Validation, and Retransmission

DTLS is designed for datagram transport, so a handshake review should expect datagram questions. UDP datagrams can be lost, duplicated, reordered, or sent with a spoofed source address. A strong record says what behavior was expected for this path and what the trace or logs actually showed.

Build the review around paired positive and negative evidence. For bridge-12, the positive path might show a normal handshake after the route-validation step and a protected telemetry record accepted by the gateway. The negative path should show at least one configured refusal or hold point: a stale PSK identity rejected, a missing reachability proof denied, a disallowed parameter set refused, or a retransmission pattern that did not bypass the policy. Capture timestamps do not need to prove every cryptographic primitive. They need to support the operational claim: the configured DTLS path accepts the intended peer, handles normal datagram loss, and refuses material or behavior outside the release policy.

12.5.4 Under the Hood: Keep Full Handshake and Resumption Claims Separate

A full handshake and a resumed session can both be valid, but they do not prove the same thing. Full-handshake evidence usually covers fresh negotiation, peer authentication, selected protection parameters, transcript confirmation, and the first protected-data boundary. Resumption evidence should show the ticket or pre-shared-key identity, freshness policy, replay-sensitive operation decision, and which full-handshake evidence it depends on.

The separation matters after outages and rotations. Suppose bridge-12 performs a full handshake on Monday, then resumes a session after a gateway restart on Tuesday. The Monday record can support the original identity and policy decision, but the Tuesday record still needs to name the resumption identity, freshness rule, server-side acceptance behavior, and whether any early application data is allowed. If credential material rotated overnight, or if the gateway moved behind a different NAT path, the reviewer should not inherit Monday's evidence silently. A resumed telemetry retry may be acceptable under a freshness rule; a resumed actuator command, provisioning operation, or billing-like event may need a stricter replay-sensitive decision or a fresh full handshake.

The full-handshake record becomes useful only when it can be compared with later resumed sessions. Figure 12.2 shows the fields that keep those two claims separate while preserving the evidence needed for either decision.

+

In Figure 12.2, begin with the endpoint and policy scope, then follow the observed flights and validation result into the identity and transcript evidence. Finish at the decision, limit, owner, and retest trigger. A resumed session can reuse previously established state, but it must not be reported as if every full-handshake check ran again; that distinction carries the chapter’s evidence discipline into optimization and release review.

12.5.5 Summary

A DTLS handshake review should prove more than eventual connection success. It should record the endpoint scope, peer identity evidence, observed message flights, route-validation behavior, selected parameters, transcript confirmation, and the boundary where protected application data begins.

For IoT systems, the datagram behavior is part of the security evidence. Loss, retransmission, route validation, resumption, and early-data decisions can all change what the handshake result actually proves.

12.5.6 Key Takeaway

12.5.7 See Also

12.6.1 Start With the Untrusted Datagram

Picture a battery sensor that sends one short alarm without keeping a connection open. Each message is a datagram: one packet sent as a separate item. An attacker on the path should not be able to read it, change it, or replay an old alarm as if it were new.

Start by naming both ends and the key or certificate they trust. Decide how the receiver will spot an old or repeated packet. Then test a normal exchange, a lost packet, a delayed reply, a restart, and a changed key. Keep the message record and the reason each failure was accepted or rejected.

The security layer protects datagram traffic. It does not secure a weak sensor, a stolen key, a bad command rule, or data after the receiver has opened it. Its setup can also cost time, memory, radio use, and battery energy. Those costs matter on a small device.

Go deeper in two steps. The Practitioner section builds a secure-datagram review record. Under the Hood follows setup messages, protected records, replay checks, and the limits of separate packets.

Keep the first claim narrow. State that the tested message was hidden, unchanged, tied to the expected peer, and checked for old copies. Do not claim that every later app choice is safe. The receiver still has to check the sender’s right and the message value.

Measure the setup on the real device. Keep bytes, time, memory, and energy. Repeat it after a lost packet and a restart. A small board may pass the security check yet miss its response or battery goal. Both results belong in the same release choice.

Use a short test card. Name both ends. Name the trust item they share. Name the message and the action it may cause. State how the receiver spots an old packet. State when the session must end.

Capture one good setup and one good data exchange. Then lose a setup packet. Repeat a data packet. Change it. Delay it. Restart one end. Use the wrong trust item. Record which layer rejects each case.

Finish at the app. Check who may send the command, whether the value is in range, and whether the action is still safe. Transport proof and app proof belong in the same review, but they are not the same claim.

Start with a UDP datagram that crosses a network you do not fully trust. DTLS can protect the record, bind it to a peer identity, and limit replay, but it does not decide whether the application should trust the command itself. The security review follows that boundary from handshake evidence to application acceptance.

12.6.2 Overview: DTLS Protects Datagrams, Not the Whole System

Datagram Transport Layer Security, or DTLS, adapts TLS-style protection for UDP traffic. It is useful when an IoT design wants datagram behavior with encrypted, authenticated records. The selected credential mode and endpoint policy determine peer authentication, while replay handling depends on the configured record policy.

The main review mistake is treating "uses DTLS" as a complete security conclusion. DTLS can protect a transport exchange under a specific session context. It does not replace application authorization, device trust, payload validation, key operations, or service monitoring.

For example, a battery meter might send a CoAP reading over UDP to a site gateway. A DTLS fundamentals review should separate four facts: the meter and gateway completed the intended handshake, the later CoAP request appeared as protected DTLS records, a duplicate or stale protected record did not create a second accepted reading, and the application still made its own decision about whether that reading was plausible and authorized. If the record only says "DTLS on port 5684," it misses the useful boundary. If it says "meter-09 to gateway-east accepted PSK identity meter-09, protected the reading exchange, rejected a duplicate record before delivery to the CoAP handler, and left calibration validity to the application review," the claim becomes testable.

To keep DTLS’s protection claim separate from application trust, inspect where the record layer sits in Figure 12.3. The stack placement explains both what DTLS adds and what remains above or below its boundary.

+

Read Figure 12.3 from UDP and IP upward into the DTLS record layer, then continue to the application protocol. Inside the DTLS layer, identify encryption, integrity, and peer-authentication functions; outside it, keep datagram loss and application authorization as separate questions. This layered reading anchors the chapter’s claim that protected records do not prove the whole system outcome.

12.6.3 Practitioner: Build a DTLS Review Record

A useful DTLS fundamentals review record does not ask only whether DTLS is present. It names the protected traffic, endpoint roles, observation point, handshake context, record evidence, replay evidence, limits, owner, and retest trigger.

Start with a narrow claim sentence before collecting packet captures. For the meter example, the claim might be: "uplink readings from meter-class devices to the gateway are protected by DTLS and duplicate protected records do not reach the CoAP application as new readings." That sentence tells the reviewer where to look: the gateway capture, the endpoint log that binds the session to the expected identity, a normal protected request, a replay or duplicate-record attempt, and the application outcome. It also prevents overclaiming. The same evidence cannot prove device calibration, business-rule authorization, or cloud-side retention.

The architectural boundary needs a review record that ties it to a real endpoint pair and session. Figure 12.4 shows the fields that keep that evidence narrow and repeatable.

+

In Figure 12.4, begin with scope and handshake identity, then follow the protected record and replay evidence. Finish with limits, decision, owner, and retest trigger. That order connects a packet trace to its session context while preserving the application and operational questions DTLS cannot answer for the reviewer.

For a protected CoAP request, a defensible conclusion might be: "This request was carried inside a DTLS-protected session with the recorded peer identity and protection policy." That conclusion does not decide whether the CoAP action was authorized. It only states the transport-protection evidence.

If a record claims replay handling but only shows one normal protected record, hold the replay claim. Ask for evidence that a repeated or stale protected record is rejected, ignored, or recorded according to policy, and confirm whether the application receives duplicate data.

12.6.4 Under the Hood: Handshake, Records, Replay, and Datagram Limits

The handshake establishes session state: endpoint identity evidence, selected version, selected protection policy, and keys for later records. A completed handshake is important, but it is not the same as a full application security decision.

Protected records carry encrypted and integrity-checked application data after the handshake. Reviewers should connect each record to the session that produced it and keep epoch, sequence, replay policy, and acceptance outcome visible.

The datagram substrate is why those fields matter. UDP datagrams can be lost, duplicated, or reordered; oversized IP packets may also be fragmented below UDP. A DTLS review has to distinguish handshake retransmission from application-level duplicate effects. A practical trace might show handshake messages retransmitted while the peer is still establishing state, then protected application records under one epoch and sequence progression once the session is active. If record sequence 41 is accepted, a replayed copy of sequence 41 should be rejected, ignored, or logged according to policy, while sequence 42 can still be accepted when it is authentic and within the replay window. The reviewer should not infer those outcomes from encryption alone; they need transport evidence and, when the claim mentions business effects, the application log that shows whether a second reading, command, or actuator event was created.

Retest triggers keep evidence current. Reopen the review when endpoint roles, identity material, protocol stack, firmware, protection policy, replay policy, observation point, or application authorization logic changes.

12.6.5 Summary

DTLS adapts TLS-style transport protection for UDP. Its architecture matters because handshake context, protected records, datagram behavior, replay policy, and application boundaries answer different review questions.

A strong DTLS fundamentals record names the protected traffic, peer identity model, negotiated protection policy, protected record evidence, replay evidence when claimed, open limits, owner, and retest trigger. It avoids turning transport protection into a broader security conclusion.

12.6.6 Key Takeaway

12.6.7 See Also