15  Inside the DTLS Handshake

Keywords

DTLS handshake, DTLS cookie exchange, DTLS message flights, DTLS retransmission, DTLS session resumption

15.1 Start With the First Flight

A DTLS handshake is a sequence of evidence, not a single success flag. Begin with the first flight: who sent it, what route validation is needed, what retransmission behavior is expected, and what identity proof will be accepted. Resumption and optimization make sense only after the full handshake story is clear.

15.2 Overview: A DTLS Handshake Is Evidence, Not Just Success

Datagram Transport Layer Security, or DTLS, establishes a protected session for traffic that uses UDP-style datagrams. A completed handshake is useful, but it is not the whole review. The evidence needs to show which peer identity was accepted, how the message flights behaved, whether route validation applied, and where the protected session boundary begins.

That matters for IoT because the handshake often crosses lossy links, constrained gateways, NAT or firewall boundaries, and devices with long-lived credentials. A reviewable record keeps the cryptographic claim and the datagram behavior together.

For example, a field bridge might connect to gateway gw-03 using a provisioned PSK identity bridge-12. A useful handshake record does not stop at “connected.” It says the first client flight was observed, the gateway required the expected reachability proof before expensive state was committed, the accepted server flight matched the configured policy, the bridge proved possession of the expected identity material, transcript confirmation completed, and only then did protected telemetry begin. If the trace also shows one lost flight and a retransmission that converged without changing the selected policy, the reviewer can separate normal datagram recovery from a security exception. The result is a bounded claim: this peer and path established a DTLS session under this policy, with these limits and retest triggers.

The handshake evidence path should connect observed flights, validation behavior, identity checks, transcript confirmation, and the decision to allow protected application data.

Candidate Scope

Name the endpoint role, credential set, DTLS policy, peer identity, and traffic path under review.

Message Flights

Record the observed handshake exchanges and any loss, duplicate, or retransmission behavior.

Peer Evidence

Show the accepted certificate, pre-shared-key identity, pinned key, or locally approved identity model.

Boundary

State when protected application data begins and which later changes require a new review.

15.3 Practitioner: Review Flights, Validation, and Retransmission

DTLS is designed for datagram transport, so a handshake review should expect datagram questions. UDP datagrams can be lost, duplicated, reordered, or sent with a spoofed source address. A strong record says what behavior was expected for this path and what the trace or logs actually showed.

Build the review around paired positive and negative evidence. For bridge-12, the positive path might show a normal handshake after the route-validation step and a protected telemetry record accepted by the gateway. The negative path should show at least one configured refusal or hold point: a stale PSK identity rejected, a missing reachability proof denied, a disallowed parameter set refused, or a retransmission pattern that did not bypass the policy. Capture timestamps do not need to prove every cryptographic primitive. They need to support the operational claim: the configured DTLS path accepts the intended peer, handles normal datagram loss, and refuses material or behavior outside the release policy.

Review item

Evidence question

Retest trigger

Route validation

Was a cookie, retry, or equivalent reachability check expected and observed for this peer path?

Endpoint address, server policy, gateway path, DTLS version policy, or validation secret changes.

Message flights

Which handshake flights were observed, and did retransmission or reordering change the result?

Retransmission timer, lossy-link profile, packet path, implementation, or proxy behavior changes.

Parameter selection

Did the selected version, protection parameters, and peer identity match the configured policy?

Cipher policy, credential material, identity store, trust anchor, or endpoint role changes.

Transcript confirmation

Was the handshake transcript confirmed before protected application data was accepted?

Handshake library, policy, identity material, or message-processing behavior changes.

Evidence record shape

For each candidate, record the role, credential reference, route-validation expectation, observed flights, selected parameters, authentication evidence, transcript confirmation, decision, owner, and retest trigger. The goal is repeatable review, not a packet-by-packet implementation tutorial.

15.4 Under the Hood: Keep Full Handshake and Resumption Claims Separate

A full handshake and a resumed session can both be valid, but they do not prove the same thing. Full-handshake evidence usually covers fresh negotiation, peer authentication, selected protection parameters, transcript confirmation, and the first protected-data boundary. Resumption evidence should show the ticket or pre-shared-key identity, freshness policy, replay-sensitive operation decision, and which full-handshake evidence it depends on.

The separation matters after outages and rotations. Suppose bridge-12 performs a full handshake on Monday, then resumes a session after a gateway restart on Tuesday. The Monday record can support the original identity and policy decision, but the Tuesday record still needs to name the resumption identity, freshness rule, server-side acceptance behavior, and whether any early application data is allowed. If credential material rotated overnight, or if the gateway moved behind a different NAT path, the reviewer should not inherit Monday's evidence silently. A resumed telemetry retry may be acceptable under a freshness rule; a resumed actuator command, provisioning operation, or billing-like event may need a stricter replay-sensitive decision or a fresh full handshake.

The review record keeps cryptographic negotiation, datagram behavior, and retest triggers in one place so the claim does not outgrow the evidence.

Scope the candidate Endpoint role, credential reference, DTLS policy, peer identity, and traffic path.

Observe the flights Initial messages, validation step, retransmission behavior, and selected parameters.

Confirm the boundary Transcript confirmation before protected application data is accepted.

Separate resumption Ticket or PSK identity, freshness policy, early-data decision, and replay-sensitive limits.

Retest on change Reopen the record after credential, policy, route, retransmission, endpoint, or resumption changes.

Do not overstate early data

When a resumption mode allows application data before the full handshake boundary is re-established, the review must decide whether that data is safe for the specific operation. Telemetry retry, command execution, provisioning, and billing-like events can have different replay consequences.

15.5 Summary

A DTLS handshake review should prove more than eventual connection success. It should record the endpoint scope, peer identity evidence, observed message flights, route-validation behavior, selected parameters, transcript confirmation, and the boundary where protected application data begins.

For IoT systems, the datagram behavior is part of the security evidence. Loss, retransmission, route validation, resumption, and early-data decisions can all change what the handshake result actually proves.

15.6 Key Takeaway

Keep DTLS handshake evidence bounded: full handshake, resumption, route validation, retransmission, identity, and protected-data decisions are related, but each claim needs the evidence and retest trigger that fits that exact path.

15.7 See Also

DTLS Fundamentals DTLS for IoT DTLS Threats and Authentication DTLS Cost and Performance