15 Analyzing IoT Protocols

testing-validation

traffic-analysis

protocols

Keywords

IoT protocol analysis, traffic analysis, MQTT review, CoAP review, packet evidence

Protocol analysis reads captured traffic as a structured conversation rather than a pile of packets. Instead of asking only whether packets exist, you decode each message and ask whether it matches the expected protocol role, topic or path, method or type, sequence state, response, and error handling. Done well on devices and networks you are authorized to test, it turns a capture into evidence about how your IoT system actually communicates.

A note on scope before anything else: everything here assumes you are analyzing traffic on networks and devices you own or are explicitly authorized to test, for debugging, validation, and security hardening. Capturing or decoding other people’s traffic without authorization is a different matter and is not what this chapter is about.

15.1 Start Simple: Read the Conversation

Once a packet is visible, the next question is what part of the conversation it belongs to. A publish, request, acknowledgement, reset, or response code tells a different story than a raw packet count. The learner’s job is to translate the traffic into a short claim: what was sent, who could see it, and what reply would prove the next step.

Keep that story small. A broker-side MQTT capture can prove that a publish reached the broker; it cannot prove every subscriber acted on it. A CoAP request with no visible reply can prove the request left this point; it cannot name the cause without more evidence. The protocol decode is powerful because it is specific, not because it explains everything.

15.2 Decoding the Conversation, Not Just Counting Packets

A packet capture answers “what crossed this point, when, and in what order.” Protocol analysis goes a step further: it decodes each packet into the structured message the protocol defines — a connect, a publish, a request, a response, an acknowledgement — and checks that message against the contract you expected. For IoT that usually means application protocols such as MQTT and CoAP and link-layer protocols such as BLE and Zigbee, each with its own message types, addressing, and state rules.

The single most important idea is that a decoded message is still bounded evidence. It is bounded by where the capture was taken (a broker-side capture cannot speak to a subscriber it never saw), by the time window, and by what is readable — encrypted payloads hide their contents even when the message envelope is visible. And a subtle but crucial rule runs through all of it: “message observed” is not “message accepted.” Seeing a publish leave a device does not prove a subscriber received it; the receiver’s acknowledgement or response code is what carries that evidence.

If you only need the intuition, this layer is enough: decode each message to its protocol role and compare it against the expected contract, but bound every finding to the capture point, the time window, and what was readable. Do this only on systems you are authorized to test — for debugging, validation, and security — and record what the capture could not see.

Think of reading a transcript of a meeting you were authorized to record. You can check who spoke, in what order, and whether a reply came — that is protocol decoding. But you cannot hear what was said in a sealed side conversation (an encrypted payload), and you cannot know what someone who was off-mic (outside your capture point) said at all. The transcript is real evidence, precisely bounded by the room and the microphone.

Protocol analysis evidence runs from a bounded capture through the decode and expected conversation to an observed message, then names the gap, the decision, and the retest trigger.

The One-Minute View

Decode to the contract

Read each message as a protocol role — topic, path, method, type — and compare it against what you expected.

Bound to the capture

The finding holds only for the capture point, time window, and the parts that were readable.

Observed is not accepted

A message seen leaving a device is not proof a receiver got it; the response or ack carries that evidence.

Beginner Examples

An MQTT publish is decoded to confirm it uses the expected topic and message role after a local input changes — on a broker you administer.
A CoAP exchange is checked so the method and path match the contract and a response code comes back.
A capture shows a device’s request but no response, so the finding is recorded as “request sent, no visible response” rather than a guess at the cause.

Overview Knowledge Check

If you can decode a message to its contract and bound the claim to the capture, you have the core idea. Continue to Practitioner for the protocol families and a reviewable record.

15.3 Protocol Families and the Review Record

Different IoT protocols carry different conversation state, so the decode you check depends on the family. The table names what each family puts on the wire and what to compare against the contract; remember that the radio protocols below require a link-layer or radio capture, not an IP-interface capture.

Family

Typical Transport

Conversation State To Decode

Common Gap

MQTT

TCP, often over TLS, via a broker.

Connect, publish, subscribe, QoS acknowledgement, retained and will messages.

Broker-side capture cannot show subscriber delivery.

CoAP

UDP, often over DTLS.

Method, path, options, response code, observe, confirmable and acknowledgement.

UDP loss and retries blur “no response” causes.

BLE (link layer)

Bluetooth Low Energy radio.

GATT services and characteristics, reads, writes, notifications, pairing.

Needs radio capture on the right channel; pairing may encrypt.

Zigbee / mesh

Low-power mesh radio.

Join and association, binding, cluster and command, acknowledgement.

Link-layer encryption and mesh routing hide hops.

Keep Each Decode Tied to the Record

A protocol finding should be short, traceable, and repeatable. Capture, for each finding: the scope (device, interface, capture point, and the firmware or configuration under test), the decode (protocol, message role, topic or path, sequence state), the evidence (expected message, observed message, response, and any related device log), the gap (encrypted content, a missing capture point, filtered setup traffic, or an undocumented format), the decision (approve, hold, retest, or request another capture), and the reopen rule (a changed artifact, decoder, capture point, or message contract).

A protocol review record groups scope, decode, evidence, gap, decision, and the reopen rule so a finding stays auditable.

Worked Example: An MQTT Publish and a CoAP Response Gap

Suppose a device should publish a status message when a local input changes, captured between the device and a broker you administer. A weak record says “MQTT capture looks normal, publish was visible, no issue.” A strong record states the capture point is device-to-broker, the decoded publish uses the expected topic and role, the acknowledgement state matches the delivery setting under review, the payload is either readable and checked against the contract or explicitly marked unreadable, and whether subscriber delivery was inside or outside the capture scope. Now suppose a device sends a CoAP request and the application reports a timeout. The decode must distinguish three different findings: “request not sent” (nothing left the device), “request sent, no visible response” (the request is on the wire but no reply appears at any capture point), and “response visible but not accepted by the device.” Each points at a different cause, and over UDP you must also weigh loss and retries before blaming any one component.

Practitioner Knowledge Check

If you can decode to the right family’s contract and keep each finding tied to a bounded record, you can stop here. Continue to Under the Hood for decoders, encryption, and conversation state.

15.4 Decoders, Encryption, and Conversation State

The deeper layer is about what a decode can and cannot recover. Three mechanisms matter: how a dissector turns bytes into fields, what encryption does to the evidence, and how conversation state and missing context shape the conclusion.

Dissectors Map Bytes to Fields Using the Spec

A protocol decoder — often called a dissector — reads the raw bytes and, using its knowledge of the protocol’s structure, presents them as named fields: message type, topic, method, options, response code. This works because the protocol is documented and the bytes follow it. A custom or proprietary format has no public structure, so a generic decoder shows it as opaque bytes; interpreting it requires a documented description of the format. Treating a guessed interpretation of an undocumented payload as fact is a common error — if the format is unknown, the honest record says so.

Encryption Hides the Payload, Not Always the Envelope

Most production IoT links are encrypted — TLS for MQTT, DTLS for CoAP, link-layer encryption for BLE and Zigbee — and that is a security feature, not a defect. For analysis it means the message contents are not readable from the capture even when envelope metadata (endpoints, sizes, timing) is. On systems you control and are authorized to test, you can sometimes decrypt with keys you legitimately hold for your own devices, which is a normal part of validating your own product; you do not have, and should not seek, the keys for systems you are not authorized to test. When a payload is encrypted and you cannot decrypt it, record the finding as metadata-only and bound the conclusion accordingly.

Conversation State and Missing Context

Many findings are really about the state of a conversation: an exchange that stops at a specific step, a retry that never resolves, a response code that does not match the request. Reading state correctly requires the surrounding messages, so a capture that filtered out the setup handshake or sits at only one of two relevant points can make a normal exchange look broken — or a broken one look fine. This is why the capture point and the filter scope belong in every record, and why “no response” must be qualified by whether a response could even have appeared at the point you captured.

Risk

How It Misleads

Discipline

Hands Off To

Guessed payload

An undocumented format is read as if its meaning were known.

Mark unknown formats opaque until documented.

A documented format description.

Encrypted contents

A hidden payload is treated as if it were inspected.

Record metadata-only; decrypt only with authorized keys.

Device logs or an endpoint you control.

Observed read as accepted

A sent message is taken as a received one.

Require the response or acknowledgement for acceptance.

A receiver-side capture or log.

Filtered setup

Missing handshake makes a normal exchange look broken.

Record filter scope; capture the setup when state matters.

A wider capture including setup.

Single capture point

One vantage point’s blind spot becomes a false conclusion.

Bound the claim to the point; add a second point if needed.

A second capture point between components.

Common Pitfalls

Guessing an undocumented payload. Without a format description, the bytes are opaque, not understood.
Treating encrypted traffic as inspected. Metadata is visible; contents are not, unless you hold authorized keys.
Confusing observed with accepted. A sent message needs a response or ack to show it was received.
Filtering out the setup. Missing handshakes distort the apparent conversation state.
Over-generalizing one point. A clean decode at one vantage point is not a conclusion about every path.

Under-the-Hood Knowledge Check

At this depth, protocol analysis is disciplined decoding bounded by what the capture could see: map bytes to the contract, treat encrypted contents as metadata-only unless you hold authorized keys, never read “observed” as “accepted,” and keep the capture point and filter scope in every record. The strongest protocol evidence states the decoded conversation, the missing or unreadable context, and the specific additional capture or log it handed onward — always on systems you are authorized to test.

15.5 Summary

Protocol analysis decodes captured packets into the protocol’s structured messages and compares each against the expected contract, rather than only counting packets.
A decoded message is bounded evidence: it holds only for the capture point, the time window, and the parts that were readable.
IoT spans application protocols such as MQTT and CoAP and link-layer protocols such as BLE and Zigbee, each with its own conversation state to decode, and the radio protocols need a radio or link-layer capture.
“Message observed” is not “message accepted”: a sent message needs a response or acknowledgement before a delivery or acceptance claim is supported.
A reviewable finding records scope, decode, evidence, gap, decision, and a reopen rule, and distinguishes “not sent,” “sent with no visible response,” and “response not accepted.”
Dissectors map bytes to fields using the protocol spec, so undocumented or custom formats are opaque until a format description exists; do not report a guess as contents.
Encryption hides payloads while leaving metadata visible; decrypt only with keys you legitimately hold for your own devices, and otherwise record findings as metadata-only.

Key Takeaway

Protocol traffic analysis should decode enough structure to explain identity, direction, timing, payload class, errors, and security boundaries — on systems you are authorized to test. A finding is evidence only when each decoded message is tied to the capture point, the readable scope, and the specific missing context it could not close.