13  Packet Structure and Framing

In 60 Seconds

Every IoT message rides inside a packet with three core parts: a header for addressing and control, a payload for the actual sensor data, and often a trailer for integrity checks such as a checksum or CRC. Framing tells the receiver where each packet starts and ends, either with explicit lengths or special delimiter bytes. In constrained IoT links, these structural bytes matter: a 4-byte sensor value can become a 17-byte LoRaWAN frame or a 68-byte Ethernet/IP/TCP packet once all protocol overhead is added.

A network packet is like a postal envelope for digital data. It has a header (the address label telling where to deliver it), a payload (the actual letter inside), and a trailer (a seal to verify nothing was tampered with). Every piece of information sent between IoT devices – from a temperature reading to a command to turn on a light – travels inside these structured packets. Understanding their structure helps you appreciate how data moves reliably across networks.

Learning Objectives

By the end of this chapter series, you will be able to:

  • Dissect packet anatomy: Identify and label headers, payloads, and trailers in network packets across IoT protocols
  • Compare framing techniques: Differentiate between length-field and delimiter-based framing, and evaluate when each approach is appropriate
  • Compute error detection values: Calculate checksums, apply CRC polynomial division, and select the appropriate error detection method for a given reliability requirement
  • Interpret protocol specifications: Read frame format diagrams in datasheets and map byte offsets to field definitions
  • Extract packet header fields: Parse binary packet data to retrieve source/destination addresses, sequence numbers, and control flags
  • Evaluate protocol overhead: Quantify header-to-payload ratios for different IoT protocol stacks and justify protocol selection for constrained devices
  • Design custom packet formats: Construct a basic packet structure with appropriate framing, addressing, and error detection for a given IoT application
Key Concepts
  • Core Concept: Fundamental principle underlying Packet Structure and Framing — understanding this enables all downstream design decisions
  • Header: Routing and control bytes that identify source, destination, ports, flags, sequence numbers, or protocol type
  • Payload: The actual application data, such as a temperature reading, command, or telemetry sample
  • Trailer / FCS / MIC: Integrity bytes appended at the end of the frame to detect corruption or tampering
  • Length-Field Framing: A framing method where the header states how many bytes belong to the current packet
  • Delimiter Framing: A framing method using special start or end markers, often paired with byte stuffing to escape those markers in the payload
  • MTU: Maximum Transmission Unit — the largest frame or packet a link can carry without fragmentation
  • Protocol Overhead: Bytes spent on structure rather than user data; often the dominant cost on constrained IoT links
  • CRC: Cyclic Redundancy Check — the most common strong error-detection mechanism for IoT frames and packets

13.1 MVU: Minimum Viable Understanding

If you only have 5 minutes, here’s what you need to know about packet structure for IoT:

  1. Every packet has three parts: Header (addressing/control), Payload (your actual data), and Trailer (error checking)
  2. Framing tells receivers where packets start and end: Either by length fields (“next 50 bytes are mine”) or delimiter bytes (special markers like 0x7E)
  3. Error detection catches corrupted data: CRC is more reliable than checksums - use CRC-16 for most IoT, CRC-32 for safety-critical
  4. Protocol overhead matters for constrained devices: A 10-byte sensor reading can balloon to 60+ bytes after all headers are added

Bottom line: Understanding packet structure helps you choose efficient protocols for bandwidth-limited IoT networks, debug communication failures, and design custom protocols when needed.

Key Takeaway

In one sentence: Network packets wrap your sensor data in headers for routing, payloads for the actual information, and trailers for error detection - all bounded by framing mechanisms that tell receivers where each packet begins and ends.

Remember this rule: Smaller packets = less overhead per message, but more fragmentation for large data. Balance payload size against your protocol’s Maximum Transmission Unit (MTU).

Meet the Data Delivery Team!

Imagine you’re sending a present to your friend. You can’t just throw the toy in the mail - you need to wrap it up properly!

The Three Parts of Every Package (Packet):

  • Header = The address label on your package. It tells the postal service WHERE to deliver it and WHO it’s from
  • Payload = The actual present inside the box - the thing you really wanted to send!
  • Trailer = A seal or sticker that proves nobody opened your package. If the seal is broken, you know something went wrong!

How Does the Post Office Know Where One Package Ends?

  • Length field: Like writing “THIS BOX IS 30 CM LONG” on the package
  • Delimiters: Like putting a bright red ribbon at the start AND end of your present

What If Your Present Gets Damaged?

  • Checksum: Like counting all the jellybeans in a jar - if the count doesn’t match, some are missing!
  • CRC: A super-smart math trick that catches almost ANY damage, even if jellybeans get swapped around!

Fun Fact: When you watch a video online, your computer receives MILLIONS of tiny packets and puts them back together like a jigsaw puzzle!

Learning Resources:

  • Quizzes Hub - Test your packet structure knowledge with MCQs on framing, checksums, and protocol overhead
  • Simulations Hub - Experiment with packet fragmentation and reassembly in interactive demos
  • Videos Hub - Watch Wireshark tutorials dissecting real network packets
  • Knowledge Gaps Hub - Address common misconceptions about MTU, overhead, and error detection
  • Knowledge Map - See how packet structure connects to protocols like MQTT, CoAP, and LoRaWAN

Prerequisites (Read These First):

Fundamentals (Core Concepts):

Networking Foundations:

Protocol-Specific Packet Structures:

  • MQTT - MQTT fixed/variable headers, QoS, and packet types
  • CoAP - CoAP 4-byte header and message format
  • LoRaWAN - LoRa MAC header (MHDR), DevAddr, FCnt, MIC

13.2 Chapter Overview

This chapter series covers the fundamental concepts of packet structure, framing, error detection, and protocol overhead that underpin all network communication. Understanding these concepts is essential for designing efficient IoT systems, debugging network issues, and choosing appropriate protocols.

The content has been organized into four focused chapters:

Packet structure journey diagram showing how a raw sensor reading becomes a payload, is wrapped with header and trailer bytes into a packet, framed for transmission on a specific link, then interpreted by the receiver as packet anatomy, framing, error detection, and protocol overhead
Figure 13.1: Packet structure overview showing the journey from sensor data to network transmission

13.3 Chapter Series

13.3.1 1. Packet Anatomy: Headers, Payloads, and Trailers

Read: Packet Anatomy ->

Learn the three fundamental parts of every network packet:

  • Headers: Routing, identification, and control information (10-40 bytes)
  • Payloads: The actual data being transmitted (variable size)
  • Trailers: Error detection values like checksums and CRC (2-4 bytes)

Includes a real-world LoRaWAN temperature sensor packet dissection and the postal letter analogy for beginners.


13.3.2 2. Frame Delimiters and Boundaries

Read: Frame Delimiters and Boundaries ->

Understand how receivers detect packet boundaries:

  • Length fields: Header specifies payload size (used by IP, TCP, UDP, MQTT)
  • Delimiters: Special marker bytes signal start/end (used by Ethernet, HDLC, PPP)
  • Byte stuffing: Escape sequences handle delimiter conflicts in payload data
  • MTU and fragmentation: How large payloads get split across network limits

Includes an interactive packet fragmentation demo showing 6LoWPAN, IPv4, IPv6, and LoRaWAN fragmentation behavior.


13.3.3 3. Error Detection: Checksums and CRC

Read: Error Detection ->

Compare methods for ensuring data integrity:

  • Simple checksums: Fast but weak - can miss transposed bytes
  • CRC-16/CRC-32: Polynomial-based detection catching 99.9999% of errors
  • When to use each: Trade-offs between computation and reliability
  • Safety-critical systems: Why industrial protocols use CRC-32

Includes scenario-based practice designing error detection for industrial sensors and debugging corrupted Zigbee packets.


13.3.4 4. Protocol Overhead and Encapsulation

Read: Protocol Overhead and Encapsulation ->

Analyze and optimize protocol efficiency:

  • Header comparison: BLE (57% overhead) vs MQTT/TCP/IPv6 (86% overhead)
  • Encapsulation: The “Russian doll” model of protocol layering
  • Cost analysis: How overhead impacts bandwidth costs and battery life
  • Protocol selection: Choosing the right stack for constrained IoT devices

Includes overhead calculations for LoRaWAN deployments and TCP vs UDP decision scenarios.


13.4 Quick Reference: Packet Structure Summary

Component Purpose Typical Size
Header Routing, control, identification 10-40 bytes
Payload Actual data (sensor readings, commands) Variable (0-MTU)
Trailer Error detection (checksum/CRC) 2-4 bytes
Framing Method When to Use Examples
Length field Known payload size IP, TCP, UDP, MQTT, CoAP
Delimiters Variable data, controlled content Ethernet, HDLC, Serial
Byte stuffing Arbitrary binary data PPP, SLIP, binary protocols
Error Detection Detection Rate Use Case
Checksum (8-bit) 100% single-bit; ~99.6% random errors Legacy, non-critical
CRC-16 99.998% General IoT
CRC-32 99.9999% Safety-critical
Check Your Understanding: Match Packet Concepts

Match each packet component or technique to its correct description:


13.5 Start Learning

Begin with the fundamentals and progress through the series:

  1. Packet Anatomy - Start here for the basics
  2. Frame Delimiters - Learn boundary detection
  3. Error Detection - Understand data integrity
  4. Protocol Overhead - Master efficiency analysis

Begin with Packet Anatomy ->


13.6 Packet Structure Visualization

The following diagram illustrates how protocol headers stack during encapsulation - the “Russian doll” model where each layer wraps the previous layer’s data:

Vertical encapsulation diagram showing a 10-byte application payload wrapped by a 20-byte TCP header into a 30-byte segment, wrapped by a 20-byte IPv4 header into a 50-byte packet, then wrapped by a 14-byte Ethernet header and 4-byte FCS into a 68-byte Ethernet frame
Figure 13.2: Protocol encapsulation showing how a 10-byte sensor reading becomes a 68-byte transmitted packet

Overhead Analysis: A 10-byte sensor reading transmitted over Ethernet/IP/TCP becomes 68 bytes on the wire: Ethernet header (14) + IP header (20) + TCP header (20) + Payload (10) + Ethernet FCS (4) = 68 bytes, representing 85% overhead (58 bytes of overhead out of 68 total). This is why IoT protocols like CoAP/UDP and LoRaWAN are preferred for constrained devices.

Why overhead matters even more for IoT: Unlike traditional networks with 1500-byte Ethernet MTUs, IoT networks face extreme constraints. LoRaWAN has a maximum payload of just 51 bytes at the lowest data rate (SF12), meaning every overhead byte directly displaces sensor data. NB-IoT and other cellular IoT networks charge per byte transmitted, so protocol overhead translates directly to operating cost. Additionally, radio transmission consumes significant power on battery-powered devices — larger packets mean longer transmit times and faster battery drain, though sleep current often dominates total energy consumption over the device lifetime.

Check Your Understanding: Encapsulation Order

When a sensor reading is transmitted over Ethernet/IP/TCP, the protocol layers add headers in a specific order. Arrange these steps from first (innermost) to last (outermost) in the encapsulation process:

Let’s calculate the real impact of protocol overhead on IoT deployments with concrete numbers.

Scenario: 1,000 temperature sensors sending 4-byte readings every 15 minutes over LoRaWAN.

Payload Analysis:

\[ \text{Data per message} = \begin{cases} \text{Sensor value:} & 4 \text{ bytes} \\ \text{LoRaWAN header:} & 13 \text{ bytes} \\ \text{Total transmitted:} & 17 \text{ bytes} \end{cases} \]

Overhead ratio: \(\frac{13}{17} = 76\%\) of every packet is overhead, not sensor data.

Annual transmission volume:

\[ \begin{aligned} \text{Messages per sensor per year} &= \frac{24 \times 365}{15/60} = 35,040 \text{ messages} \\ \text{Total network messages} &= 35,040 \times 1,000 = 35.04 \text{ million/year} \\ \text{Data transmitted} &= 35.04M \times 17 \text{ bytes} = 595.7 \text{ MB/year} \end{aligned} \]

Cost on cellular IoT (NB-IoT at $0.10/MB): \(595.7 \times \$0.10 = \$59.57\) per year for this overhead.

Energy impact (at 20mW transmit power, 250 kbps):

\[ \begin{aligned} \text{Transmit time per packet} &= \frac{17 \text{ bytes} \times 8}{250,000 \text{ bps}} = 0.544 \text{ ms} \\ \text{Energy per transmission} &= 20 \text{ mW} \times 0.544 \text{ ms} = 0.0109 \text{ mWs} \\ \text{Annual energy (per sensor)} &= 35,040 \times 0.0109 \text{ mWs} = 382 \text{ mWs} = 0.106 \text{ mWh} \end{aligned} \]

With a CR2032 battery (220 mAh at 3V = 660 mWh), transmission alone consumes 0.106/660 = 0.016% of battery per year - sensors could theoretically last 6,000+ years on transmission energy alone. This shows why sleep current (not transmission) dominates IoT battery budgets.

Key insight: For IoT, reducing overhead from 76% to 50% matters less for battery life than reducing sleep current from 10 µA to 1 µA.


13.7 Interactive Calculators

Protocol Overhead Calculator

Calculate the overhead percentage for different packet structures.

Show code
viewof headerSize = Inputs.range([4, 60], {
  value: 13,
  step: 1,
  label: "Header Size (bytes):",
  description: "Total header bytes (all layers)"
})

viewof payloadSize = Inputs.range([1, 200], {
  value: 4,
  step: 1,
  label: "Payload Size (bytes):",
  description: "Actual sensor data"
})

viewof trailerSize = Inputs.range([0, 8], {
  value: 4,
  step: 1,
  label: "Trailer Size (bytes):",
  description: "CRC, checksum, or MIC"
})

totalPacketSize = headerSize + payloadSize + trailerSize
overheadBytes = headerSize + trailerSize
overheadPercentage = (overheadBytes / totalPacketSize * 100).toFixed(1)
efficiencyPercentage = (100 - overheadPercentage).toFixed(1)

Example Configurations:

  • LoRaWAN: Header=13, Payload=4, Trailer=4 → 76% overhead
  • MQTT/TCP/IPv4: Header=50, Payload=20, Trailer=4 → 73% overhead
  • CoAP/UDP/IPv6: Header=48, Payload=20, Trailer=0 → 71% overhead
Transmission Time & Energy Calculator

Calculate how long it takes to transmit a packet and the energy consumed.

Show code
viewof txPacketSize = Inputs.range([10, 300], {
  value: 68,
  step: 1,
  label: "Packet Size (bytes):",
  description: "Total bytes to transmit"
})

viewof dataRate = Inputs.select(
  new Map([
    ["LoRaWAN SF12 (250 bps)", 250],
    ["LoRaWAN SF7 (5.5 kbps)", 5500],
    ["NB-IoT (20 kbps)", 20000],
    ["Zigbee (250 kbps)", 250000],
    ["BLE (1 Mbps)", 1000000],
    ["Wi-Fi 802.11n (150 Mbps)", 150000000]
  ]),
  {
    value: 250000,
    label: "Data Rate:"
  }
)

viewof txPower = Inputs.range([1, 100], {
  value: 20,
  step: 1,
  label: "Transmit Power (mW):",
  description: "Radio power during transmission"
})

txTimeMs = (txPacketSize * 8 / dataRate * 1000).toFixed(3)
energyMws = (txPower * parseFloat(txTimeMs)).toFixed(3)
energyMwh = (energyMws / 3600000).toFixed(6)

Annual Energy for 1 message/hour:

  • Messages per year: 8,760
  • Total energy: mWh
  • Battery remaining after 1 year: %
MTU Fragmentation Calculator

See how a payload gets fragmented across different network MTUs.

Show code
viewof appPayload = Inputs.range([10, 2000], {
  value: 500,
  step: 10,
  label: "Application Payload (bytes):",
  description: "Data to send"
})

viewof networkType = Inputs.select(
  new Map([
    ["LoRaWAN SF12", { mtu: 51, header: 13 }],
    ["LoRaWAN SF7", { mtu: 222, header: 13 }],
    ["6LoWPAN", { mtu: 127, header: 40 }],
    ["Zigbee", { mtu: 102, header: 10 }],
    ["IPv4", { mtu: 1500, header: 20 }],
    ["Ethernet", { mtu: 1500, header: 14 }]
  ]),
  {
    value: { mtu: 222, header: 13 },
    label: "Network Type:",
    format: ([name, _]) => name
  }
)

maxPayloadPerFragment = networkType.mtu - networkType.header
fragmentCount = Math.ceil(appPayload / maxPayloadPerFragment)
totalTransmitted = fragmentCount * networkType.mtu
fragmentOverhead = totalTransmitted - appPayload
fragmentOverheadPercent = (fragmentOverhead / totalTransmitted * 100).toFixed(1)

13.8 Knowledge Check

Test your understanding of packet structure fundamentals before diving into the detailed chapters.

Scenario: A LoRaWAN soil moisture sensor transmits data through a 3-hop network path:

  1. Sensor → LoRaWAN Gateway (LoRaWAN packet)
  2. Gateway → Cloud Broker (MQTT/TCP/IPv4 over cellular)
  3. Cloud → Time-Series Database (HTTP/REST API)

Question: For a 4-byte sensor reading, calculate the total bytes transmitted at each hop and cumulative overhead.

13.8.1 Step 1: Sensor → Gateway (LoRaWAN)

Packet structure:

Component Size Value/Purpose
MHDR 1 byte Message type/direction
DevAddr 4 bytes Device address
FCtrl 1 byte Frame control
FCnt 2 bytes Frame counter
FPort 1 byte Application port
Payload 4 bytes Sensor data: moisture reading
MIC 4 bytes Message integrity code

Total: 1 + 4 + 1 + 2 + 1 + 4 + 4 = 17 bytes on LoRaWAN link

Overhead: 13 bytes header/trailer ÷ 17 bytes total = 76% overhead

13.8.2 Step 2: Gateway → Cloud (MQTT/TCP/IPv4)

The gateway extracts the 4-byte payload and forwards it via MQTT:

Packet structure:

Component Size Value/Purpose
Ethernet header 14 bytes MAC addresses (on local network)
IPv4 header 20 bytes IP addresses, TTL, protocol
TCP header 20 bytes Ports, sequence, acknowledgment
MQTT fixed header 2 bytes Packet type, flags
MQTT variable header 8 bytes Topic name length + “sensors/1”
Payload 4 bytes Sensor data
Ethernet FCS 4 bytes Frame check sequence

Total: 14 + 20 + 20 + 2 + 8 + 4 + 4 = 72 bytes on cellular link

Overhead: 68 bytes ÷ 72 bytes = 94% overhead

13.8.3 Step 3: Cloud → Database (HTTP POST)

The cloud broker stores data in InfluxDB via HTTP API:

Packet structure (simplified HTTP POST):

POST /write?db=sensors HTTP/1.1
Host: influxdb.example.com
Content-Type: application/x-www-form-urlencoded
Content-Length: 45

moisture,sensor_id=1 value=425 1702834567

Total for this HTTP request:

Component Size Notes
HTTP headers ~120 bytes Method, Host, Content-Type, etc.
Payload 45 bytes Line protocol: “moisture,sensor_id=1 value=425 timestamp”
TCP/IP headers 40 bytes IPv4 + TCP

Total: 120 + 45 + 40 = 205 bytes

Overhead: 160 bytes headers ÷ 205 total = 78% overhead

13.8.4 Step 4: Cumulative Analysis

Bytes transmitted across all hops:

Hop Link Bytes Transmitted Actual Data Overhead Ratio
1 Sensor → Gateway 17 bytes 4 bytes 76%
2 Gateway → Cloud 72 bytes 4 bytes 94%
3 Cloud → Database 205 bytes 4 bytes (45 bytes formatted) 78% (relative to formatted)
Total 294 bytes 4 bytes 98.6% cumulative overhead!

13.8.5 Step 5: Real-World Cost Impact

For 1,000 sensors, each transmitting once per hour:

Hop Annual Data @ $0.10/MB Cost
LoRaWAN (license-free) 149 MB $0 (no cost)
Cellular (gateway → cloud) 631 MB $63
Database writes (cloud → DB) 1,797 MB $0 (internal LAN)

Key insights:

  1. Multi-hop multiplies overhead: Same 4 bytes of data transmitted 3 times with different wrapping
  2. Cellular link is the cost bottleneck: 94% overhead on the paid link
  3. HTTP is inefficient: Line protocol adds 10x overhead vs binary (45 bytes vs 4 bytes)

13.8.6 Step 6: Optimization Strategies

Optimization 1: Batch sensor readings

Instead of sending 1 reading/hour, send 24 readings once/day:

  • Before: 72 bytes × 24 msgs = 1,728 bytes/day
  • After: (68 bytes header + 96 bytes payload) = 164 bytes/day
  • Savings: 90% reduction!

Optimization 2: Use binary database protocol

Instead of HTTP + line protocol, use InfluxDB’s native binary protocol:

  • Before: 205 bytes per write
  • After: ~50 bytes (binary encoding)
  • Savings: 76% reduction

Optimization 3: Compress payloads

Use CBOR instead of MQTT JSON/text payload:

  • Before: MQTT topic “sensors/1” + JSON {"moisture":425} = 8 + 18 = 26 bytes
  • After: CBOR binary encoding = 8 bytes
  • Savings: 69% reduction

13.8.7 Final Optimized Architecture

With batching + binary protocols:

Hop Before After Savings
Sensor → Gateway 17 bytes/hour 17 bytes/day (batched) 96%
Gateway → Cloud 72 bytes/hour 80 bytes/day (batched + binary) 96%
Cloud → Database 205 bytes/hour 60 bytes/day (binary protocol) 97%
Annual cellular cost (1,000 sensors) $63/year $7/year 89% savings

Key takeaway: Understanding packet structure at every hop reveals optimization opportunities. Multi-hop overhead compounds, making batching and binary formats essential for constrained IoT.


Common Pitfalls

Relying on theoretical models without profiling actual behavior leads to designs that miss performance targets by 2-10×. Always measure the dominant bottleneck in your specific deployment environment — hardware variability, interference, and load patterns routinely differ from textbook assumptions.

Optimizing one parameter in isolation (latency, throughput, energy) without considering impact on others creates systems that excel on benchmarks but fail in production. Document the top three trade-offs before finalizing any design decision and verify with realistic workloads.

Most field failures come from edge cases that work in the lab: intermittent connectivity, partial node failure, clock drift, and buffer overflow under peak load. Explicitly design and test failure handling before deployment — retrofitting error recovery after deployment costs 5-10× more than building it in.

13.9 Summary

This chapter introduced the fundamental concepts of packet structure and framing that underlie all network communication:

Concept Key Points
Packet Anatomy Every packet has a header (routing/control), payload (data), and trailer (error detection)
Framing Length fields or delimiters mark packet boundaries; byte stuffing handles delimiter conflicts
Error Detection Checksums are fast but weak; CRC provides strong detection for critical applications
Protocol Overhead Headers can dominate small payloads; choose protocols based on overhead vs. functionality trade-offs

Key Design Decisions for IoT:

  1. Choose error detection based on consequence of corruption: CRC-32 for safety-critical, CRC-16 for general IoT, checksums only for non-critical legacy systems
  2. Minimize overhead for constrained devices: CoAP/UDP instead of HTTP/TCP, header compression (6LoWPAN), payload aggregation
  3. Match MTU to network constraints: Understand fragmentation costs on each link (LoRaWAN: 51-222 bytes, 6LoWPAN: 127 bytes, Ethernet: 1500 bytes)

13.10 What’s Next

Now that you understand packet structure, framing, and error detection, explore these related topics:

Chapter What You Will Learn
Data Formats for IoT How sensor data gets encoded inside the packet payload using JSON, CBOR, MessagePack, and Protocol Buffers
Sensor to Network Pipeline The full journey of a sensor reading from analog signal to transmitted network packet
Networking Fundamentals How packets flow through network layers, switching, and routing infrastructure
Transport Protocols TCP vs UDP trade-offs for IoT, congestion control, and connection management
MQTT MQTT packet types, QoS levels, and how the fixed/variable header structure works in practice
LoRaWAN Overview LoRaWAN MAC layer packet structure, device classes, and adaptive data rate