By the end of this chapter, you will be able to:
Prerequisites (Read These First): - Packet Anatomy - Headers, payloads, and trailers fundamentals
Companion Chapters (Packet Structure Series): - Packet Structure Overview - Index of all packet structure topics - Error Detection - Checksums, CRC, and data integrity - Protocol Overhead - Header comparison and encapsulation
Networking Foundations: - Networking Fundamentals - Network layers and packet flow - Layered Models - OSI and TCP/IP encapsulation - 6LoWPAN - IPv6 header compression for constrained networks
How does a receiver know where one packet ends and another begins?
Networks use several techniques:
The header specifies payload size:
Header: [... Length=100 ...] | Payload: [100 bytes] | Trailer: [CRC]Unique byte sequences mark boundaries:
[0x7E] Header | Payload | Trailer [0x7E]What if payload contains the delimiter byte? - Solution: Escape it with a special character - Example: If delimiter is 0x7E, replace payload 0x7E with 0x7D 0x5E - Receiver reverses the escaping
Enhanced Byte Stuffing Example:
Suppose you want to send payload [0x10, 0x7E, 0x20] where 0x7E is the frame delimiter:
Original: [0x7E] [0x10, 0x7E, 0x20] [0x7E]
^^^^ ^^^^^^^^^^^^^^^^^ ^^^^
Start Payload End
Problem: Payload contains 0x7E!
Escaped: [0x7E] [0x10, 0x7D, 0x5E, 0x20] [0x7E]
^^^^^^^^^
Escape sequence
Receiver: Sees 0x7D -> reads next byte -> converts 0x5E back to 0x7EMaximum Transmission Unit (MTU) is the largest packet size a network can handle:
| Network Type | MTU Size | Notes |
|---|---|---|
| Ethernet | 1500 bytes | Standard MTU for most wired networks |
| Wi-Fi | 2304 bytes | Theoretical max, usually 1500 in practice |
| LoRaWAN | 51-242 bytes | Varies by data rate (DR0-DR5) |
| BLE | 27 bytes | Without fragmentation |
| Zigbee | 127 bytes | IEEE 802.15.4 frame size |
If your payload exceeds MTU, the network layer fragments it into multiple packets, each with its own header/trailer overhead.
Explore how large packets get fragmented when they exceed the Maximum Transmission Unit (MTU). Adjust the payload and MTU sizes to see fragmentation in action.
Tips for optimal fragmentation: - Choose MTU close to your typical payload size to minimize fragments - 6LoWPAN uses 8-byte offset units, so align payloads accordingly - Consider payload batching to amortize fragment header overhead - Monitor reassembly timeout in constrained networks (fragments may arrive out of order)
%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#16A085', 'secondaryColor': '#E67E22', 'tertiaryColor': '#7F8C8D'}}}%%
flowchart TD
START([Which framing<br/>technique?]) --> Q1{Known payload<br/>length?}
Q1 -->|"Yes, always"| LEN["Length Field<br/>โโโโโโโ<br/>Header tells receiver<br/>exactly how many<br/>bytes to read"]
Q1 -->|"Variable/Unknown"| Q2{Can control<br/>payload content?}
Q2 -->|"Yes"| DELIM["Delimiters<br/>โโโโโโโ<br/>Use unique marker<br/>bytes 0x7E at<br/>start and end"]
Q2 -->|"No - arbitrary data"| STUFF["Byte Stuffing<br/>โโโโโโโ<br/>Escape delimiter<br/>bytes that appear<br/>in payload"]
LEN --> PROS1["Pros: Fast, no scanning<br/>Cons: Must know length first"]
DELIM --> PROS2["Pros: Simple detection<br/>Cons: Delimiter conflicts"]
STUFF --> PROS3["Pros: Handles any data<br/>Cons: Variable overhead"]
PROS1 --> EX1["Use: IP, TCP, UDP<br/>MQTT, CoAP"]
PROS2 --> EX2["Use: Ethernet preamble<br/>HDLC, Serial"]
PROS3 --> EX3["Use: PPP, SLIP<br/>Binary protocols"]
style START fill:#2C3E50,stroke:#16A085,stroke-width:3px,color:#fff
style Q1 fill:#16A085,stroke:#2C3E50,stroke-width:2px,color:#fff
style Q2 fill:#16A085,stroke:#2C3E50,stroke-width:2px,color:#fff
style LEN fill:#2C3E50,stroke:#16A085,stroke-width:2px,color:#fff
style DELIM fill:#2C3E50,stroke:#16A085,stroke-width:2px,color:#fff
style STUFF fill:#2C3E50,stroke:#16A085,stroke-width:2px,color:#fff
style PROS1 fill:#E67E22,stroke:#2C3E50,stroke-width:1px,color:#fff
style PROS2 fill:#E67E22,stroke:#2C3E50,stroke-width:1px,color:#fff
style PROS3 fill:#E67E22,stroke:#2C3E50,stroke-width:1px,color:#fff
style EX1 fill:#ecf0f1,stroke:#2C3E50,stroke-width:1px,color:#000
style EX2 fill:#ecf0f1,stroke:#2C3E50,stroke-width:1px,color:#000
style EX3 fill:#ecf0f1,stroke:#2C3E50,stroke-width:1px,color:#000
The Myth: โTo save bandwidth, I should make my packets as small as possible by minimizing headers and sending frequent tiny updates.โ
Why This Is Wrong:
While minimizing packet size seems logical, header overhead doesnโt scale linearly. Consider this example:
Scenario 1: Many Small Packets - Send 100 temperature readings (2 bytes each) as 100 separate packets - Each packet needs 13-byte LoRaWAN header + 4-byte MIC = 17 bytes overhead - Total: (2 + 17) x 100 = 1,900 bytes - Overhead: 1,700 bytes (89.5%)
Scenario 2: One Batched Packet - Send all 100 readings in one packet - Packet: 13-byte header + 200-byte payload + 4-byte MIC = 217 bytes - Overhead: 17 bytes (7.8%) - Savings: 88% less bandwidth!
Additional Hidden Costs of Small Packets: 1. Radio wake-up energy: Each transmission requires powering up the radio (10-100 mW for 1-5 seconds) 2. Protocol overhead: Channel access, preamble, and synchronization for each packet 3. Gateway load: More packets = more processing and database writes 4. Collision risk: More transmissions = higher chance of interfering with other devices
The Right Approach: - Batch data when real-time updates arenโt critical - Balance packet size with latency requirements - Consider MTU limits: Donโt exceed Maximum Transmission Unit (LoRaWAN DR0 = 51 bytes) - Use compression: CBOR or Protocol Buffers can reduce payload size while batching multiple readings
Real-World Impact: A smart building with 500 temperature sensors sending every 10 minutes: - Small packets (2 bytes each): 1.9 GB/year, battery life 1 year - Batched packets (20 readings): 0.23 GB/year, battery life 5+ years - Bandwidth savings: 88%, Cost savings: $167/year at $0.10/MB
The Takeaway: Headers are overhead you pay per packet, not per byte. Batch your data intelligently to minimize both bandwidth costs and energy consumption.
Concept: Understanding byte stuffing for delimiter handling.
A stream of bytes arrives: 0x7E 0x45 0x7D 0x5E 0x7E. Using HDLC-style byte stuffing where 0x7E is the delimiter and 0x7D is the escape character, what is the actual payload?
A is correct. First 0x7E = start delimiter. 0x45 = data. 0x7D 0x5E = escaped 0x7E (0x7E XOR 0x20 = 0x5E). Last 0x7E = end delimiter. Byte stuffing allows delimiter bytes to appear in data without breaking frame boundaries.
Frame delimiters and boundaries enable reliable packet detection:
Key Takeaways: - Choose framing method based on whether payload length is known in advance - Byte stuffing adds variable overhead but handles arbitrary binary data - Fragmentation multiplies header overhead - batch data when possible - 6LoWPAN fragmentation is designed for constrained IEEE 802.15.4 networks
Continue exploring packet structure with these related chapters: