By the end of this chapter, you will be able to:
When data flows over a network, the receiver needs to know where one message ends and the next begins – like how spaces separate words in a sentence. Packet framing solves this by adding markers or length counters that tell the receiver exactly how to slice up the stream of raw data into individual messages. Without framing, all the data would run together into meaningless noise.
Prerequisites (Read These First):
Companion Chapters (Packet Structure Series):
Networking Foundations:
How does a receiver know where one packet ends and another begins?
Networks use several techniques:
The header specifies payload size:
Header: [... Length=100 ...] | Payload: [100 bytes] | Trailer: [CRC]Unique byte sequences mark boundaries:
[0x7E] Header | Payload | Trailer [0x7E]What if payload contains the delimiter byte? - Solution: Escape it with a special character - Example: If delimiter is 0x7E, replace payload 0x7E with 0x7D 0x5E - Receiver reverses the escaping - Used by: HDLC, PPP, SLIP
Operates at the bit level rather than byte level: - After five consecutive 1 bits in the data, the sender inserts a 0 bit - The receiver removes any 0 bit that follows five consecutive 1 bits - Prevents the data from mimicking the flag pattern 01111110 (0x7E) - Used by: HDLC (at the bit level), CAN bus
A modern encoding that guarantees bounded overhead: - Eliminates all zero bytes from the payload by encoding run lengths - Overhead is exactly 1 byte per 254 payload bytes (worst case ~0.4%) - Produces predictable, fixed-overhead frames – unlike byte stuffing, which is data-dependent - Used by: Embedded serial protocols, USB CDC, sensor networks over UART
Enhanced Byte Stuffing Example:
Suppose you want to send payload [0x10, 0x7E, 0x20] where 0x7E is the frame delimiter:
Original: [0x7E] [0x10, 0x7E, 0x20] [0x7E]
^^^^ ^^^^^^^^^^^^^^^^^ ^^^^
Start Payload End
Problem: Payload contains 0x7E!
Escaped: [0x7E] [0x10, 0x7D, 0x5E, 0x20] [0x7E]
^^^^^^^^^
Escape sequence
Receiver: Sees 0x7D -> reads next byte -> converts 0x5E back to 0x7E0x7E | 0x10 0x7E 0x20 | 0x7E0x7E0x7E | 0x10 0x7D 0x5E 0x20 | 0x7E0x7D, it decodes the next byte with XOR 0x20 and restores the original delimiter bytePut the byte stuffing steps in the correct order:
Maximum Transmission Unit (MTU) is the largest packet size a network can handle:
If your payload exceeds MTU, the network layer fragments it into multiple packets, each with its own header/trailer overhead.
Let’s quantify fragmentation overhead’s impact on bandwidth efficiency for a constrained IoT network.
Scenario: Temperature sensors in a cold-chain logistics container transmit 1,000 bytes of data (50 temperature readings at 20 bytes each) via 6LoWPAN over IEEE 802.15.4 (Zigbee physical layer).
Network parameters:
Calculate fragmentation:
Fragment 1 (FRAG1):
Fragments 2-8 (FRAGN):
Total fragments needed: \[N_{\text{fragments}} = 1 + \left\lceil \frac{1000 - 123}{122} \right\rceil = 1 + \left\lceil 7.19 \right\rceil = 8 \text{ fragments}\]
Overhead calculation: \[\text{Header overhead} = 4 + (7 \times 5) = 39 \text{ bytes}\]
Bandwidth efficiency: \[\text{Efficiency} = \frac{1000}{1000 + 39} \times 100\% = 96.2\%\]
Alternative: batching into smaller packets
Instead of one 1,000-byte payload, send ten 100-byte packets (10 readings each): - Each packet: 13-byte header + 100-byte payload + 4-byte MIC = 117 bytes - Total transmitted: \(10 \times 117 = 1{,}170\) bytes - Efficiency: \(\frac{1000}{1170} \times 100\% = 85.5\%\)
Key insight: For this scenario, fragmentation is MORE efficient (96.2%) than batching into smaller unfragmented packets (85.5%). The crossover point depends on MTU size and header overhead. Rule of thumb: if payload is 2-3× MTU, fragmentation is acceptable; if 10× MTU, consider compression or batching.
FRAG1 header leaves 123 bytes for payload.FRAGN header leaves 122 bytes per later fragment.1000 / 1039 = 96.2%, so most transmitted bytes are still useful payload.Let’s trace a complete byte stuffing scenario for an HDLC-like protocol transmitting a sensor reading:
Scenario: Send temperature reading {"temp": 30.5} using HDLC framing where 0x7E is the frame delimiter.
Step 1: Original payload
JSON: {"temp": 30.5}
ASCII hex: 7B 22 74 65 6D 70 22 3A 20 33 30 2E 35 7DStep 2: Identify conflicts
0x7E0x7E0x7D (closing brace in JSON)0x7D is the escape character itself, so it needs escaping!0x7E (delimiter) and 0x7D (escape) need escapingStep 3: Apply byte stuffing rules
0x7E (delimiter) → 0x7D 0x5E0x7D (escape char) → 0x7D 0x5DStep 4: Stuffed payload
Original: 7B 22 74 65 6D 70 22 3A 20 33 30 2E 35 7D
^^
Stuffed: 7B 22 74 65 6D 70 22 3A 20 33 30 2E 35 7D 5D
^^^^^
} escapedStep 5: Full frame
[0x7E] [7B 22 ... 7D 5D] [CRC16] [0x7E]
^^^^^ ^^^^^^^^^^^^^^^^ ^^^^^^^ ^^^^^
Start Stuffed Payload Trailer EndStep 6: Receiver unstuffs
0x7E → Start of frame0x7B (opening brace) → Keeps as-is (not special)0x7D 0x5D → Converts to 0x7D (closing brace)0x7E → End of frameOverhead cost: Original 14 bytes became 15 bytes after stuffing (7% overhead). In worst case (payload full of 0x7E bytes), overhead could reach 100%.
Packet Structure Series:
Protocol Implementations:
Network Layer:
IoT-Specific MTU:
Explore how large packets get fragmented when they exceed the Maximum Transmission Unit (MTU). Adjust the payload and MTU sizes to see fragmentation in action.
Tips for optimal fragmentation:
0x7E show frame boundaries directly. Best when visible boundaries and easy resynchronization matter more than payload flexibility.Match each framing technique to the protocol that uses it:
The Myth: “To save bandwidth, I should make my packets as small as possible by minimizing headers and sending frequent tiny updates.”
Why This Is Wrong:
While minimizing packet size seems logical, header overhead doesn’t scale linearly. Consider this example:
Scenario 1: Many Small Packets
Scenario 2: One Batched Packet
Additional Hidden Costs of Small Packets:
The Right Approach:
Real-World Impact: A smart building with 500 temperature sensors sending every 10 minutes: - Small packets (2 bytes each): 1.9 GB/year, battery life 1 year - Batched packets (20 readings): 0.23 GB/year, battery life 5+ years - Bandwidth savings: 88%, Cost savings: $167/year at $0.10/MB
The Takeaway: Headers are overhead you pay per packet, not per byte. Batch your data intelligently to minimize both bandwidth costs and energy consumption.
Concept: Understanding byte stuffing for delimiter handling.
The Problem:
Developers calculate that their 100-byte payload fits comfortably in a 127-byte MTU (IEEE 802.15.4), forgetting that byte stuffing can expand payloads unpredictably.
Real Example:
Original payload: 100 bytes
Protocol: HDLC with 0x7E delimiter and 0x7D escape
MTU: 127 bytes (Zigbee)
Worst case: Payload contains all 0x7E bytes
→ 100 bytes × 2 (each becomes 0x7D 0x5E) = 200 bytes!
→ DOESN'T FIT! Packet fragmentation required.Why This Happens:
Byte stuffing adds overhead when delimiter bytes (0x7E) or escape bytes (0x7D) appear in the payload. The overhead is data-dependent – not fixed!
Real-World Impact:
A smart city project used SLIP (Serial Line IP) for sensor backhaul over RS-485. They tested with ASCII text (low escape rate) but deployed with binary sensor data (high escape rate). Result: - Expected: 90-byte payloads fit in 127-byte frames - Actual: 90-byte payloads expanded to 91-93 bytes after escaping - Problem: 3% of packets fragmented unexpectedly, causing 200ms latency spikes - Cost: $15,000 to debug and switch to length-field framing
The Fix:
Option 1: Budget for worst-case overhead (conservative)
// For HDLC-style byte stuffing with 0x7E delimiter
#define MAX_PAYLOAD 100
#define MTU 127
#define WORST_CASE_OVERHEAD (MAX_PAYLOAD * 2) // Every byte could double
// Check if we fit
if (payload_size + WORST_CASE_OVERHEAD > MTU) {
// Use fragmentation or compression
}Option 2: Pre-scan payload and calculate actual overhead (optimal)
size_t calculate_stuffing_overhead(const uint8_t* data, size_t len) {
size_t escapes = 0;
for (size_t i = 0; i < len; i++) {
if (data[i] == 0x7E || data[i] == 0x7D) {
escapes++;
}
}
return escapes; // Each escape adds 1 byte
}
size_t actual_size = payload_size + calculate_stuffing_overhead(payload, payload_size);
if (actual_size + HEADER_SIZE > MTU) {
// Fragment
}Option 3: Avoid byte stuffing entirely (best)
Use length-field framing instead:
// Frame structure:
// [Start: 0x7E] [Length: 2 bytes] [Payload: N bytes] [CRC: 2 bytes]
//
// Length field tells receiver exact size - no escaping needed!
// Delimiter only at start, not end - payload can contain anythingComparison:
Decision Rule:
Delimiter-based protocols fail fast when teams test only with clean ASCII examples and forget that real binary payloads can contain the same byte values as frame markers. If you use delimiters, validate the escaping path with representative payloads before shipping.
Byte stuffing overhead is data-dependent. Designing buffers and MTU limits around average expansion works until a delimiter-heavy payload arrives and suddenly forces fragmentation or truncation. Size the frame for the worst legal payload, not the nicest one.
Fragmentation is not just a size calculation. The receiver also needs tags, offsets, ordering rules, and timeouts for missing fragments. If one fragment disappears, the whole datagram is usually useless, so reassembly behavior must be treated as part of the framing design.
Frame delimiters and boundaries enable reliable packet detection:
0 bits after five consecutive 1 bits to prevent flag pattern mimicryKey Takeaways:
Continue exploring packet structure and related networking concepts:
Continue to Error Detection ->
Sammy the Sensor is sending messages over a busy radio channel. But there’s a problem – “How does the receiver know where my message starts and ends?”
Max the Microcontroller has three ideas:
“Idea 1: Write the length first!” Max says. “Start with ‘This message is 10 bytes long.’ Then the receiver counts exactly 10 bytes. Done! That’s what MQTT and TCP do.”
Lila the LED suggests another way: “Idea 2: Use special bookends! Put a special marker at the start and end, like putting quotation marks around a sentence: [START] Hello World [END].”
“But what if my message contains the special marker?” asks Sammy.
“Idea 3: Byte stuffing!” says Lila. “It’s like using an escape character. If your message has the special word, you add a code before it so the receiver knows ‘this is data, not a marker.’ Like using backslash in programming!”
Bella the Battery has a warning: “And watch out for messages that are TOO BIG! If your message is bigger than the maximum size the network allows (called MTU), it gets split into pieces. Each piece needs its own wrapper, which wastes space. So it’s better to send ONE medium message than TEN tiny ones – less wrapping paper!”
The Squad’s Rule: Every message needs a frame – like an envelope for a letter. The receiver needs to know where it starts, where it ends, and how big it is!
Scenario: You need to send 500 bytes of sensor data over a network with a 127-byte MTU (like IEEE 802.15.4/Zigbee). Each fragment adds a 5-byte fragmentation header.