12 Packet Framing

Boundaries, Length Fields, Delimiters, Escaping, and Reassembly

fundamentals

packet

framing

12.1 In 60 Seconds

Framing is how a receiver finds message boundaries in a continuous byte stream. Use a fixed size, a length field, or a delimiter; escape any payload bytes that can imitate a delimiter; and define fragmentation when one message is too large for the link frame.

12.2 Start With the Story

Start with a message crossing a link where the receiver must know where the packet starts, what each field means, and whether any byte was corrupted. The core idea in Packet Framing is simple: packet and format design is about boundaries, field meaning, compactness, checks, and the evidence that the receiver can parse the payload safely. This page focuses that idea on Packet-structure how receivers find frame boundaries, choose length or delimiter framing, handle escaped bytes. In everyday IoT, JSON, CBOR, Protobuf, custom binary, framing bytes, CRCs, and overhead budgets are practical choices with reliability and debugging costs. Start simple: draw the packet as fields, mark the payload and checks, then justify the format only after the parse path is clear.

12.3 Finding Message Boundaries in a Stream

A communication link often delivers a continuous run of bytes with no built-in marks between messages. Framing is the agreement that lets a receiver split that stream back into the discrete packets the sender intended. Without framing, the receiver has data but no idea where one message stops and the next begins.

Picture a sentence written with no spaces or punctuation, such as thecatsatonthemat. The letters are all there, but you can only read it once you agree where words begin and end. Framing adds those spaces and full stops to a byte stream.

If you only need the intuition, this layer is enough: a receiver cannot act on a message until it knows where the message ends, framing is the agreement that supplies those boundaries, and the two everyday tools are a length field and a delimiter marker.

There are two common ways to mark boundaries. You can tell the receiver how long each message is with a length field, or you can place a special delimiter marker between messages. Both work, and each carries different trade-offs.

Boundary strategies: count bytes with a length field, scan for a delimiter, protect delimiter bytes with escaping, read fixed-size records, or fragment larger messages.

In practice, the boundary rule is visible in the trace. A UART sensor protocol may start with a sync byte and a length byte. A TCP application protocol may length-prefix each binary record because TCP is a stream, not a message service. A text log protocol may use newline delimiters. A BLE notification, LoRaWAN payload, or low-power serial link may choose a compact binary layout to avoid wasting bytes on repeated markers. The receiver can only parse safely when that rule is explicit.

The One-Minute Framing Decision

Pick the boundary rule

Choose a length field, a delimiter marker, or fixed-size frames, then write the rule down so both ends agree.

Stop data imitating the marker

If a delimiter value can appear inside the payload, the design must escape it so it is not mistaken for a boundary.

Plan for big messages

Anything larger than the link’s maximum frame must be fragmented and reassembled in a defined way.

Beginner Examples

A sensor that always sends the same fixed-size reading can use fixed-length frames, so the boundaries are implied by the constant length.
A text-style protocol often ends each message with a newline, using that newline as a delimiter.
A binary protocol often places a length field in the header so the receiver knows exactly how many payload bytes follow.

Boundary Knowledge Check

If this gives you the core idea, you can stop here. Continue to Practitioner when you need to choose and implement a framing scheme.

12.4 Apply It: Choose and Implement a Framing Scheme

The practical job is to choose a boundary method, handle the cases where data could imitate a boundary, and define how large messages are split and rebuilt. A clear, written framing rule lets two independent implementations interoperate.

Walkthrough: From Stream to Reliable Boundaries

Pick the boundary method. Fixed-length for uniform records, length-prefix for variable binary messages, or a delimiter for text and streams where resynchronization matters.
If length-prefix, define the field. Set its size, units, byte order, and whether it counts the header. Bound the maximum so a corrupt or hostile length cannot request an unreasonable read or buffer.
If delimiter, define escaping. Choose the marker, then specify how payload bytes equal to the marker, and to the escape byte itself, are escaped on send and restored on receive.
Add synchronization if the link needs it. A preamble and start marker let the receiver lock onto the bit timing before the frame body begins.
Define the MTU and a fragmentation rule. State how a large message is split, numbered, and rebuilt, including how missing, duplicate, and out-of-order fragments are handled and when reassembly times out.
Keep framing separate from error detection. Framing locates the message; a checksum or CRC then judges whether the located bytes are intact.

Worked Example: Byte Stuffing a Delimiter

Suppose a protocol uses a flag value to mark frame boundaries and an escape value to signal substitution. The transmitter scans the payload before sending:

Wherever a payload byte equals the flag, replace it with the escape value followed by a substituted byte.
Wherever a payload byte equals the escape value, replace it with the escape value followed by a different substituted byte.
The receiver reverses the rule: when it sees the escape value, it consumes the next byte and restores the original.

The result is that the flag value only ever appears as a real boundary on the wire, never inside payload data.

Comparison of framing techniques

Scheme

How a Boundary Is Found

Strength

Cost

Fixed-length

Every frame is the same known size.

Trivial to parse with no extra fields.

Wastes space for short messages and cannot carry variable data.

Length-prefix

Read the length, then read exactly that many bytes.

Compact and exact for variable-size messages.

A corrupted length desynchronizes the stream until recovery logic acts.

Delimiter with escaping

Scan for the marker, then unescape the data.

Self-resynchronizing after corruption.

Escaping adds bytes and processing to every frame.

Incremental Practice

Beginner

For uniform fixed-size sensor records, argue in one sentence why fixed-length framing is the simplest choice.

Intermediate

For a length-prefixed binary protocol, write the rule that prevents a corrupt length from allocating an unreasonable buffer.

Advanced

Design a fragmentation header that lets a receiver detect both a missing middle fragment and a duplicate fragment.

Delimiter Escaping Knowledge Check

If your job is to define and implement a framing scheme, you can stop here. Continue to Under the Hood for synchronization, stuffing, and reassembly mechanics.

12.5 Under the Hood: Synchronization, Stuffing, and Reassembly

The deeper layer explains how a receiver first locks onto the bits, how stuffing keeps a marker unique, and how a corrupted boundary recovers.

Bit Synchronization Comes First

Before bytes can even be read, the receiver must lock onto the bit timing. A preamble, a known alternating pattern, lets the receiver’s clock synchronize, and a start-frame delimiter then marks the first real bit of the frame. Without this step, even uncorrupted bytes can be sampled at the wrong boundaries and decoded as nonsense.

Stuffing Keeps a Marker Unique

When a frame marker is a specific bit pattern, the transmitter can use bit stuffing: it inserts an extra 0 bit after a run of consecutive 1 bits in the data, for example after five ones, so the data can never reproduce the marker pattern. The receiver removes the stuffed bit. Byte stuffing is the byte-oriented version of the same idea, escaping any payload byte that matches the flag or escape value. Both techniques guarantee the boundary marker is something only the framing layer produces.

Losing and Regaining Frame Synchronization

Length-prefix framing is efficient but fragile to a corrupted length. If the length field flips, the receiver reads the wrong number of bytes, and every following boundary is wrong until a recovery mechanism, such as a timeout, an idle gap, or a resynchronization marker, restores alignment.
Delimiter framing degrades more gracefully. After corruption, the receiver can discard bytes until it finds the next delimiter and resume at the next frame. The cost is the escaping overhead on every frame.

MTU, Fragmentation, and Reassembly

Reassembly needs state: fragments share a datagram tag and size, later fragments add an offset, and the receiver buffers and orders them, delivering only if all arrive before the 60-second timeout.

Concern

What It Means

What the Frame Must Carry

Failure If Ignored

MTU limit

The largest frame the link can carry.

A way to know a message exceeds the limit and must be split.

Oversize messages are dropped or silently truncated.

Ordering

Fragments may arrive out of order.

An identifier plus an offset or sequence number.

Fragments are reassembled in the wrong order.

Completion

Knowing when the set is whole.

A last-fragment flag or total count.

Reassembly never completes or completes early.

Resource bound

Partial messages occupy buffers.

A reassembly timeout and a buffer limit.

Missing fragments pin buffers open indefinitely.

Framing Versus Error Detection

Framing and error detection are different layers that are easy to conflate. Framing answers “where is the message?” Error detection answers “are the located bytes intact?” A robust receiver does both: it finds the boundary, then runs the checksum or CRC over the framed bytes. A correct boundary with corrupt contents, or correct contents read at the wrong boundary, are distinct failures.

Common Pitfalls

Trusting a length field without bounding it. A corrupt or hostile length can request a huge read or buffer allocation.
Escaping only the delimiter. The escape byte itself must also be escaped, or its appearance in data becomes ambiguous.
Confusing framing with error detection. Finding a boundary does not prove the bytes inside it are correct.
Omitting a reassembly timeout. A single missing fragment can hold buffers open until they are exhausted.
Assuming byte alignment without bit synchronization. Without a preamble and start marker, even correct bytes can be mis-framed.

Reassembly Knowledge Check

At this depth, framing is a chain of agreements: synchronize the bits, mark the boundary in a way data cannot imitate, recover when a boundary is lost, and split and rebuild messages too large for one frame. Documenting each step is what lets two independent implementations exchange packets reliably.

12.6 Summary

Framing is the agreement that lets a receiver split a continuous byte stream into discrete messages.
The common boundary methods are fixed length, a length-prefix field, and a delimiter marker.
Delimiter framing needs escaping, through byte or bit stuffing, so payload data cannot imitate the boundary marker.
A length field must be bounded so a corrupt value cannot desynchronize the stream or request an unreasonable buffer.
Messages larger than the link’s MTU must be fragmented and reassembled with identifiers, ordering, completion markers, and a timeout.
Framing locates a message, while error detection with a checksum or CRC is a separate layer that judges whether the bytes are intact.

Key Takeaway

Decide explicitly how a receiver finds each message boundary, whether by length, delimiter, or fixed size, make payload data unable to imitate that boundary, and treat synchronization, fragmentation, and error detection as separate, documented concerns.