IoT Fundamentals · Study deck
Binary Data Formats for IoT
Picture a compact sensor record whose receiver swaps the byte order and reports the wrong temperature.
Physics Phoebe is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: CBOR Encoding Every CBOR item begins with one byte whose top three bits select a major type (unsigned integer, negative integer, byte string, text string, array, map, tag, or float and simple values).
- Explain: The important idea is not "always make messages smaller." The important idea is a trade: binary formats save bytes only when both the sender and the receiver share the rules for decoding them.
- Explain: Binary is compact but needs a contract Typed bytes drop most of the textual overhead, but the receiver must know the types, field meaning, and byte order.
Major section
In 60 Seconds
JavaScript Object Notation, or JSON, is a text format built from named values.
- A protocol means shared rules for exchanging messages.
- A binary format must define equally clear field boundaries and meaning.
- Encode one known record, change its version, truncate it, corrupt one byte, and decode it with another implementation.
Major section
In 60 Seconds (continued)
Saving bytes was not useful because the meaning was no longer shared.
- This runway does not prove that one format fits every link.
- The deeper sections compare compact formats, schemas, framing, byte order, checks, versioning, tooling, and ownership costs.
- Binary formats help when bytes, power, or storage are constrained, but the format is a contract.
Major section
What Compact Binary Buys
A binary data format keeps the same facts but stores them as typed bytes instead of text, so the message gets much smaller.
- The important idea is not "always make messages smaller." The important idea is a trade: binary formats save bytes only when both the sender and the receiver share the rules for decoding them.
Major section
What Compact Binary Buys (continued)
They pay off when bandwidth, power, or storage are tight and both ends agree on a schema or decoding contract.
- When the link is comfortable and humans need to inspect data, readable text is often the better default.
- Full sentences are clear to anyone but slow and bulky.
- Smaller is not automatically better.
Major section
What Compact Binary Buys (continued)
Shorthand is far more compact, but only readers who know your shorthand can recover the meaning.
- For example, a battery soil-moisture node on LoRaWAN may send temperature, relative humidity, battery voltage, and a timestamp every half hour.
- CBOR can keep the message self-describing while shrinking the encoding, while a custom layout might store temperature as signed centidegrees and battery voltage as millivolts.
- Exact byte counts change with field names, numeric ranges, and identifiers.
Major section
What Compact Binary Buys (continued)
That last step is only sensible if the gateway, decoder tests, and version notes are owned by the same deployment.
- The stable pattern is what matters: JSON repeats readable structure, CBOR keeps typed structure with less text overhead, and raw binary removes nearly everything except the values and the decoder contract.
- The panels keep the measurement fixed, so differences come from the representation rather than extra sensor data.
- The One-Minute View Text is readable but heavy JSON carries field names, quotes, and digits as text on every message.
Major section
What Compact Binary Buys (continued)
Binary is compact but needs a contract Typed bytes drop most of the textual overhead, but the receiver must know the types, field meaning, and byte order.
- Beginner Examples A device on Wi-Fi or Ethernet sending to a dashboard can keep JSON, because bandwidth and battery are not the binding constraint.
- A LoRaWAN or Sigfox node with a tiny per-message budget benefits from a compact format, because every byte is airtime and energy.
- An opaque eight-byte payload that no current tool can decode is a liability, not an optimization.
Major section
Apply It: Choose and Apply a Binary Format
CBOR and MessagePack: Self-Describing Binary CBOR (Concise Binary Object Representation, IETF RFC 8949) is "binary JSON": the same data model of maps, arrays, numbers, strings, and booleans, encoded as typed bytes.
- CBOR is the native payload format for CoAP and a strong default for LoRaWAN and NB-IoT.
Major section
Apply It: Choose and Apply a Binary Format (continued)
Custom Binary: Hand-Packed Bytes Custom binary defines a fixed byte layout that your code packs and unpacks directly.
- Protocol Buffers: Schema-Defined Binary Protocol Buffers (Protobuf) is defined by a.proto schema.
- Each field is identified by a small field number instead of a name, so field names never travel on the wire.
- Each branch asks what the receiver needs, making parser ownership part of the format decision.
Major section
Apply It: Choose and Apply a Binary Format (continued)
That makes it very compact and very fast to parse with generated code, but it is not self-describing: both ends need the schema to decode a message.
- It produces the smallest messages and the fastest parsing, but there is no tooling, no self-description, and no automatic schema evolution.
- Custom binary appears last because owning its byte layout also means owning documentation, tests, and migration.
- The sizes below are approximate and specific to this payload, but the ranking holds in general.
Major section
Under the Hood: How the Bytes Are Built
Integers are compact: values 0 to 23 fit in the initial byte, then one extra byte covers up to 255, two cover up to 65535, and four or eight cover larger values.
- Each format reaches its size by removing a different layer of overhead.
- This map is 55 bytes.
- Best Fit and Main Risk.
Major section
Under the Hood: How the Bytes Are Built (continued)
Floats can be stored as 16, 32, or 64 bits, so a low-precision reading can use a two-byte half-float.
- Understanding the encoding lets you estimate payload sizes, debug wire-format problems, and evolve a schema without breaking deployed devices.
- The field-name strings are now the largest part, which is exactly the overhead Protobuf removes next.
- Protocol Buffers Encoding A Protobuf message is a series of fields.
Major section
Under the Hood: How the Bytes Are Built (continued)
CBOR Encoding Every CBOR item begins with one byte whose top three bits select a major type (unsigned integer, negative integer, byte string, text string, array, map, tag, or float and simple values).
- Each field starts with a tag byte computed as (field_number << 3) | wire_type, so the tag encodes both which field and how to read its value.
- Small numbers take one byte; a 64-bit value can take up to ten.
- A leading version or sync byte lets a decoder detect the layout and frame the message.
- High-volume and multi-team contracts; cannot decode without the schema.
Major section
Under the Hood: How the Bytes Are Built (continued)
Reusing Protobuf field numbers.: A recycled number lets a new field decode stale data as if it were valid.
- CoAP and constrained links needing JSON-like flexibility; still carries field-name bytes.
- Common Pitfalls Assuming a library fits the device.: A tree-building CBOR parser can need far more RAM than a streaming one.
- The right format records each of these instead of treating "smaller" as the only goal.
Major section
Summary
Custom binary is the smallest and the most fragile; reserve it for ultra-constrained links you fully control.
- Binary formats shrink messages by replacing readable text with typed bytes, but only pay off when both ends share a decoding contract.
- CBOR (RFC 8949) and MessagePack are self-describing binary with a JSON-like model; CBOR is CoAP's native payload format.
- Judge a format by size, parsing cost, memory, schema coordination, and how it will evolve, not by byte count alone.
Deck summary
Key takeaways
JavaScript Object Notation, or JSON, is a text format built from named values.
- Saving bytes was not useful because the meaning was no longer shared.
- A binary data format keeps the same facts but stores them as typed bytes instead of text, so the message gets much smaller.
- They pay off when bandwidth, power, or storage are tight and both ends agree on a schema or decoding contract.
- Shorthand is far more compact, but only readers who know your shorthand can recover the meaning.
Retrieval practice
Recall check 1 of 3
Physics Phoebe says: answer from memory, then check your reasoning.
Q1When is a compact binary payload format usually the right direction?
Show answer
Answer: C Binary formats save bytes only when the decoding rules are explicit and shared by both ends.
Retrieval practice
Recall check 2 of 3
Physics Phoebe says: answer from memory, then check your reasoning.
Q2A LoRaWAN sensor must send temperature, humidity, and a timestamp every 15 minutes, and the payload limit at its data rate is about 51 bytes. The team wants minimal deployment coordination. Which format fits best?
Show answer
Answer: B CBOR comfortably fits this payload and needs no shared schema, which keeps a simple deployment simple.
Retrieval practice
Recall check 3 of 3
Physics Phoebe says: answer from memory, then check your reasoning.
Q3A team must add a new optional field to a deployed Protobuf message without breaking devices still running the old firmware. What is the safe change?
Show answer
Answer: C Unknown field numbers are skipped by older decoders, so new and old code interoperate.
Print reference
Answers
Answer key.
- C · Binary formats save bytes only when the decoding rules are explicit and shared by both ends.
- B · CBOR comfortably fits this payload and needs no shared schema, which keeps a simple deployment simple.
- C · Unknown field numbers are skipped by older decoders, so new and old code interoperate.