14 Choosing an IoT Data Format

14.1 In 60 Seconds

Decode One Reading After Every Handoff

Picture a temperature device sending 215 to a dashboard. The number could mean 215 degrees, 21.5 degrees, a raw count, or an error code. The format is useful only when every receiver recovers the same field, unit, scale, time, and version.

Bandwidth means how much data a link can carry in a given time. A payload means the useful reading or command inside a message. A protocol means the shared rules for a message exchange.

Encode one known reading in each candidate format, move it through the real path, and decode it at the end. Change a field, omit a unit, use an older version, repeat the message, and restart the receiver. Compare bytes, energy, readability, and rejection behavior.

This runway does not prove that the smallest format is the best. The deeper sections compare text, compact standard encodings, shared schemas, custom binary, change cost, link limits, and decoder evidence.

Choose the simplest inspectable data format that still fits the measured link and change model. JSON is a good default when bandwidth and power are comfortable, CBOR or MessagePack help when compact standard encoding is useful, Protocol Buffers fit shared schema contracts, and custom binary belongs only where measured byte limits justify its decoder discipline.

The mathematical gist. The chapter’s rounded vineyard ledger uses $P_{rx}=\mathrm{EIRP}-PL-L_{extra}$ and $M=P_{rx}-S$. With 14 dBm EIRP and 125 dB path loss, received power is $-111$ dBm. Adding the illustrative 15 dB canopy/terrain loss gives $-126$ dBm, leaving 10.9 dB against the catalog SF12 sensitivity but $-3.1$ dB against SF7. That motivates measuring or ADR before a 51-byte EU863-870 DR0 payload case is treated as the design profile.

Math Bridge · guided foundationsWhy does the vineyard link push us toward a constrained payload?Let Phoebe derive the 868 MHz path and show exactly where SF12 keeps margin while SF7 crosses zero. Learn the maths

14.2 Start With the Story

You will choose a message format that fits your radio budget and remains readable by the receiver after updates. Start by measuring a representative payload and recording how both ends will decode it.

Follow the format review across four beats to see how the device, link, parser, and future schema shape one defensible choice.

14.3 Match Format to Link Budget

A data format is the language a device uses to write down a reading before it leaves the chip. The same temperature value can be sent as the text {"temp":23.5} or as a few packed bytes. Both carry the measurement, but they cost very different amounts of bandwidth, radio time, and battery.

The important idea is not "make every message as small as possible." The important idea is fit: the format should be the most readable one that still fits the link you actually have.

Think of packing a suitcase. Labelling every item ("blue shirt", "toothbrush") is easy to unpack later but wastes space; that is JSON. Vacuum-sealing with short labels saves room while staying organized; that is a compact binary format such as CBOR. Compressing everything into the smallest possible bundle is the most efficient but you must remember exactly where each item went; that is custom binary.

A real format choice starts with the device's link, not the developer's preference. A smart-parking sensor on NB-IoT may have enough room for a readable JSON message because the payload limit is measured in hundreds of bytes. A soil probe on LoRaWAN at a low data rate may need CBOR or integer scaling because the available payload is much smaller. A Sigfox tracker with a roughly 12-byte uplink may force a hand-packed layout with a version byte, bit flags, and scaled integers. Use Figure 14.1 to separate those link pressures from the format properties they constrain: begin at the format families, then read across the decision dimensions rather than treating compactness as the only axis.

In Figure 14.1, the left-to-right move from text through standard binary to custom binary reduces how much meaning travels visibly inside the payload. Next, read down the dimensions: payload size and inspection affect day-to-day operation, while schema, change, and tooling determine whether a receiver can keep decoding the message as the fleet evolves. No column wins every row. That is why the running decision starts with measured link pressure and then preserves as much inspectability and standard tooling as the budget allows.

The One-Minute Format Decision

Beginner Examples

• A mains-powered home hub on Wi-Fi can send JSON freely; the link is comfortable and readability speeds development.
• A battery field sensor on a low-rate LoRaWAN profile may benefit from CBOR or scaled integers when its measured application-payload or airtime budget is tight.
• A Sigfox asset tracker with a roughly 12-byte uplink often needs hand-packed custom binary, because even CBOR may not fit.

Format Fit Knowledge Check

A first format choice is defensible when it names the link budget, the expected payload size, and the decode contract the receiver will use.

14.4 Apply It: Choose and Size a Payload

The practical workflow starts with the link budget and ends with a short decode record. The record matters because a format that fit during a Wi-Fi prototype can break once the same device moves to a constrained radio, or once a new field is added.

Walkthrough: From Fields to a Fitting Payload

1. State the link budget. Record the maximum payload, the data rate, the duty-cycle rules, and any per-message cost.
2. List the fields. Name each field with its type, range, and required resolution. This is what actually has to cross the link.
3. Estimate the readable size. Encode the message as JSON first and measure it. If it fits the budget with margin, you may be done.
4. Step down only as needed. If JSON is too large, try a compact binary format such as CBOR, then integer scaling, then a hand-packed custom layout.
5. Check packet cost. Put the application body inside the complete radio frame, then calculate airtime and transmit charge for the selected profile.
6. Document the contract. Write down field order, units, scaling, byte order, invalid codes, and a version so the receiver can decode it later.

The six steps above define the evidence to collect; Figure 14.2 now tests that evidence against four deployment shapes. Read from the roomy-link case toward the severely constrained uplink, watching which constraint first makes a more compact representation worth its additional decode contract.

In Figure 14.2, the Wi-Fi dashboard keeps JSON because the link is comfortable and readable logs have operational value. The gateway and wearable cases introduce tighter payload or energy limits, so measured JSON, CBOR, or a shared schema become candidates rather than automatic answers. Only the final, severely constrained case reaches custom binary, and even there the diagram carries the versioned contract with the bytes. The order connects the workflow to a defensible stopping rule: move to a less inspectable format only when the measured constraint eliminates the simpler choice.

Worked Example: An 8-Byte Vineyard Application Contract

This example is deliberately bounded to an EU863-870 LoRaWAN uplink at DR0 -- LoRa SF12 at 125 kHz -- with no FOpts. Under that profile, the application-payload budget is up to 51 bytes; other regions, data rates, MAC-command use, and device profiles have different limits. The trusted ingestion service attaches authenticated device and session identity outside the application body, so every candidate below encodes the same layout version and four measurements: soil moisture, soil temperature, air temperature, and battery voltage.

The exact sample values are version 1, soil moisture 37.5% VWC, soil temperature 18.6 C, air temperature 23.4 C, and battery voltage 3.612 V. Descriptive JSON is exactly 105 bytes: {"soil_moisture":37.5,"soil_temperature":18.6,"air_temperature":23.4,"battery_voltage":3.612,"version":1}. Short-key minified JSON is exactly 45 bytes: {"v":1,"m":37.5,"st":18.6,"at":23.4,"b":3612}. A valid 25-byte CBOR example uses short text keys with the same integer-scaled values. The custom layout is exactly 8 bytes.

An "8-byte custom binary" claim is credible only after every byte has a stable meaning. VineyardPayloadV1 therefore fixes field order, byte order, scaling, valid ranges, invalid codes, and one golden vector. Use Figure 14.3 to audit that claim before accepting the airtime saving: start with the authenticated context above the payload, then walk byte 0 through byte 7, and finally decode the golden vector. The eight application bytes carry a layout version and four measurements; the trusted ingestion envelope carries stable device identity, session and frame evidence, FPort, and receive time. DevAddr is session- and network-scoped, so the ingestion service maps it to a stable registry identity rather than treating it as a permanent business key.

In Figure 14.3, the upper band keeps identity and session evidence outside the application body, preventing an eight-byte payload claim from quietly omitting receiver context. The byte strip then fixes version, field order, little-endian interpretation, signedness, and scaling. The golden vector at the bottom is the executable bridge between specification and implementation: firmware and backend tests must both recover version 1, 37.5% VWC, 18.6 C, 23.4 C, and 3.612 V. V1 has radio-budget margin but no undefined byte that can silently acquire a new meaning, so a new sensor, device-side sample time, application sequence, or integrity field requires a new version or a larger authenticated envelope. Encoding resolution is not sensor accuracy: 0.1 C storage does not prove a sensor is accurate to 0.1 C.

These packet values use one representative unconfirmed uplink: SF12/125 kHz, coding rate 4/5, eight programmed preamble symbols, explicit header, payload CRC, low-data-rate optimisation, no FOpts, and the minimum 13 bytes of LoRaWAN framing around the application body. Assume this example node draws a measured 44 mA during transmit. The 8-byte contract uses about 43.6% less TX charge than the 45-byte JSON body and about 24.9% less than the 25-byte CBOR body -- not five to six times less. Receive windows, retries, sensor and MCU work, sleep current, and battery self-discharge are excluded. The calculation reports charge; energy in joules additionally requires supply voltage.

Standards note: LoRaWAN application-payload limits are region- and data-rate-specific and may shrink when FOpts carries MAC commands. See the LoRa Alliance RP002-1.0.5 Regional Parameters and TS001-1.0.4 Link Layer specification. Packet airtime should be verified with the selected radio library or the Semtech LoRa Calculator.

Try It: Size Your Own Message

Take a sensor you know. Write the message as JSON and count its UTF-8 bytes, then rewrite the same fields with short keys, then as integer-scaled values. Test each result against this example's 51-byte DR0 application budget, and record which step recovered the most space.

Incremental Practice

Payload Sizing Knowledge Check

A format that fits today still needs a versioned contract so future fields and older receivers can coexist.

14.5 Under the Hood: Encoding Mechanics, Schema, and Cost

The mechanics explain why the formats differ in size and why some changes are safe while others break deployed devices. Each format makes a different trade between readability, self-description, and bytes.

Airtime and Energy

Payload bytes affect battery cost, but application-byte serialization time is not complete packet airtime:

Use the first line only to reason about application bytes in isolation. A deployable estimate uses the complete frame and selected radio profile, then adds receive windows, retries, sensing, MCU work, sleep current, and power-conversion losses as required by the system boundary.

How the Formats Encode

CBOR is an IETF standard (RFC 8949, originally RFC 7049) and pairs naturally with CoAP on constrained networks. Protocol Buffers are a schema-compiled format common in gRPC services where the contract is stable and versioned. MessagePack is a similar binary-JSON bridge that needs no schema. XML remains in some legacy industrial and enterprise systems but is too verbose for tight radio links.

Integer Scaling Beats Floats

A common waste is sending every number as a 64-bit float. A temperature stored as 23.5 in IEEE 754 double precision takes 8 bytes. Stored as temperature x 10 = 235 in a 16-bit integer, it takes 2 bytes with identical decimal resolution when the declared range fits that integer. Figure 14.4 places that field-level saving inside a complete record: read from JSON toward custom binary, but compare the assumptions under each bar before treating their heights as a prediction for another payload.

In Figure 14.4, JSON is largest because names and text syntax travel with the values. CBOR removes text syntax while retaining a self-describing data model; Protocol Buffers replaces names with schema field numbers; custom binary removes nearly all in-band description and relies on the external byte contract. The bars are illustrative rather than universal because field names, numeric widths, schemas, and framing change the totals. Their reliable lesson is the trade: every saved descriptive byte moves more responsibility into a shared schema, version, or decoder—the same responsibility made explicit in the vineyard contract above.

Schema Evolution

IoT fleets are rarely updated all at once. Devices, gateways, and cloud consumers may run different versions for a long time, so the format has to survive staged change.

• Add fields, do not repurpose them. New consumers use new fields; old consumers ignore what they do not understand. Never silently reuse a field for a new meaning.
• Keep meaning stable. A field name or number should keep the same unit and interpretation across versions.
• Version the contract. When a format cannot be self-describing enough, carry a layout or decoder version so the receiver knows how to read it.
• Test old and new. Run old producer to new consumer, new producer to old consumer, and any gateway conversion before deployment.

Common Pitfalls

1. Encoding numbers as strings. Sending "23.5" instead of 23.5 forces string parsing, breaks numeric queries, and adds bytes.
2. Using floats for every field. Eight bytes per value is rarely needed; integer scaling keeps the precision and cuts the size.
3. Choosing binary too early. Custom binary adds encoder, decoder, and debugging cost; pay it only when the link budget actually requires it.
4. Ignoring schema evolution. Adding or renaming a field without a versioning rule can break consumers that have not been updated yet.

Custom Binary Knowledge Check

Format-to-Protocol Fit

A mature format choice is a chain of costs: link budget, readability, self-description, schema stability, and forward compatibility. A trustworthy payload records each one instead of treating the bytes as self-explanatory.

14.6 Summary

Format selection is an evidence chain, not a contest for the fewest bytes. Begin with the complete link and packet budget, preserve an inspectable standard representation while it fits, and record every assumption that a receiver needs to recover units and meaning. When measurement proves that a more compact encoding is necessary, move deliberately: integer scaling, compact standard formats, shared schemas, and custom layouts each save bytes by relocating descriptive information into a stronger external contract. Version that contract and test it across old and new producers and consumers.

The vineyard example shows how those decisions stay connected. Its link profile sets the application budget, measured encodings reveal which representations fit, and the byte diagram makes the smallest candidate independently decodable. The airtime calculation then uses the complete frame rather than equating an application-byte ratio with an energy ratio. That order matters: a compact payload is not trustworthy until both ends agree on its layout, and a byte saving is not an operational benefit until the selected radio profile turns it into measured airtime or charge. The bullets below retain the chapter’s main checks as a quick review after that reasoning.

• A data format choice spans inspection, schema, evolution, tooling, and measured byte cost; there is no single best format or universal ordering.
• Size the link first: maximum payload, data rate, and message cost decide how compact the format must be.
• Begin with the simplest inspectable format that fits, then compare measured encodings and accept added contract complexity only when the evidence justifies it.
• On a constrained link, smaller application bodies can reduce packet airtime and TX charge, but the saving must be calculated from the complete frame and selected radio profile.
• Integer scaling and short keys recover most of the space that naive JSON and 64-bit floats waste.
• Schema evolution must be planned: add fields additively, keep meaning stable, version the contract, and test mixed versions.

14.7 See Also