17 Processing Readings Into Packets
From Raw Counts to Reviewable, Network-Ready Packets
17.1 In 60 Seconds
A digital sample becomes reviewable evidence only after firmware gives it meaning, the payload gives it a contract, and packet assembly gives it the wrapper needed for delivery. Optimize the middle pipeline in that order: process the value correctly, encode it clearly, then reduce traffic and bytes where the link cost makes that worth doing.
17.2 Start With the Story
Start with a physical signal that is noisy, delayed, sampled, quantized, calibrated, filtered, packed, and finally sent as a number someone will trust. The core idea in Processing Readings Into Packets is simple: signal processing is the bridge between the physical world and digital evidence, so every sampling, ADC, filter, and calibration choice changes the value that leaves the device. This page focuses that idea on Sensor pipeline chapter on stages 4 to 6: how firmware processing, payload formatting, and packet assembly turn acquired readings into network-ready evidence. In everyday IoT, temperature drift, vibration spikes, audio snippets, and lab traces all become decisions only after their limits and uncertainty are made visible. Start simple: trace one measurement through the chain, keep the raw-to-processed evidence, and move advanced math into the deeper review only when the simple chain no longer explains the result.
Phoebe’s Why
Before processing can decide what a sample means, two upstream physical limits already decided what the sample can contain. Sampling too slowly does not just blur a signal – a fast component folds back and pretends to be a slow one, an error nothing downstream can undo. Quantizing too coarsely rounds every reading onto one of a fixed set of codes, adding a small but bounded amount of noise. Both limits are exactly why the chapter’s own example, the code 2361, cannot mean anything by itself: it is one of \(2^{12}=4{,}096\) possible bins spanning whatever voltage range the ADC saw, and the same bits describe wildly different physical temperatures depending on which transfer function turns counts into degrees.
The Derivation
Sampling copies the signal spectrum every \(f_s\); neighboring copies stay separate only while
\[f_s \geq 2f_{max}\]
A component sampled too slowly folds back to
\[f_{alias} = |f_{signal} - k f_s|\]
For an \(N\)-bit ADC with reference \(V_{ref}\), the bin width and ideal signal-to-noise ratio are
\[q = \frac{V_{ref}}{2^N}, \qquad \mathrm{SNR}_{dB} = 6.02N + 1.76\]
A raw code becomes a voltage, and a linear sensor’s own governing equation turns that voltage into a physical unit – two separate, both-required conversions:
\[V = \mathrm{code}\times\frac{V_{ref}}{2^N}, \qquad V_{out} = V_{0} + S\,T\]
Worked Numbers: Two Readings of the Same Bits
Catalog-typical 12-bit ADC at \(V_{ref}=3.3\) V: \(q = 3300/4096 = 0.806\) mV; ideal \(\mathrm{SNR} = 6.02(12)+1.76 = 74.0\) dB.
- The chapter’s own code, 2361, as a raw ADC count: \(V = 2361\times3300/4096 = 1.90\) V
- Run that voltage through a standard linear sensor’s transfer function (catalog-typical TMP36-class device, \(V_{out}=0.5+0.01\,T\)): \(T = (1.90-0.5)/0.01 = 140\,^{\circ}\mathrm{C}\)
- The same bits, read as the chapter’s other stated meaning – a scaled integer divided by 100: \(T = 2361/100 = 23.6\,^{\circ}\mathrm{C}\)
- The two interpretations of the identical 12 bits disagree by \(140-23.6 = 117\,^{\circ}\mathrm{C}\) – large enough to be operationally dangerous, not just cosmetically wrong
- Separately, a fast component this pipeline must not lose: a vibration spike at 600 Hz sampled at 1,000 Hz folds to \(|600-1(1000)| = 400\) Hz, a false low-frequency reading no processing stage can later separate from a real one
Neither error is a bug in stage four through six’s math. Both are the reason this chapter insists that units, scale, and contract travel with the value: the physics only bounds how much information 4,096 codes and a 1,000 Hz sample clock can carry, and the payload contract is the only thing that tells a receiver which of the many valid readings of those bits is the true one.
17.3 A Digital Sample Is Not Yet Evidence
The first three pipeline stages turn a physical condition into a digital sample. Stages four through six decide what that sample means, how it is encoded, and how it is wrapped for delivery. A raw converter count is not yet something a receiver can trust; it becomes evidence only after processing, formatting, and packet assembly.
The important idea is not “send everything you measure.” The important idea is that these middle stages decide what the network is asked to carry, so a good design sends calibrated, well-labeled values only when the system actually needs them.
If you only need the intuition, this layer is enough: convert raw counts into meaningful values with units, encode them so the receiver can decode them, and add delivery metadata only when it helps. Optimize in this order: avoid unnecessary packets first, keep the payload contract clear, then reduce bytes where the link makes byte count matter.
Think of a lab measurement on its way to a colleague. The number alone is not enough; it needs a unit, a label, and an addressed envelope before it is useful at the other end. Stages four through six are that labeling and packaging step.
A common mistake is to treat these stages as a simple serialization step. They are really where the device turns a measurement into a promise. A value such as 2361 might be a 12-bit ADC count, a register value from a digital sensor, or a scaled integer that already means 23.61 degrees C. The receiver cannot tell which one is true unless the firmware and payload contract make the meaning explicit.
This is also where the system can reduce traffic without hiding important behavior. Firmware may average a noisy stream, suppress unchanged readings, or send only when a threshold or state transition occurs. Those choices are safe only when the review record explains the filter, the send rule, and the event that must still be preserved.
The One-Minute View
Process the sample
Convert raw counts into engineering values, apply calibration and filtering, and decide whether the reading is worth sending at all.
Format the payload
Choose a data contract: field meaning, units, scale, version, and byte layout the receiver can decode now and after a small change.
Assemble the packet
Wrap the payload with the addressing, ordering, integrity, and security metadata the chosen path actually needs.
Beginner Examples
- A temperature sensor should not ship a bare ADC count. Processing turns it into a calibrated value with units before formatting.
- A compact binary payload can be a fine choice, but only if its schema, scale, signedness, and byte order are recorded.
- A battery node that filters noise and sends only state changes still has to prove that its filtering preserves the event of interest.
Sample Context Knowledge Check
If this gives you the mental model, you can stop here. Continue to Practitioner when you need to build or debug the processing and formatting stages.
17.4 Apply It: Process the Value, Then Choose the Contract
The practical work splits cleanly: stage four decides what value is eligible to leave the device, and stage five decides how that value is encoded so a receiver can decode it.
Stage 4: Digital Processing
Digital processing starts with an ADC count or a digital sensor register and turns it into a calibrated quantity, then decides what to do with it.
Start the implementation by naming the source of the number. For an analog channel, record the ADC resolution, voltage reference, sampling interval, analog front-end range, and any hardware gain before writing the conversion formula. For a digital sensor, record the register map, manufacturer scaling rule, update rate, and error or status bits that travel with the reading. This prevents a later decoder from treating a count, a scaled integer, and a physical unit as interchangeable values.
Then separate three decisions that are often mixed together in firmware. Calibration explains how the raw value becomes an engineering value. Cleaning explains which samples are ignored, smoothed, clamped, or marked invalid. The send rule explains when one processed value becomes a network message. Keeping those decisions separate makes code review easier because a battery-saving change to the send rule should not silently change the calibration or hide an out-of-range sensor condition.
The practitioner evidence should be boring and replayable: a known input, the raw observation, the processed output, the payload bytes, and the decoded cloud value. When those five artifacts agree, a gateway, test fixture, or future firmware maintainer can prove the middle pipeline still means the same thing after a schema update or calibration change. This record is also the fastest way to isolate whether a field issue came from sensing, firmware math, gateway decoding, or cloud ingestion.
Calibrate
Apply scale, offset, units, and sensor-specific correction so downstream systems see engineering values, not converter counts.
Clean
Filter noise, reject impossible spikes, and respect settling time before creating alerts or summary values.
Decide
Send on threshold, trend, event, schedule, or state change. Not every valid sample needs to become a packet.
The basic conversion is a linear calibration:
Engineering Value = Raw Count x Scale + Offset
That formula is only trustworthy when the review record also states the ADC reference, bit depth, sensor range, unit, calibration method or date, and expected uncertainty. If a payload carries a raw count on purpose, the contract must carry enough metadata for the receiver to reconstruct the same conversion.
For a simple analog channel, the implementation record should let another engineer replay the conversion. Record whether the input was sampled as a 10-bit, 12-bit, or 16-bit value, which voltage reference or full-scale range was used, and whether the sensor output is linear over the operating range. If the firmware applies a two-point calibration, keep the two known inputs, the observed counts, and the resulting scale and offset with the firmware version that used them.
Filtering also needs a testable reason. A moving average can reduce random noise, but it can delay a fast event. A median filter can reject a single spike, but it can also hide a short pulse if the window is too long. A threshold with hysteresis can prevent repeated alarms near a boundary, but the chosen high and low limits should be traceable to the physical behavior being monitored. The processing stage is reviewable when the team can point to the failure it prevents and the event it still allows through.
Finally, the send decision should be written as a rule, not as an intuition. Examples include “send every 10 minutes,” “send when temperature changes by at least 0.5 degrees C,” “send when the door state changes,” or “send the minimum, maximum, and mean for each one-minute window.” Each rule creates a different packet pattern, so it affects battery life, gateway load, and how much evidence is available during an incident review.
Stage 5: Data Formatting
Formatting turns a processed value into a payload contract. The choice controls debugging, decoder behavior, compatibility, and how future fields are added, not just byte size.
Readable payload
Text formats are easy to inspect at APIs, logs, and early prototypes. They cost more bytes and still need units, versioning, and field-name discipline.
Structured binary
Self-describing binary such as CBOR-style maps cuts overhead while keeping typed fields. Teams still need decoders, tests, and schema-change rules.
Fixed binary layout
Compact byte layouts work when the schema is stable and every receiver shares the same version, units, signedness, width, and byte order.
Before a format is treated as stable, the payload contract should record field names or byte offsets, units and scaling, signedness, width and byte order for numeric fields, timestamp meaning and clock source, schema version and optional-field policy, how missing or invalid or saturated values are represented, and decoder tests using known example payloads.
The format choice should match the boundary. JSON is useful when humans inspect API logs, browser tools, or early prototype traces. CBOR-style typed maps can keep field names or numeric keys while reducing the byte cost. A fixed binary layout is appropriate only when every receiver has the same decoder and the team can afford stricter migration rules. Moving from one form to another at a gateway is often cleaner than forcing the same representation onto a tiny radio hop and a cloud debugging workflow.
payload_version: 2
temperature_c: int16, scale 0.01 C per count
humidity_pct: uint16, scale 0.01 percent per count
status_flags: uint8
timestamp_s: uint32, device monotonic seconds
byte_order: network (big-endian)
For the sample contract above, the decoder should include at least one golden payload with expected decoded values. That fixture catches byte-order mistakes, signedness mistakes, missing scale factors, and schema-version drift. It is especially important when a field such as temperature_c is stored as an integer count but presented to users as a decimal engineering value.
A gateway is a natural translation boundary: keep the constrained device hop compact and documented, then re-encode to a readable, inspectable contract where humans and cloud services consume the data. The gateway should preserve the original packet time, device identity, schema version, and any status flags so the cloud record can still be traced back to the device-side evidence.
Formatting Boundary Knowledge Check
If you can process a value and choose a defensible contract, you can stop here. Continue to Under the Hood for packet assembly and the whole-transaction view.
17.5 Under the Hood: Packet Assembly and the Whole Transaction
Packet assembly wraps the payload with the metadata the chosen path needs. Headers are not automatically waste: they carry addressing, type, length, sequence, reliability, integrity, and security context. The design question is whether each wrapper is needed at that boundary.
Different stacks put this information in different places. MQTT commonly uses a topic and message metadata above TCP/IP. CoAP commonly uses a method, path, message ID, token, and UDP/IP headers. BLE GATT uses services, characteristics, attributes, and link-layer behavior. LoRaWAN separates the application payload from MAC and network-session metadata. The exact names change, but the review question stays the same: which layer tells the receiver what the payload is, who sent it, whether it is fresh, and whether it arrived intact?
Sequence numbers and timestamps are not just bookkeeping. A sequence number helps a receiver detect a lost or duplicated packet even when the payload value looks normal. A timestamp or monotonic sample counter helps distinguish “the room is still 21.4 degrees C” from “the gateway is replaying an old reading.” If the application suppresses unchanged readings, these fields become even more important because absence of a packet is part of the signal.
Packet assembly should also be idempotent from the receiver’s point of view. If a retry arrives after the first copy was accepted, the application should have enough identity, sequence, or sample-time information to avoid counting the same physical event twice. That is why “smallest packet” is not the same as “best packet”: a few bytes of identity can prevent bad analytics, duplicate alerts, and confusing incident timelines.
Application metadata
Topic, resource path, message type, schema version, device ID, and content type tell the receiver what the payload means.
Delivery metadata
Sequence numbers, acknowledgements, retry behavior, and timestamps reveal missing, duplicated, delayed, or stale readings.
Protection metadata
Integrity checks, authentication, encryption, and replay protection depend on the threat model and the protocol boundary.
Payload Size Is Not the Whole Transaction
When reviewing overhead, measure the entire transaction, not just payload bytes:
- radio or interface wake time;
- association, join, or session setup when it recurs often;
- request and response behavior, plus acknowledgement and retry policy;
- receive windows or listening time;
- security handshake and key-refresh behavior;
- the number of packets created by one useful event.
The practical rule: if the system sends the same routine value every few seconds, reducing packet count usually beats shaving a few bytes. If it sends rare but latency-sensitive events, preserve clear identifiers, timestamps, and reliability metadata even when that adds bytes.
Byte count still matters, but it is only one part of the cost model. A shorter payload can be a bad trade if it removes the schema version or status flags needed to debug the system later. A longer payload can be acceptable if messages are rare and the added fields prevent unsafe ambiguity. A good packet budget therefore records the useful application bytes, the wrapper bytes, the number of packets per useful event, and the receive or acknowledgement behavior that happens around the transmission.
Security metadata should be reviewed at the same boundary as the transport decision. A local debug UART, a BLE connection to a phone, a LoRaWAN uplink, and an MQTT message over TLS expose different risks and different trust anchors. Integrity checks, message authentication codes, encryption, replay windows, and key rotation all add operational cost. Removing them to save bytes is only defensible when the threat model and deployment boundary make that safe.
Middle-Pipeline Review Record
Common Pitfalls
- Shipping raw counts without a contract. They are valid only when receivers know reference, bit depth, scale, offset, unit, and calibration. Otherwise they are small but ambiguous.
- Choosing a format by byte count alone. Consider who debugs it, how the schema evolves, which libraries exist, and where a gateway can translate between compact and readable forms.
- Treating every header as waste. Remove unnecessary wrappers, but keep the addressing, ordering, integrity, and security metadata the real failure modes demand.
Transaction Cost Knowledge Check
At this depth, the middle pipeline turns measurements into evidence. The best design is not the smallest byte string; it is the smallest reviewable record that preserves meaning, supports maintenance, and avoids unnecessary transmissions.
17.6 Summary
- Digital processing converts raw counts into calibrated, filtered, decision-ready values with units.
- Send rules should suppress redundant packets while preserving the events that matter.
- Data formatting is a contract: units, scale, version, byte order, and worked examples matter as much as byte count.
- Packet assembly adds the addressing, ordering, integrity, and security context the chosen path needs.
- A gateway can keep constrained links compact while giving cloud systems readable, inspectable records.
- Overhead should be measured across the whole transaction, not just the payload bytes.
Stages four to six turn measurements into evidence. The best middle-pipeline design is the smallest reviewable record that preserves meaning, supports maintenance, and avoids unnecessary transmissions.
17.7 See Also
Pipeline and Signal Acquisition
Review stages one to three: physical measurement, signal conditioning, and ADC conversion.
Pipeline Transmission
Continue to stage seven: link choice, transaction cost, latency, retries, and delivery evidence.
Data Formats for IoT
Go deeper on text, structured binary, fixed binary, and schema-evolution choices for the payload contract.