By the end of this chapter series, you will be able to:
Core Concept: Every IoT sensor reading undergoes a 7-stage transformation journey: Physical Phenomenon to Electrical Signal to Digital Value to Processed Data to Formatted Payload to Protocol Packet to Radio Waves. Understanding this pipeline is essential for debugging, optimization, and system design.
Why It Matters: When your temperature sensor shows “25.6C” in the cloud dashboard, that number traveled through 7 distinct transformations - each adding latency, consuming energy, and potentially introducing errors. Engineers who understand this pipeline can diagnose why readings are delayed, why battery life is poor, or why data is corrupted.
Key Takeaway: Network transmission dominates both latency (60-70%) and energy consumption (80-90%) in most IoT systems. Optimizing the pipeline means reducing transmissions (edge processing) and shrinking payloads (binary formats) rather than speeding up local processing.
Read on for the full picture, or jump directly to a specific chapter below.
Explore Further:
This topic has been split into three focused chapters for easier learning:
Meet the Sensor Squad Characters!
Sammy says: “Hey friends! Today I’m going to show you how I send a temperature reading all the way to the cloud! It’s like a super exciting relay race!”
Lila asks: “But Sammy, when you feel that it’s warm, how does that number end up on someone’s phone?”
Sammy explains: “Great question! Let me show you my 7-step journey!”
| Step | What Happens | Relay Race Analogy |
|---|---|---|
| 1 | I feel the heat! | Runner 1 starts with a baton |
| 2 | Heat becomes electricity | Baton passed to Runner 2 |
| 3 | Electricity becomes a number | Runner 3 counts the baton |
| 4 | Number gets cleaned up | Runner 4 polishes the baton |
| 5 | Number put in an envelope | Runner 5 wraps it nicely |
| 6 | Envelope gets an address | Runner 6 adds the shipping label |
| 7 | Zoom! Off to the cloud! | Helicopter delivers the package! |
Max shouts: “Tell us the whole adventure!”
Sammy tells the story:
It was a sunny day, and I could feel the warmth - 25.6 degrees Celsius!
First, the heat made me produce a tiny bit of electricity - like squeezing a lemon makes juice!
Then my friend ADC (the number counter) turned that electricity into the number 2560.
After that, the smart microcontroller divided by 100 to get 25.6 - the real temperature!
“But wait!” said Lila. “You can’t just yell a number into space!”
“That’s right!” I replied. “So I wrapped my number in a JSON package: {”temp”: 25.6}.”
Then the network helper added addresses - like putting stamps on a letter!
Finally, WHOOOOSH! The radio sent it flying through the air to the cloud!
Bella says: “Did you know? The wireless sending part (step 7) uses 80% of all the battery power! That’s why smart sensors try to send fewer messages - it’s like taking the stairs vs. riding a helicopter. The helicopter is faster but uses way more energy!”
| Word | What It Means |
|---|---|
| Pipeline | Steps that data follows, like a water slide with different sections |
| ADC | Analog-to-Digital Converter - turns electricity into numbers |
| Payload | The actual message (like the present inside a wrapped box) |
| Latency | How long the whole journey takes |
Max challenges you: “Play the telephone game with your friends! Whisper a message through 7 people. Did it arrive correctly? That’s like data going through the pipeline - each step needs to be accurate!”
Every IoT sensor reading passes through up to 7 transformation stages:
Integrated digital sensors (e.g., BME280, SHT40) with I2C or SPI interfaces handle Stages 1-3 internally — the sensor chip performs measurement, conditioning, and ADC conversion on-die and outputs a digital value directly. In these cases, firmware interacts starting at Stage 4.
Figure 1: Chapter-specific pipeline overview showing all seven transformations, the domains they belong to, and why transmission usually dominates the system budget.
Network transmission typically dominates latency (60-70%) and energy consumption (80-90%) — so optimize for fewer transmissions (edge processing) and smaller payloads (binary formats) before optimizing local processing speed.
Concrete example — a LoRaWAN temperature sensor: ADC conversion ~0.1 ms, processing ~2 ms, formatting ~1 ms, transmission ~50 ms. Transmission accounts for 50/53.1 ms = 94% of total pipeline time.
Adjust the time spent in each pipeline stage to see how transmission dominates total latency:
Check your understanding of why transmission dominates the pipeline energy budget:
When a temperature sensor measures 25.6°C, that number doesn’t magically appear in a cloud database. It goes through a remarkable transformation journey:
{"temp": 25.6}Simple Analogy: It’s like sending a letter. You write a message (sensor reading), put it in an envelope (data format), add an address and stamp (packet headers), and mail it (network transmission).
Scenario: A DHT22 temperature/humidity sensor on a NodeMCU (ESP8266) sends data to a cloud dashboard over Wi-Fi every 60 seconds.
Pipeline Flow:
Stage 1: DHT22 measures 23.4°C, 65% humidity
Stage 2-3: (Internal to DHT22 - outputs digital values via single-wire protocol)
Stage 4: ESP8266 reads digital values: temp=234, hum=650
Stage 5: Format as JSON: {"temp":23.4,"hum":65}
Stage 6: Wrap in HTTP POST with headers (~120 bytes total)
Stage 7: Wi-Fi transmits packet (~2 ms at 54 Mbps)Key Learning: Even this simple case has 7 stages. Most beginners only think about “read sensor, send to cloud” (Stages 1 and 7), missing the crucial formatting and processing steps in between.
Scenario: 10 soil moisture sensors in a greenhouse, sampled every 5 minutes. Instead of sending 10 individual readings, the edge gateway aggregates and computes statistics before transmitting.
Pipeline Flow:
Stage 1-3: Each of 10 sensors measures moisture (outputs 0-1023 ADC value)
Stage 4: Edge gateway applies calibration: ADC → % moisture
Sensor 1: 512 → 45%, Sensor 2: 678 → 62%, ... Sensor 10: 423 → 38%
Stage 4 (cont): Compute aggregates:
- Average: 49.3%
- Min: 38%
- Max: 67%
- Variance: 8.2%
Stage 5: Format as compact CBOR array (18 bytes vs 120 bytes for 10 individual JSON objects)
Stage 6: LoRaWAN packet assembly (31 bytes total with headers)
Stage 7: Transmit once over LoRaWAN (~200 ms air time)Key Learning: Edge processing (Stage 4) reduces 10 transmissions to 1 transmission. Network energy savings: 90%. This is why edge computing matters in IoT pipelines.
Compare: If each sensor transmitted individually: - 10 transmissions × 200 ms = 2000 ms total air time - 10 transmissions × 120 mA = 1200 mA-ms = 0.333 mAh - Aggregated: 1 transmission × 200 ms × 120 mA = 24 mA-ms = 0.0067 mAh - Energy savings: 98%
Scenario: Industrial facility with mixed sensor types: Modbus RTU temperature sensors, BLE beacons for asset tracking, and LoRaWAN environmental sensors. A gateway collects from all three and forwards to cloud via cellular.
Pipeline Flow:
Path A (Modbus sensors):
Stage 1-3: (Internal to Modbus sensor)
Stage 4: Gateway polls Modbus registers every 10 seconds (16 sensors)
Stage 5: Translate Modbus registers → JSON array
Stage 6: Buffer in gateway memory
Path B (BLE beacons):
Stage 1-7: BLE beacons broadcast every 1 second
Stage 4: Gateway BLE scanner receives advertisements
Stage 5: Parse BLE payload (Eddystone format) → Extract UUID, RSSI, battery
Stage 6: Buffer in gateway memory
Path C (LoRaWAN sensors):
Stage 1-7: LoRaWAN devices transmit every 15 minutes
Stage 4: Gateway receives LoRaWAN uplinks
Stage 5: Decrypt, verify MIC, extract payload
Stage 6: Buffer in gateway memory
Unified Path (Gateway → Cloud):
Stage 4 (gateway edge): Every 60 seconds, aggregate all buffered data:
- 16 Modbus readings
- ~60 BLE beacon sightings
- ~4 LoRaWAN packets
Stage 5: Format as single MQTT message with JSON payload (~800 bytes)
Stage 6: MQTT packet with QoS 1 + TLS overhead (~1200 bytes total)
Stage 7: 4G LTE transmission (~15 ms)Key Learning: Multi-protocol gateways perform protocol translation (Stage 5) and temporal aggregation (Stage 4) to reduce cellular data costs. Without aggregation, this gateway would send 140 messages/min instead of 1 message/min = 99.3% reduction in cellular transmissions.
Cost Impact:
Gateway Aggregation ROI Calculation: Industrial IoT deployment with 200 sensors:
Scenario A - Direct Cellular (each sensor has SIM): - Cellular data plan: $5/month/sensor × 200 = $1,000/month - Transmission: 1 message/min × 50 bytes = 72 MB/month/sensor - Network: 200 separate cellular connections
Scenario B - Gateway Aggregation (1 gateway, local protocols): - Gateway cellular: $20/month (business plan, 10 GB) - Sensor hardware: BLE/Zigbee ($4 vs $15 for cellular module) = $2,200 savings - Gateway cost: $500 one-time
\[\text{Monthly savings} = 1000 - 20 = \$980\]
\[\text{Payback period} = \frac{500\text{ (gateway)}}{980\text{/month}} = 0.51\text{ months}\]
Aggregation efficiency: 200 sensors × 60 msg/hr = 12,000 msg/hr
Without aggregation: \(12,000 \times 50 = 600\) KB/hr
With 1-minute aggregation batches: \(60 \times 50\text{ bytes} = 3\) KB (50 sensors/batch) × 4 batches = 12 KB/hr + 200 bytes overhead = 12.2 KB/hr
\[\text{Data reduction} = \frac{600 - 12.2}{600} \times 100\% = 98\%\]
Annual TCO: Scenario A = $12,000. Scenario B = $240 + $500 amortized = $740. Savings: $11,260/year
Scenario: A battery-powered temperature sensor transmits readings every 60 seconds over LoRaWAN (SF9, 125 kHz BW, 10 km range). Battery: 2000 mAh at 3.3V. Let’s compare JSON vs CBOR encoding across the full pipeline.
Data Payload:
JSON Encoding (Stage 5):
{"id":"TMP-00127","temp":23.6,"hum":67.2,"batt":3.14,"ts":1707398400}CBOR Encoding (Stage 5):
A5 # map(5)
62 6964 # "id"
69 544D502D3030313237 # "TMP-00127"
64 74656D70 # "temp"
FA 41BD3333 # float(23.6)
63 68756D # "hum"
FA 4286CCCD # float(67.2)
64 62617474 # "batt"
FA 4048F5C3 # float(3.14)
62 7473 # "ts"
1A 65C6F000 # uint(1707398400)Packet Assembly (Stage 6) - LoRaWAN:
| Component | JSON Overhead | CBOR Overhead |
|---|---|---|
| Payload | 72 bytes | 37 bytes |
| LoRaWAN FHDR | 13 bytes | 13 bytes |
| Frame options | 0 bytes | 0 bytes |
| MIC (authentication) | 4 bytes | 4 bytes |
| Total packet | 89 bytes | 54 bytes |
Network Transmission (Stage 7) - LoRaWAN SF9:
LoRaWAN air time formula (simplified):
T_packet = T_preamble + T_payload
T_preamble = (8 + 4.25) * 2^SF / BW = 12.25 * 2^9 / 125000 = 50.2 ms
T_symbol = 2^SF / BW = 2^9 / 125000 = 4.096 ms
T_payload = n_symbols * T_symbolFor SF9, BW=125kHz, CR=4/5: - JSON (89 bytes): 26 symbols → 157 ms air time - CBOR (54 bytes): 17 symbols → 120 ms air time - Air time savings: 37 ms (24% reduction)
Energy Consumption (per transmission):
LoRaWAN module (e.g., RFM95W) power consumption: - TX at +20 dBm: 120 mA during transmission - MCU processing: 15 mA during encoding/formatting (assume 10 ms) - Sleep mode: 0.2 µA (negligible)
JSON path:
CBOR path:
Long-Term Impact:
Device transmits every 60 seconds = 1,440 messages/day = 525,600 messages/year
| Metric | JSON | CBOR | CBOR Benefit |
|---|---|---|---|
| Energy per message | 0.00527 mAh | 0.00403 mAh | 23.5% savings |
| Energy per day | 7.59 mAh | 5.80 mAh | 1.79 mAh/day |
| Energy per year | 2,770 mAh | 2,117 mAh | 653 mAh/year |
| Battery life (2000 mAh) | 263 days | 345 days | +82 days (31% longer) |
| Air time per day | 226 seconds | 173 seconds | 53 sec/day |
| Duty cycle @ 1% (EU868) | 0.26% | 0.20% | More headroom for retries |
Key Insights:
Lesson: In the sensor-to-network pipeline, format choice at Stage 5 directly impacts battery life at Stage 7. The 23.5% energy savings from CBOR comes almost entirely from reduced transmission time, not from faster encoding. This validates the chapter’s key principle: optimize for fewer, smaller transmissions.
Teams often spend days tuning ADC timing or filter coefficients while transmission still consumes 80-90% of the energy budget. Profile the full path first, then target transmission frequency, payload size, and retry behaviour before micro-optimising local compute.
When a BME280 or TMP117 is used, the firmware is not dealing with a raw analog waveform anymore. Treating those parts like discrete analog chains leads to unnecessary circuitry, duplicated calibration logic, and misleading debugging assumptions about where an error originates.
Shipping each sensor sample individually makes packet overhead and radio airtime dominate cost. Aggregate, threshold, or summarize at the edge whenever the application allows it, especially for slow-changing environmental data.
The sensor-to-network pipeline describes the up-to-7-stage transformation that every IoT reading undergoes — from a physical phenomenon (heat, light, pressure) through analog conditioning and digitisation, into processed, formatted, and packetised data, and finally across a wireless link to the cloud. Integrated digital sensors collapse the first three stages into one chip, but the later stages remain the same.
The single most important optimisation insight is that network transmission dominates both latency and energy. Reducing the number of transmissions (via edge processing and aggregation) and shrinking payload size (via binary formats like CBOR) yields far greater savings than speeding up local computation.
Cross-module connection: This pipeline connects to Protocol Selection via transmission characteristics. See Protocol Selection Framework for choosing protocols based on Stage 7 requirements.
Continue with the detailed three-part walkthrough of the sensor-to-network pipeline: