57  Sensor to Network Pipeline

57.1 Learning Objectives

By the end of this chapter series, you will be able to:

  • Trace the Data Journey: Follow sensor data from physical measurement to network transmission
  • Identify Pipeline Stages: Understand each transformation step from analog signal to network packet
  • Design Data Pipelines: Create efficient sensor-to-network architectures for IoT applications
  • Apply Signal Processing: Implement filtering, sampling, and conditioning techniques
  • Optimize Transmission: Balance data quality, latency, and bandwidth in pipeline design
NoteChapter Series Overview

This topic has been split into three focused chapters for easier learning:

  1. Pipeline Overview and Signal Acquisition - Stages 1-3
    • Complete pipeline architecture
    • Physical measurement (Stage 1)
    • Signal conditioning (Stage 2)
    • ADC conversion (Stage 3)
  2. Processing and Formatting - Stages 4-6
    • Digital processing and calibration (Stage 4)
    • Data formatting: JSON, CBOR, binary (Stage 5)
    • Packet assembly and protocol overhead (Stage 6)
  3. Transmission and Optimization - Stage 7 + Complete Journey
    • Network transmission technologies (Stage 7)
    • Complete sensor-to-cloud trace
    • Latency and energy analysis
    • Optimization strategies and design frameworks

57.2 The 7-Stage Pipeline

Every IoT sensor reading passes through exactly 7 transformation stages:

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#16A085', 'secondaryColor': '#E67E22', 'tertiaryColor': '#ecf0f1', 'clusterBkg': '#fff', 'clusterBorder': '#2C3E50', 'edgeLabelBackground':'#fff'}}}%%
graph LR
    A[Physical<br/>Phenomenon<br/>25.6°C] --> B[Sensor<br/>Element<br/>2.56V]
    B --> C[ADC<br/>Conversion<br/>0x0A00]
    C --> D[Processing<br/>Filtering<br/>25.6°C]
    D --> E[Formatting<br/>JSON/CBOR<br/>payload]
    E --> F[Network<br/>Packet<br/>Transmission]

    style A fill:#E67E22,stroke:#16A085,stroke-width:2px,color:#fff
    style B fill:#2C3E50,stroke:#16A085,stroke-width:2px,color:#fff
    style C fill:#16A085,stroke:#2C3E50,stroke-width:2px,color:#fff
    style D fill:#2C3E50,stroke:#16A085,stroke-width:2px,color:#fff
    style E fill:#16A085,stroke:#2C3E50,stroke-width:2px,color:#fff
    style F fill:#E67E22,stroke:#2C3E50,stroke-width:2px,color:#fff

Figure 57.1: Complete sensor-to-network pipeline showing data transformation at each stage
Stage Name Input Output Covered In
1 Physical Measurement Heat, light, pressure Voltage/current Overview
2 Signal Conditioning Weak analog signal Clean, scaled signal Overview
3 ADC Conversion Analog voltage Digital binary value Overview
4 Digital Processing Raw ADC value Calibrated measurement Processing
5 Data Formatting Numeric value JSON/CBOR/binary Processing
6 Packet Assembly Formatted data Protocol packet Processing
7 Network Transmission Complete packet Radio waves Transmission

57.3 Key Takeaway

Network transmission typically dominates latency (60-70%) and energy consumption (80-90%) - so optimize for fewer transmissions (edge processing) and smaller payloads (binary formats) before optimizing local processing speed.

When a temperature sensor measures 25.6°C, that number doesn’t magically appear in a cloud database. It goes through a remarkable transformation journey:

  1. Physical World: Heat affects the sensor material
  2. Electrical Signal: Material change creates a voltage (e.g., 2.56V)
  3. Digital Number: ADC converts voltage to binary (0000101000000)
  4. Processed Value: Firmware calculates temperature = 25.6°C
  5. Formatted Data: Value becomes JSON: {"temp": 25.6}
  6. Network Packet: Data gets wrapped with headers for transmission
  7. Cloud Storage: Finally stored in a database

Simple Analogy: It’s like sending a letter. You write a message (sensor reading), put it in an envelope (data format), add an address and stamp (packet headers), and mail it (network transmission).