58  Pipeline Overview and Signal Acquisition

58.1 Learning Objectives

By the end of this chapter, you will be able to:

  • Understand the Pipeline Architecture: Identify the 7 stages of the sensor-to-network data journey
  • Trace Physical Measurements: Follow sensor data from phenomenon detection to electrical signals
  • Apply Signal Conditioning: Implement amplification, filtering, and offset adjustment
  • Perform ADC Conversion: Convert analog signals to digital values with appropriate resolution

This Series: - Sensor Pipeline Index - Series overview - Processing and Formatting - Stages 4-6 - Transmission and Optimization - Stage 7 and complete journey

Fundamentals: - Data Representation - Binary encoding and number systems - Signal Processing Essentials - ADC and filtering deep dive

Sensing: - Sensor Fundamentals - Sensor types and characteristics - Sensor Circuits - Signal conditioning circuits

58.2 Prerequisites

Before diving into this chapter, you should be familiar with:

When a temperature sensor measures 25.6°C, that number doesn’t magically appear in a cloud database. It goes through a remarkable transformation journey:

  1. Physical World: Heat affects the sensor material
  2. Electrical Signal: Material change creates a voltage (e.g., 2.56V)
  3. Digital Number: ADC converts voltage to binary (0000101000000)
  4. Processed Value: Firmware calculates temperature = 25.6°C
  5. Formatted Data: Value becomes JSON: {"temp": 25.6}
  6. Network Packet: Data gets wrapped with headers for transmission
  7. Cloud Storage: Finally stored in a database

Simple Analogy: It’s like sending a letter. You write a message (sensor reading), put it in an envelope (data format), add an address and stamp (packet headers), and mail it (network transmission). Each step transforms and wraps your original message.

This chapter covers stages 1-3: how physical phenomena become digital values.

TipMVU: The IoT Data Pipeline

Core Concept: Every IoT sensor reading passes through exactly 7 transformation stages: physical sensing, signal conditioning, ADC conversion, digital processing, data formatting, packet assembly, and network transmission.

Why It Matters: Understanding this pipeline is essential for troubleshooting IoT deployments. When data looks wrong at the cloud, you can trace back through stages to find where corruption or loss occurred. It also reveals that most IoT “bugs” are actually misconfigured transformations - wrong calibration offsets, incorrect byte ordering, or mismatched data types between stages.

Key Takeaway: The first three stages (physical sensing, conditioning, ADC) determine the fundamental data quality. No amount of cloud processing can recover information lost to poor signal conditioning or insufficient ADC resolution.

58.3 The Complete Data Pipeline

⏱️ ~8 min | ⭐ Foundational | 📋 P02.C04.U01

58.3.1 Pipeline Overview

The sensor-to-network pipeline consists of distinct stages, each transforming data:

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#16A085', 'secondaryColor': '#E67E22', 'tertiaryColor': '#ecf0f1', 'clusterBkg': '#fff', 'clusterBorder': '#2C3E50', 'edgeLabelBackground':'#fff'}}}%%
graph LR
    A[Physical<br/>Phenomenon<br/>25.6°C] --> B[Sensor<br/>Element<br/>2.56V]
    B --> C[ADC<br/>Conversion<br/>0x0A00]
    C --> D[Processing<br/>Filtering<br/>25.6°C]
    D --> E[Formatting<br/>JSON/CBOR<br/>payload]
    E --> F[Network<br/>Packet<br/>Transmission]

    style A fill:#E67E22,stroke:#16A085,stroke-width:2px,color:#fff
    style B fill:#2C3E50,stroke:#16A085,stroke-width:2px,color:#fff
    style C fill:#16A085,stroke:#2C3E50,stroke-width:2px,color:#fff
    style D fill:#2C3E50,stroke:#16A085,stroke-width:2px,color:#fff
    style E fill:#16A085,stroke:#2C3E50,stroke-width:2px,color:#fff
    style F fill:#E67E22,stroke:#2C3E50,stroke-width:2px,color:#fff

Figure 58.1: Complete sensor-to-network pipeline showing data transformation at each stage

This variant shows the same pipeline with timing information - critical for understanding IoT system responsiveness:

%% fig-alt: "Sensor-to-network pipeline with timing breakdown showing latency at each stage. Physical Measurement takes 1ms (sensor response time). Signal Conditioning adds 0.1ms (op-amp settling). ADC Conversion takes 10μs for 12-bit conversion. Processing/Filtering adds 5ms for moving average calculation. Formatting to CBOR takes 2ms. Network Transmission (LoRaWAN SF7) takes 41ms for packet transmission. Total end-to-end latency is 49ms from physical event to data reaching gateway. This timing perspective helps engineers identify bottlenecks and optimize for latency-sensitive applications."
%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#16A085', 'secondaryColor': '#E67E22', 'tertiaryColor': '#ecf0f1', 'fontSize':'11px'}}}%%
flowchart TB
    subgraph stage1["STAGE 1: Physical<br/>⏱️ 1ms"]
        P1["Sensor response time"]
    end

    subgraph stage2["STAGE 2: Conditioning<br/>⏱️ 0.1ms"]
        P2["Op-amp settling"]
    end

    subgraph stage3["STAGE 3: ADC<br/>⏱️ 10μs"]
        P3["12-bit conversion"]
    end

    subgraph stage4["STAGE 4: Processing<br/>⏱️ 5ms"]
        P4["Moving average filter"]
    end

    subgraph stage5["STAGE 5: Formatting<br/>⏱️ 2ms"]
        P5["CBOR encoding"]
    end

    subgraph stage6["STAGE 6: Network<br/>⏱️ 41ms"]
        P6["LoRaWAN SF7 TX"]
    end

    total["TOTAL LATENCY:<br/>~49ms sensor to gateway"]

    stage1 --> stage2 --> stage3 --> stage4 --> stage5 --> stage6
    stage6 --> total

    style stage1 fill:#E67E22,stroke:#2C3E50,color:#fff
    style stage2 fill:#16A085,stroke:#2C3E50,color:#fff
    style stage3 fill:#16A085,stroke:#2C3E50,color:#fff
    style stage4 fill:#2C3E50,stroke:#16A085,color:#fff
    style stage5 fill:#16A085,stroke:#2C3E50,color:#fff
    style stage6 fill:#E67E22,stroke:#2C3E50,color:#fff
    style total fill:#2C3E50,stroke:#E67E22,color:#fff

Figure 58.2: Pipeline with timing breakdown showing where latency occurs

Why this variant helps: The original shows what data transformations happen. This variant shows how long each stage takes. For latency-sensitive applications (industrial control, safety systems), knowing that network transmission dominates latency (41ms out of 49ms total) guides optimization decisions - you’d focus on choosing a faster protocol rather than optimizing ADC speed.

58.4 Stage 1: Physical Measurement

⏱️ ~5 min | ⭐ Foundational | 📋 P02.C04.U02

58.4.1 Sensor Transduction

Sensors detect physical phenomena:

  • Temperature: Thermistor resistance changes with heat
  • Light: Photodiode current varies with illumination
  • Pressure: Piezo element voltage changes under force
  • Motion: Accelerometer capacitance shifts with movement

Output: Analog electrical signal (voltage or current)

58.4.2 Real-World Example: Industrial Vibration Sensor

Follow a vibration reading from sensor to cloud in a manufacturing plant monitoring a critical motor bearing:

1. Sensing: MEMS accelerometer (ADXL345) detects vibration, outputs 0-3.3V analog signal proportional to acceleration

2. Signal Conditioning: - Anti-aliasing filter at 500Hz removes high-frequency noise - Op-amp provides 2× gain for better ADC utilization

3. ADC Conversion: - 12-bit ADC samples at 1kHz - Voltage of 1.65V → raw value: (1.65/3.3) × 4096 = 2048

This physical measurement forms the foundation of all subsequent processing. Poor sensor selection or placement compromises all downstream stages.

58.5 Stage 2: Signal Conditioning

⏱️ ~7 min | ⭐⭐ Intermediate | 📋 P02.C04.U03

58.5.1 Analog Signal Processing

Raw sensor signals often need adjustment:

Amplification: - Sensor output: 0-100mV - ADC input range: 0-3.3V - Solution: Op-amp amplifies by 33×

Filtering: - Remove high-frequency noise - Low-pass filter smooths signal - Example: RC filter with cutoff at 10 Hz

Offset Adjustment: - Sensor outputs -1V to +1V - ADC accepts 0V to 3.3V - Solution: Add 1.65V offset

Output: Clean analog signal matching ADC input range

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#16A085', 'secondaryColor': '#E67E22', 'tertiaryColor': '#ecf0f1', 'clusterBkg': '#fff', 'clusterBorder': '#2C3E50', 'edgeLabelBackground':'#fff'}}}%%
flowchart LR
    A[Sensor<br/>0-100mV] --> B[Amplifier<br/>×33]
    B --> C[Low-Pass<br/>Filter<br/>10 Hz]
    C --> D[Offset<br/>Adjust<br/>+1.65V]
    D --> E[ADC<br/>0-3.3V<br/>Range]

    style A fill:#E67E22,stroke:#2C3E50,stroke-width:2px,color:#fff
    style B fill:#2C3E50,stroke:#16A085,stroke-width:2px,color:#fff
    style C fill:#16A085,stroke:#2C3E50,stroke-width:2px,color:#fff
    style D fill:#2C3E50,stroke:#16A085,stroke-width:2px,color:#fff
    style E fill:#E67E22,stroke:#2C3E50,stroke-width:2px,color:#fff

Figure 58.3: Signal conditioning circuit flow: amplification, filtering, and offset adjustment prepare sensor output for ADC conversion

This variant uses an everyday cooking analogy to explain why each conditioning step is necessary:

%% fig-alt: "Analogy diagram comparing signal conditioning to preparing ingredients for cooking. Raw sensor signal is like raw ingredients - too small, mixed with debris, not ready to use. Amplification is like chopping larger pieces - makes the signal the right size for the ADC to work with. Filtering is like washing vegetables - removes unwanted noise and debris from the signal. Offset adjustment is like seasoning to taste - shifts the signal to the range the ADC expects. Final output is like prepared ingredients ready for the pot - clean signal in the right voltage range. This kitchen analogy helps beginners understand that each step transforms raw sensor output into something the ADC can properly digest."
%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#16A085', 'secondaryColor': '#E67E22', 'tertiaryColor': '#7F8C8D'}}}%%
graph TB
    subgraph KITCHEN["Kitchen Analogy"]
        K1["Raw Ingredients<br/>Too small, mixed debris"]
        K2["Chop Larger<br/>Right size pieces"]
        K3["Wash Clean<br/>Remove debris"]
        K4["Season<br/>Adjust to taste"]
        K5["Ready to Cook<br/>Perfect prep"]
    end

    subgraph SIGNAL["Signal Conditioning"]
        S1["Raw Sensor<br/>0-100mV (too weak)"]
        S2["Amplify ×33<br/>0-3.3V (usable)"]
        S3["Filter 10Hz<br/>Remove noise"]
        S4["Offset +1.65V<br/>Center in ADC range"]
        S5["Ready for ADC<br/>0-3.3V clean"]
    end

    K1 -.->|"Same as"| S1
    K2 -.->|"Same as"| S2
    K3 -.->|"Same as"| S3
    K4 -.->|"Same as"| S4
    K5 -.->|"Same as"| S5

    K1 --> K2 --> K3 --> K4 --> K5

    style K1 fill:#E67E22,stroke:#2C3E50,stroke-width:2px,color:#fff
    style K2 fill:#16A085,stroke:#2C3E50,stroke-width:2px,color:#fff
    style K3 fill:#16A085,stroke:#2C3E50,stroke-width:2px,color:#fff
    style K4 fill:#16A085,stroke:#2C3E50,stroke-width:2px,color:#fff
    style K5 fill:#E67E22,stroke:#2C3E50,stroke-width:2px,color:#fff
    style S1 fill:#7F8C8D,stroke:#2C3E50,stroke-width:1px,color:#fff
    style S2 fill:#7F8C8D,stroke:#2C3E50,stroke-width:1px,color:#fff
    style S3 fill:#7F8C8D,stroke:#2C3E50,stroke-width:1px,color:#fff
    style S4 fill:#7F8C8D,stroke:#2C3E50,stroke-width:1px,color:#fff
    style S5 fill:#7F8C8D,stroke:#2C3E50,stroke-width:1px,color:#fff

Figure 58.4: Signal conditioning as kitchen preparation: amplification makes signals the right “size,” filtering removes “debris,” and offset adjusts to the ADC’s “taste”

Why this variant helps: The original shows technical values (33× gain, 10 Hz cutoff). This analogy connects to everyday experience - everyone understands that raw ingredients need preparation before cooking. It helps beginners grasp why each step exists before diving into how it works.

58.6 Stage 3: Analog-to-Digital Conversion

⏱️ ~8 min | ⭐⭐ Intermediate | 📋 P02.C04.U04

58.6.1 ADC Quantization

ADC converts continuous voltage to discrete digital value:

Example: 12-bit ADC, 3.3V reference

Sensor voltage: 2.56V
ADC reading: (2.56 / 3.3) × 4096 = 3174 = 0x0C66

Key Parameters: - Resolution: 8-bit (256 levels), 12-bit (4096), 16-bit (65536) - Sampling rate: 10 Hz, 100 Hz, 1 kHz - Reference voltage: 3.3V, 5V, internal

Output: Binary number representing voltage

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#16A085', 'secondaryColor': '#E67E22', 'tertiaryColor': '#ecf0f1', 'clusterBkg': '#fff', 'clusterBorder': '#2C3E50', 'edgeLabelBackground':'#fff'}}}%%
graph TD
    A[Analog Signal<br/>0-3.3V] --> B[8-bit ADC<br/>256 levels]
    A --> C[12-bit ADC<br/>4096 levels]
    A --> D[16-bit ADC<br/>65536 levels]

    B --> E[Resolution<br/>12.9 mV/step]
    C --> F[Resolution<br/>0.8 mV/step]
    D --> G[Resolution<br/>0.05 mV/step]

    E --> H[Fast<br/>Low Power<br/>±0.4% error]
    F --> I[Medium<br/>Medium Power<br/>±0.025% error]
    G --> J[Slow<br/>High Power<br/>±0.0015% error]

    style A fill:#E67E22,stroke:#2C3E50,stroke-width:2px,color:#fff
    style B fill:#2C3E50,stroke:#16A085,stroke-width:2px,color:#fff
    style C fill:#16A085,stroke:#2C3E50,stroke-width:2px,color:#fff
    style D fill:#2C3E50,stroke:#16A085,stroke-width:2px,color:#fff
    style E fill:#7F8C8D,stroke:#2C3E50,stroke-width:2px,color:#fff
    style F fill:#7F8C8D,stroke:#2C3E50,stroke-width:2px,color:#fff
    style G fill:#7F8C8D,stroke:#2C3E50,stroke-width:2px,color:#fff
    style H fill:#E67E22,stroke:#2C3E50,stroke-width:2px,color:#fff
    style I fill:#16A085,stroke:#2C3E50,stroke-width:2px,color:#fff
    style J fill:#2C3E50,stroke:#16A085,stroke-width:2px,color:#fff

Figure 58.5: ADC resolution comparison showing trade-offs between quantization levels, conversion speed, power consumption, and accuracy

This variant uses a ruler measurement analogy to make quantization tangible:

%% fig-alt: "Analogy comparing ADC resolution to ruler precision. Three rulers measure the same 2.56V signal. Coarse ruler (8-bit ADC) has only inch marks - measures 2.5V with ±0.4% error, like estimating 2 and a half inches. Medium ruler (12-bit ADC) has millimeter marks - measures 2.56V with ±0.025% error, precise to 2 inches and 56 hundredths. Fine ruler (16-bit ADC) has micrometer marks - measures 2.5600V with ±0.0015% error. The key insight is that more resolution means smaller measurement steps, but the ruler must still be as accurate as the measurement needs. Using a micrometer to measure lumber is overkill; using inch marks for machining fails."
%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#16A085', 'secondaryColor': '#E67E22', 'tertiaryColor': '#7F8C8D'}}}%%
graph TB
    subgraph SIGNAL["Signal to Measure: 2.56V"]
        SIG["Actual voltage<br/>between 0V and 3.3V"]
    end

    subgraph COARSE["8-bit: Coarse Ruler"]
        R8["Only inch marks<br/>━━━━━━━━━━"]
        R8V["Reads: ~2.5V<br/>Error: ±0.4%"]
        R8U["Use: Rough checks<br/>Is it ON or OFF?"]
    end

    subgraph MEDIUM["12-bit: Millimeter Ruler"]
        R12["Fine tick marks<br/>━|━|━|━|━|━"]
        R12V["Reads: 2.56V<br/>Error: ±0.025%"]
        R12U["Use: Most sensors<br/>Temperature, pressure"]
    end

    subgraph FINE["16-bit: Micrometer"]
        R16["Precision scale<br/>━|||━|||━|||━"]
        R16V["Reads: 2.5600V<br/>Error: ±0.0015%"]
        R16U["Use: Scientific<br/>Audio, medical"]
    end

    SIG --> COARSE
    SIG --> MEDIUM
    SIG --> FINE

    style SIG fill:#E67E22,stroke:#2C3E50,stroke-width:2px,color:#fff
    style R8 fill:#7F8C8D,stroke:#2C3E50,stroke-width:1px,color:#fff
    style R8V fill:#E67E22,stroke:#2C3E50,stroke-width:1px,color:#fff
    style R8U fill:#ECF0F1,stroke:#7F8C8D,stroke-width:1px,color:#000
    style R12 fill:#7F8C8D,stroke:#2C3E50,stroke-width:1px,color:#fff
    style R12V fill:#16A085,stroke:#2C3E50,stroke-width:1px,color:#fff
    style R12U fill:#ECF0F1,stroke:#16A085,stroke-width:1px,color:#000
    style R16 fill:#7F8C8D,stroke:#2C3E50,stroke-width:1px,color:#fff
    style R16V fill:#2C3E50,stroke:#16A085,stroke-width:1px,color:#fff
    style R16U fill:#ECF0F1,stroke:#2C3E50,stroke-width:1px,color:#000

Figure 58.6: ADC resolution as ruler precision: 8-bit is like measuring with inch marks, 12-bit like millimeters, 16-bit like a micrometer

Why this variant helps: The original shows abstract numbers (256 levels, 4096 levels). This ruler analogy connects to physical measurement - everyone has used rulers with different precision. It helps learners understand that ADC resolution is about measurement granularity, not just bigger numbers.

WarningCommon Misconception Alert: “More Samples = Better Data”

Myth: “If I sample at 1000 Hz instead of 100 Hz, my sensor readings will be 10× more accurate.”

Reality: Higher sampling rates DON’T improve accuracy beyond the Nyquist limit. Here’s why:

The Nyquist-Shannon Theorem: You only need to sample at 2× the highest frequency in your signal.

Example: Temperature sensor monitoring room climate - Temperature changes slowly: 0.1°C per minute (0.0017°C/second) - Highest meaningful frequency: ~0.01 Hz (one cycle per 100 seconds) - Nyquist requirement: Sample at 0.02 Hz = once every 50 seconds - Sampling at 1 Hz (every second): Already 50× oversampled! - Sampling at 1000 Hz: Wasteful - collecting 1000× redundant data

What Actually Improves Accuracy: 1. Better ADC resolution: 12-bit (0.025% error) vs 8-bit (0.4% error) = 16× improvement 2. Signal conditioning: Proper filtering removes noise before ADC 3. Calibration: Correct sensor offset and gain errors 4. Sensor quality: Choose a sensor with ±0.1°C accuracy, not ±1°C

Real Cost of Over-Sampling: - Energy: 1000 samples @ 5mA vs 1 sample @ 5mA = 1000× power waste - Bandwidth: Transmitting 1000 values vs 1 value = 1000× data waste - Storage: Storing 86,400,000 samples/day vs 1,728 samples/day = 50,000× waste

58.7 Summary

The first three stages of the sensor-to-network pipeline establish data quality:

  1. Physical Measurement (Stage 1): Sensor converts phenomenon to electrical signal
    • Sensor type and placement determine measurement fidelity
    • Output is analog voltage or current
  2. Signal Conditioning (Stage 2): Prepare signal for digitization
    • Amplify weak signals to match ADC range
    • Filter noise before conversion
    • Adjust offset for unipolar ADC input
  3. ADC Conversion (Stage 3): Transform analog to digital
    • Resolution determines measurement precision
    • Sampling rate must exceed 2× signal bandwidth (Nyquist)
    • Higher resolution costs more power and time

Key Takeaways: - Signal quality established in stages 1-3 cannot be recovered later - Match ADC resolution to actual sensor accuracy requirements - Over-sampling wastes power without improving accuracy - Proper signal conditioning is often more important than ADC resolution

58.8 What’s Next

In the next chapter, Processing and Formatting, we’ll explore stages 4-6: how digital values become calibrated measurements, get encoded into efficient formats, and are assembled into network packets ready for transmission.

Continue to Processing and Formatting →