18 Sensor Processing: Filter Selection and Validation
18.1 Start With the Decision
A vibration filter can remove noise and erase the fault at the same time. Its cutoff must follow the machine signal and sample rate.
18.2 Route Overview
This is part 2 of 2. Review Sensor Processing: Sampling and Signals for the preceding evidence.
18.3 Learning Objectives
- Select a vibration filter from spectral evidence.
- Validate filter output against timing and amplitude requirements.
18.4 Chapter Roadmap
- Vibration Filter Selection
- Checkpoint: Filter Choice
- Sensor Calibration
- Soil Moisture Calibration
- Checkpoint: Calibration
- Common Processing Pitfalls
- Knowledge Check
- Match: Data Processing Concepts
- Sensor Processing Pipeline
- Label the Diagram
- Code Challenge
- Sampling and Aliasing Limits
- Checkpoint: Pipeline Readiness
- Next Acquisition Practice
- Summary
- Common Pitfalls
- What’s Next
18.5 Vibration Filter Selection
Scenario: A food processing plant needs to monitor vibration on 12 conveyor belt motors. Excessive vibration indicates bearing wear requiring maintenance within 2-4 weeks.
Given:
Start by sensor: ADXL345 accelerometer on each motor (100 Hz sampling). Then normal vibration: 0.5-2.0 g RMS. Next bearing wear threshold: >3.5 g RMS sustained for 10+ minutes. After that noise sources: (1) electrical interference from motor drives (+/- 0.3 g spikes), (2) adjacent machine vibration (+/- 0.15 g slow drift). Continue by mCU: ESP32 (240 MHz, FPU, 320 kB RAM). Finally reporting interval: every 60 seconds to cloud via MQTT.
Filter evaluation:
| Filter | Spike Removal | Drift Rejection | RAM (per motor) | CPU/sample | Verdict |
|---|---|---|---|---|---|
| Moving Average (N=20) | Poor — spike averages into output | Good at N=20 (0.2s window) | 80 bytes | 0.2 us | Passes spikes |
| Median Filter (N=7) | Excellent — spike completely rejected | Moderate | 28 bytes | 1.5 us | Good for spikes |
| Kalman (Q=0.01, R=0.1) | Good — spike dampened in 2-3 samples | Excellent | 20 bytes | 0.8 us | Best overall tracking |
| Median + Moving Avg | Excellent | Excellent | 108 bytes | 1.7 us | Best accuracy |
Selected approach: Two-stage filter (median then moving average).
Start with Stage 1 — Median filter (N=5): Removes electrical interference spikes. At 100 Hz, a 5-sample window covers 50 ms — fast enough to preserve real vibration changes. Finally Stage 2 — Moving average (N=50): Smooths the median output over 0.5 seconds. Reports stable RMS value for threshold comparison.
Resource budget for 12 motors:
Start by rAM: 12 motors x (20 + 200) bytes = 2,640 bytes (0.8% of ESP32 RAM). Then cPU: 12 motors x 100 samples/s x 1.7 us = 2,040 us/s = 0.2% CPU utilization (conservative estimate). Finally headroom: 99.2% RAM and 99.8% CPU available for MQTT, Wi-Fi stack, and other tasks.
Result: The two-stage filter detects bearing wear threshold crossings within approximately 0.5 seconds (the N=50 averaging window fill time at 100 Hz) while completely rejecting electrical interference spikes. The 10-minute sustained-threshold requirement is evaluated by comparing consecutive 0.5-second RMS averages over a sliding evaluation period. False alarm rate: 0.1% (vs. 12% with moving average alone). Missed detection rate: 0% for sustained threshold exceedances over 5 minutes.
Key Insight: For vibration monitoring, always use a median filter as the first stage to eliminate electrical spikes. A moving average alone will spread spike energy across the window, potentially triggering false bearing-wear alerts. The two-stage approach costs negligible additional resources on modern MCUs.
Checkpoint: Filter Choice
You now know:
- Moving average is simple but adds N/2 samples of delay; with N=10 at 100 Hz, that becomes 50 ms.
- EMA keeps only 1 float, or 4 bytes, while a 10-sample moving average stores 40 bytes.
- Median filtering is the first stage for spikes, while Kalman filtering belongs where Q, R, and changing state are meaningful.
18.6 Sensor Calibration
Filtering removes random noise, but it cannot fix systematic errors built into the sensor itself. A sensor that consistently reads 2 degrees too high will still read 2 degrees too high after filtering — just with less jitter. Calibration corrects these systematic errors. Two-point calibration addresses both offset (zero-point shift) and gain (sensitivity) errors.
You have cleaned random variation. The next failure mode is bias: a value can be stable and still be wrong.
18.6.1 Calibration Error Types
Understanding the two main calibration errors helps you design effective correction strategies:
Before selecting a correction, inspect Figure 18.1 to distinguish stable bias from sensitivity error and random scatter. Each error leaves a different pattern, so one calibration or filter cannot repair them all.
Read Figure 18.1, compare the constant displacement of offset error with the changing slope of gain error, then contrast both with sample-to-sample noise. The first two call for calibration coefficients; the last calls for noise reduction, connecting diagnosis to the processing choice that follows.
18.6.2 Two-Point Calibration
Two-point calibration creates a linear correction by measuring at two known reference points:
To see how both systematic terms can be estimated, inspect Figure 18.2 before using the formula. The process needs two separated known values because one point cannot distinguish slope error from offset.
Read Figure 18.2, record the low reference pair first, then the high pair, and follow both into gain and offset calculation before checking an independent value. This sequence connects calibration arithmetic to evidence that the corrected line works between its endpoints.
The calculation is the same idea students explored in the interactive calculator:
18.6.3 Two-Point Calibration Math
slope = (actual_high - actual_low) / (raw_high - raw_low)calibrated = actual_low + slope x (raw_value - raw_low)
18.6.4 Optional C++ Pattern
// Two-point calibration for linear sensors
struct CalibrationData {
float rawLow;
float rawHigh;
float actualLow;
float actualHigh;
};
CalibrationData cal = {
.rawLow = 512, // ADC reading at low point
.rawHigh = 3584, // ADC reading at high point
.actualLow = 0.0, // Actual value at low point
.actualHigh = 100.0 // Actual value at high point
};
float calibrate(float rawValue) {
// Linear interpolation with division guard
if (cal.rawHigh == cal.rawLow) {
return cal.actualLow; // Cannot calibrate with identical reference points
}
float slope = (cal.actualHigh - cal.actualLow) / (cal.rawHigh - cal.rawLow);
float calibratedValue = cal.actualLow + slope * (rawValue - cal.rawLow);
return calibratedValue;
}
// Store calibration in EEPROM
#include <EEPROM.h>
void saveCalibration() {
EEPROM.begin(512);
EEPROM.put(0, cal);
EEPROM.commit();
Serial.println("Calibration saved");
}
void loadCalibration() {
EEPROM.begin(512);
EEPROM.get(0, cal);
Serial.println("Calibration loaded");
}Try It: Two-Point Calibration Calculator
18.7 Soil Moisture Calibration
Scenario: You are deploying a soil moisture monitoring system for a greenhouse. The capacitive soil moisture sensor outputs an analog voltage (0-3.3V) that varies with soil moisture content. However, the raw ADC readings do not correspond to meaningful moisture percentages. The sensor reads approximately 3000 (ADC units) in completely dry soil and 1200 in saturated soil. You need accurate readings to trigger irrigation at 30% moisture.
Goal: Develop and implement a two-point calibration procedure to convert raw ADC readings into calibrated moisture percentages (0-100%).
18.7.1 Step 1: Characterize the Sensor
What we do: Measure the sensor’s output range and behavior.
Initial measurements:
| Condition | ADC Reading (12-bit, 0-4095) | Expected Moisture |
|---|---|---|
| Air (no soil) | 3450 | ~0% (baseline) |
| Bone-dry soil (oven-dried) | 3000 | 0% |
| Field capacity (well-watered) | 1800 | ~60-70% |
| Saturated soil (standing water) | 1200 | 100% |
Observations:
Start by output is inversely proportional to moisture (higher moisture = lower ADC value). Then range spans approximately 1200-3000 ADC units for the usable moisture range. Finally response is approximately linear in the 20-80% moisture range.
18.7.2 Step 2: Create Reference Points
What we do: Establish known moisture levels using gravimetric method.
Gravimetric calibration procedure:
Start with Prepare soil samples: Collect 5 containers of identical soil (200g each). Finally Create moisture levels.
Start by sample A: Oven-dry at 105C for 24 hours (0% moisture). Then sample B: Add 10g water (5% moisture by weight). Next sample C: Add 30g water (15% moisture). After that sample D: Add 60g water (30% moisture - irrigation trigger). Finally sample E: Saturate and drain (field capacity, ~60%).
Start with Record calibration data.
| Sample | Added Water (g) | Calculated Moisture (%) | ADC Reading |
|---|---|---|---|
| A | 0 | 0% | 2988 |
| B | 10 | 5% | 2865 |
| C | 30 | 15% | 2619 |
| D | 60 | 30% | 2251 |
| E | ~120 (saturated) | 60% | 1515 |
18.7.3 Step 3: Calculate Calibration Equation
What we do: Fit a linear equation to the calibration data.
Two-point calibration (using dry and field capacity points):
- Point 1 (Low): ADC = 2988, Moisture = 0%
- Point 2 (High): ADC = 1515, Moisture = 60%
Calculate slope (m) and offset (b):
Calibration equation:
18.7.4 Step 4: Apply the Equation in Firmware
What we do: Apply the Step 3 equation using the same endpoints (dry at ADC=2988 for 0%, field capacity at ADC=1515 for 60%). The firmware action is just: read ADC, calculate moisture, clamp the answer to a valid range.
18.7.4.1 Firmware Behavior
Start by read the filtered ADC value. Then map the dry and field-capacity endpoints onto the 0-60% range. Finally clamp the result to the valid 0-100% moisture range.
18.7.4.2 Optional C++ Pattern
#include <EEPROM.h>
#define SOIL_PIN 34
#define NUM_SAMPLES 10
struct Calibration {
uint32_t magic;
float dryADC;
float wetADC;
float dryMoisture;
float wetMoisture;
};
// Two-point calibration: dry (0%) to field capacity (60%)
// Matches the gravimetric reference points from Step 2
Calibration cal = {
.magic = 0xCAFEBABE,
.dryADC = 2988.0,
.wetADC = 1515.0,
.dryMoisture = 0.0,
.wetMoisture = 60.0
};
float getMoisturePercent() {
// Read with median filtering
float adcValue = readADCFiltered();
// Linear interpolation with bounds checking
float moisture = cal.dryMoisture +
(cal.wetMoisture - cal.dryMoisture) *
(cal.dryADC - adcValue) / (cal.dryADC - cal.wetADC);
// Clamp to valid range
if (moisture < 0.0) moisture = 0.0;
if (moisture > 100.0) moisture = 100.0;
return moisture;
}Verification against calibration data:
| Sample | ADC | Measured (%) | Model Prediction (%) | Error |
|---|---|---|---|---|
| A (dry) | 2988 | 0% | 0.0% | 0.0% |
| B | 2865 | 5% | 5.0% | 0.0% |
| C | 2619 | 15% | 15.0% | 0.0% |
| D | 2251 | 30% | 30.0% | 0.0% |
| E (field cap.) | 1515 | 60% | 60.0% | 0.0% |
The data points fall closely on the linear model, confirming this sensor has good linearity in the 0-60% range. For readings beyond 60% (saturated soil), the linear model extrapolates but accuracy degrades — use multi-point calibration if the full 0-100% range is needed.
18.7.5 Final Result
Outcome: Successfully calibrated soil moisture sensor with two-point linear calibration covering 0-60% moisture.
Interpret that outcome as a bounded release claim. The dry-to-trigger region is the important operating span, so the ±1.5% and ±2.0% errors support a 30% irrigation threshold with more margin than the wetter end. From 40–60%, the ±2.5% result is still accepted for this use, but it should not be silently generalised beyond the highest calibration reference. The final row makes that limit explicit: values above 60% are extrapolated and exceed 5% typical error, so saturated-soil decisions require additional reference points. Preserve the fitted gain and offset with the raw reference pairs, repeated-measurement spread, validation date, and installed probe condition. The maintenance schedule then reopens this result after soil chemistry, probe placement, electronics, or seasonal conditions change, carrying the worked calibration into an operational measurement record.
Accuracy achieved (with repeated measurements at each point):
| Moisture Range | Typical Error | Acceptable? |
|---|---|---|
| 0-20% (dry) | +/- 1.5% | Yes |
| 20-40% (trigger zone) | +/- 2.0% | Yes — sufficient for 30% irrigation trigger |
| 40-60% (moist) | +/- 2.5% | Yes |
| >60% (saturated) | >5% (extrapolated) | Use multi-point calibration |
Maintenance schedule:
Start by recalibrate every 6 months or after sensor replacement. Finally verify with known moisture sample monthly during growing season.
Checkpoint: Calibration
You now know:
- Two-point calibration uses a low and high reference so the firmware can correct both offset and gain.
- The interactive example maps raw ADC endpoints 512 and 3584 onto actual values 0 and 100.
- The soil example maps dry ADC 2988 and field-capacity ADC 1515 onto 0-60% moisture, then checks the 30% irrigation trigger.
18.8 Common Processing Pitfalls
Before testing your knowledge, review these common mistakes that trip up even experienced engineers.
18.8.1 Moving Average on Changing Signals
The Mistake: Using a large moving average window (N=32 or N=64 samples) to filter sensor data that changes rapidly, introducing unacceptable lag that makes control systems sluggish or miss transient events entirely.
Why It Happens: Moving average is simple to implement and tutorials recommend larger windows for “smoother” data. For slowly-changing signals (room temperature sampled at 1Hz), a 10-second window works well. But applying the same approach to fast signals (accelerometer at 100Hz, current sensing for motor control) adds N/2 samples of delay - a 32-sample filter at 100Hz introduces 160 ms lag, making closed-loop control unstable.
The Fix: Match filter characteristics to signal dynamics:
- Slow signals (temperature, humidity): Moving average N=8-32 at 1Hz sampling, 4-16 second settling time
- Medium signals (distance, pressure): Exponential moving average (EMA) with alpha=0.1-0.3, responds faster while filtering noise
- Fast signals (motor current, vibration): Use IIR filters (Butterworth, Chebyshev) designed for specific cutoff frequency
EMA Formula: filtered = alpha x new_value + (1-alpha) x previous_filtered
18.8.2 Single-Point Nonlinear Calibration
The Mistake: Calibrating a thermistor, pH sensor, or photodiode at only one reference point (e.g., room temperature, pH 7, or ambient light), then assuming the calibration applies across the entire measurement range.
Why It Happens: Single-point calibration is quick - adjust offset so the reading matches one known value and ship. This works for sensors with linear response and negligible gain error. But many sensors are inherently non-linear: thermistors follow the Steinhart-Hart equation (exponential), pH electrodes have temperature-dependent Nernst slope, photodiodes have logarithmic response at high intensity.
The Fix: Use two-point calibration minimum for linear sensors, three or more points for non-linear sensors:
- Linear sensors (RTD, 4-20mA transmitters): Calibrate at 10% and 90% of range
- Thermistor (NTC): Use Steinhart-Hart equation with three calibration points (0C, 25C, 100C)
- pH sensor: Calibrate at pH 4.0, 7.0, and 10.0 buffers
Warning Signs: You need multi-point calibration if: (1) sensor datasheet shows non-linear response curve, (2) accuracy degrades significantly away from single calibration point, (3) sensor type is known to be non-linear.
18.9 Knowledge Check
Test your understanding of sensor data processing concepts with these questions.
The questions below are the checkpoint loop for the whole chapter: filter type, calibration math, latency, drift, and nonlinear sensors.
18.10 Match: Data Processing Concepts
18.11 Sensor Processing Pipeline
18.12 Label the Diagram
18.13 Code Challenge
18.14 Sampling and Aliasing Limits
The main chapter above shows how to smooth, calibrate, validate, and publish sensor readings after acquisition. The companion page checks the acquisition boundary itself: sample rate, Nyquist limits, alias frequencies, analog anti-alias filtering, and oversampling trade-offs decide whether the data entering those filters is trustworthy.
Checkpoint: Pipeline Readiness
You now know:
- A complete processing pipeline moves through acquisition, cleaning, transformation, analysis, validation, and publishing.
- The label quiz’s 4 hotspots make the same sequence visible: acquisition, cleaning, transformation, and analysis.
- Filtering and calibration are downstream decisions; the companion page checks whether sampling captured the physical signal first.
18.15 Next Acquisition Practice
Continue with Sampling, Aliasing, and Anti-Alias Boundaries to verify that a sensor pipeline captures the physical signal before firmware filtering or calibration begins.
18.16 Summary
This chapter covered essential sensor data processing techniques:
Start with Moving Average Filter: Simple noise reduction by averaging N samples, best for slow-changing signals. Then Kalman Filter: Adaptive filtering (optimal for linear systems with Gaussian noise) that balances predictions with measurements. Next Median Filter: Spike/outlier removal by selecting the middle value. After that Two-Point Calibration: Corrects offset and gain errors using two reference points. Continue by Multi-Point Calibration: Handles non-linear sensors with piecewise interpolation. Finally EEPROM Storage: Persists calibration across power cycles.
Common Pitfalls
18.16.1 Filtering Out Real Signal Variations
A moving average window that is too large smooths out genuine rapid changes in the measured quantity. A temperature spike from a briefly opened oven door may be a real event, not noise. Size the filter window to be shorter than the fastest legitimate change you need to detect.
18.16.2 Reference Accuracy in Calibration
The calibrated sensor can never be more accurate than the reference standard you calibrated against. Using a budget thermometer as a reference for a precision sensor is self-defeating. Use references at least 4x more accurate than your target accuracy.
18.16.3 Not Validating After Calibration
After applying calibration coefficients, test the sensor at several intermediate values — not just the two reference points. Nonlinearity errors will not be visible at the calibration endpoints but will appear at intermediate values.
18.16.4 EEPROM Calibration Loss
Flashing new firmware can erase EEPROM calibration data depending on memory layout. Always check coefficients on startup and alert the user if values read back as 0xFF (erased flash) or are physically implausible.
18.17 What’s Next
18.18 Continue Your Route
This final part closes the route from Vibration Filter Selection through What’s Next. Return to Sensor Processing: Sampling and Signals or continue from the sensors module index.