Analytics & ML · Study deck
Imputation and Noise Filtering
Picture a cold-room chart with six missing minutes just before an alarm.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Implement Missing Data Handling: Apply appropriate imputation strategies (forward-fill, interpolation, seasonal decomposition) for different sensor types
- Compare Imputation Methods: Evaluate trade-offs between forward-fill, interpolation, and seasonal decomposition based on data characteristics
- Design Noise Filters: Implement moving average, median, and exponential smoothing filters and assess their signal conditioning performance
- Distinguish Strategy by Sensor Type: Justify the correct imputation and filtering approach based on sensor semantics and physical behavior
Major section
Start With the Story
The analyst can carry the last value forward, draw a line across the gap, or leave it empty.
- Each choice changes what a later manager may believe about the food and the fault.
- A smooth line is not proof that the missing event never happened.
- This opening does not choose one repair for every signal.
Major section
Start With the Story (continued)
A later repair may support review or study, but it must not silently rewrite the record used for a safety action.
- Practitioner matches the rule to the meaning and the decision.
- Under the Hood examines noise shape, window choice, bias, model checks, and how each change affects later work.
- If any answer is no, do not use the fixed series for a live choice.
Major section
In 60 Seconds
Missing and noisy sensor readings corrupt every downstream analysis, making data imputation (filling gaps) and filtering (smoothing noise) essential preprocessing steps before any IoT analytics pipeline can produce reliable results.
- The choice of imputation strategy — forward-fill, interpolation, or model-based — fundamentally affects anomaly detection sensitivity and ML model accuracy.
- Filling gaps cannot restore information acquisition never captured.
Major section
Why Imputation and Filtering Matter
Missing data and noise are inevitable in real IoT deployments.: Sensors lose power, networks drop packets, and electronic noise corrupts readings.
- Rather than discard incomplete data, we can intelligently fill gaps and smooth noise.
- Key question this chapter answers: "How do I fill gaps in sensor data and smooth out noise without losing important information?".
Major section
Imputation and Filtering Basics
Core Concept: Missing value imputation fills data gaps using neighboring or historical values, while noise filtering smooths random fluctuations to reveal the underlying signal - both must be matched to sensor semantics.
- Gaps cause errors or require discarding entire time windows.
- Proper handling preserves data integrity while enabling downstream processing.
Major section
Missing Value Imputation
The first half of the chapter is about deciding what a gap means.
- A missing temperature reading, a missing motion event, and a missing daily cycle do not deserve the same treatment.
Major section
Key Concepts
Missing data imputation: The process of estimating and filling in missing sensor readings using statistical or model-based methods rather than discarding incomplete records.
- Missingness mechanism: The reason data is missing — Missing Completely at Random (MCAR), Missing at Random (MAR), or Missing Not at Random (MNAR) — which determines which imputation methods are statistically valid.
Major section
Median Filter
The median filter is the next tool because it can ignore a spike when most nearby readings are sane.
- Moving averages reduce random variation, but a single bad spike still enters the average.
Major section
Putting Numbers to It
Where $N$ is the window size.
- Trade-off:: Larger window → better noise rejection BUT longer delay detecting real changes.
- If monitoring hourly HVAC cycles (3600s), 5-10 sample window (2-4.5s latency) is acceptable.
- If detecting rapid door opening events (10s timescale), use $N=3$ (1.0s latency max).
Major section
Temperature Filter Windows
Scenario: You have a temperature sensor monitoring a cold storage facility.
- The sensor reports every 10 seconds.
- You've observed occasional spikes due to electrical interference when the cooling compressor starts (5-10C jumps), and you need to filter these without losing legitimate temperature trends.
- For cold storage monitoring (not time-critical), 20 seconds is acceptable.
Major section
Choose Imputation Strategy
When sensor data goes missing, selecting the correct imputation strategy depends on sensor characteristics and downstream requirements.
- YES → Forward-fill acceptable for gaps < 10x sampling interval.
- YES → Use seasonal decomposition fill.
- YES → Use spatial interpolation from neighbors.
- NO → Use linear interpolation or mark as missing.
Major section
Detect Outliers Before Imputing
The Mistake: Running imputation before outlier detection, causing outliers to be forward-filled or interpolated into the data stream, permanently corrupting adjacent readings.
- Engineers add imputation first (to handle missing data), then later realize they need outlier detection.
- By then, the pipeline order is established and changing it requires refactoring.
Deck summary
Key takeaways
The analyst can carry the last value forward, draw a line across the gap, or leave it empty.
- A later repair may support review or study, but it must not silently rewrite the record used for a safety action.
- Missing and noisy sensor readings corrupt every downstream analysis, making data imputation (filling gaps) and filtering (smoothing noise) essential preprocessing steps before any IoT analytics pipeline can produce reliable results.
- Missing data and noise are inevitable in real IoT deployments.: Sensors lose power, networks drop packets, and electronic noise corrupts readings.
- The first half of the chapter is about deciding what a gap means.
Retrieval practice
Recall check 1 of 5
Data Dora says: answer from memory, then check your reasoning.
Q1A PIR motion sensor loses 30 seconds of data during a brief network outage. What is the most appropriate imputation choice?
Show answer
Answer: A Imputation must respect signal semantics; a missing PIR event maps to no motion (zero), not interpolation.
Retrieval practice
Recall check 2 of 5
Data Dora says: answer from memory, then check your reasoning.
Q2A motion sensor (PIR) has 30 seconds of missing data due to a network outage. What is the most appropriate imputation strategy?
Show answer
Answer: A For event-driven sensors like PIR motion detectors, zero (no motion) is the correct imputation.
Retrieval practice
Recall check 3 of 5
Data Dora says: answer from memory, then check your reasoning.
Q3A humidity sensor shows 10 consecutive readings of exactly 78.00%. What quality concern does this indicate?
Show answer
Answer: D Real analog sensors always have some noise - even in stable environments, readings fluctuate by at least the ADC resolution.
Retrieval practice
Recall check 4 of 5
Data Dora says: answer from memory, then check your reasoning.
Q4You have a pressure sensor with occasional spike readings of 0 or 4095 (ADC limits) due to electrical interference. Which filter is best?
Show answer
Answer: B The median filter is ideal for spike removal because it completely ignores outliers.
Q5Place each data-repair responsibility where it lives so you can repair missing or noisy readings without hiding which values were observed and which were inferred.
Show answer
Answer: A repair missing or noisy readings without hiding which values were observed and which were inferred.
Retrieval practice
Recall check 5 of 5
Data Dora says: answer from memory, then check your reasoning.
Q6Complete the sensor data quality pipeline:
Show answer
Answer: A Sensor data cleaning validates physical limits while preserving missing markers, then interpolates the gaps and smooths with an exponential moving average.
Print reference
Answers 1 of 2
Answer key.
- A · Imputation must respect signal semantics; a missing PIR event maps to no motion (zero), not interpolation.
- A · For event-driven sensors like PIR motion detectors, zero (no motion) is the correct imputation.
- D · Real analog sensors always have some noise - even in stable environments, readings fluctuate by at least the ADC resolution.
- B · The median filter is ideal for spike removal because it completely ignores outliers.
Print reference
Answers 2 of 2
Answer key.
- A · repair missing or noisy readings without hiding which values were observed and which were inferred.
- A · Sensor data cleaning validates physical limits while preserving missing markers, then interpolates the gaps and smooths with an exponential moving average.