25 Imputation and Noise Filtering

25.1 Start With the Story

Repair a Gap Without Hiding It

Picture a cold-room chart with six missing minutes just before an alarm. The analyst can carry the last value forward, draw a line across the gap, or leave it empty. Each choice changes what a later manager may believe about the food and the fault.

Start with what the reading means. Record whether it is a level, count, switch event, or slow trend. Keep the raw row, gap start, gap end, repair rule, reason, and version beside the repaired result. Mark made values so they can never look like direct observations.

Test the rule on known gaps of different length and during fast change. Compare alerts made from raw and repaired data. Check delay, false alarms, missed alarms, and the result after a restart. A smooth line is not proof that the missing event never happened.

Keep any urgent local alarm based on evidence the device can still see. A later repair may support review or study, but it must not silently rewrite the record used for a safety action.

This opening does not choose one repair for every signal. Practitioner matches the rule to the meaning and the decision. Under the Hood examines noise shape, window choice, bias, model checks, and how each change affects later work.

Use a short release check. Is the raw gap still visible? Is each made value marked? Does the rule fit this kind of reading? Did the alarm change? Can another person run the same repair? If any answer is no, do not use the fixed series for a live choice.

Picture an IoT team using the ideas in Imputation and Noise Filtering during a live operations review. A device has produced messy evidence, an analytic step is about to change an alert or control decision, and someone has to explain why the result should be trusted.

Read this page as that path from sensor evidence to accountable action. Start with what the system observes, keep the model or data treatment visible, and finish with the check that would convince an operator, maintainer, or auditor to act.

25.2 Learning Objectives

By the end of this chapter, you will be able to:

• Implement Missing Data Handling: Apply appropriate imputation strategies (forward-fill, interpolation, seasonal decomposition) for different sensor types
• Compare Imputation Methods: Evaluate trade-offs between forward-fill, interpolation, and seasonal decomposition based on data characteristics
• Design Noise Filters: Implement moving average, median, and exponential smoothing filters and assess their signal conditioning performance
• Distinguish Strategy by Sensor Type: Justify the correct imputation and filtering approach based on sensor semantics and physical behavior

25.3 Imputation Filtering Check

The mathematical gist. Sampling every 60 seconds means $f_s=1/60$ Hz and a Nyquist ceiling of $1/120$ Hz, so the fastest unaliased cycle lasts 120 seconds. A 90-second HVAC cycle would fold into a false 180-second trend before any imputation runs. Separately, a 12-bit ADC over 0–100% humidity has a 0.0244% step, 0.00705% RMS quantisation noise, and a 74.0 dB ideal SNR ceiling. Filling gaps cannot restore information acquisition never captured.

Math Bridge · guided foundationsWhat can no imputation method recover?Let Data Dora connect sample interval, aliasing, ADC steps, and the limits of gap filling. Learn the maths

25.4 Prerequisites

Before diving into this chapter, you should be familiar with:

• Data Validation and Outlier Detection: Understanding validation as the first stage of the data quality pipeline
• Signal Processing Essentials: Basic concepts of filtering and signal conditioning
• Sensor Fundamentals: Knowledge of different sensor types and their output characteristics

25.5 Missing Value Imputation

The first half of the chapter is about deciding what a gap means. A missing temperature reading, a missing motion event, and a missing daily cycle do not deserve the same treatment.

25.6 Forward Fill for Sensor Data

Best for slowly-changing values like temperature:

class ForwardFillImputer:
    def __init__(self, max_gap=60):  # Maximum gap in samples
        self.last_valid = None
        self.gap_count = 0
        self.max_gap = max_gap

    def impute(self, value, is_valid):
        if is_valid:
            self.last_valid = value
            self.gap_count = 0
            return value, "original"

        if self.last_valid is None:
            return None, "no_history"

        self.gap_count += 1
        if self.gap_count > self.max_gap:
            return None, "gap_too_large"

        return self.last_valid, "imputed_ffill"

25.7 Linear Interpolation

Forward-fill answers “what if the last value still held?” Interpolation asks a different question: if the signal changed smoothly between two known readings, what values plausibly sat inside the gap?

Better for trending values when future data is available:

def linear_interpolate(data, timestamps):
    """
    Interpolate missing values (None/NaN) using linear interpolation.
    Requires knowledge of surrounding valid points.
    """
    import numpy as np

    data = np.array(data, dtype=float)
    timestamps = np.array(timestamps, dtype=float)

    valid_mask = ~np.isnan(data)
    valid_indices = np.where(valid_mask)[0]

    if len(valid_indices) < 2:
        return data

    # Interpolate
    interpolated = np.interp(
        timestamps,
        timestamps[valid_mask],
        data[valid_mask]
    )

    return interpolated

25.8 Seasonal Decomposition Fill

Some data repeats. When temperature follows a daily cycle, a method that remembers the phase of that cycle can be more honest than a straight line across the gap.

For data with known patterns (e.g., temperature with daily cycles):

def seasonal_fill(data, period=24):
    """
    Fill missing values using seasonal pattern from historical data.
    period: number of samples in one cycle (e.g., 24 for hourly data with daily cycle)
    """
    import numpy as np

    data = np.array(data, dtype=float)
    n = len(data)

    # Calculate seasonal pattern from valid data
    seasonal = np.zeros(period)
    counts = np.zeros(period)

    for i, val in enumerate(data):
        if not np.isnan(val):
            seasonal[i % period] += val
            counts[i % period] += 1

    # Average seasonal values
    with np.errstate(divide='ignore', invalid='ignore'):
        seasonal = np.where(counts > 0, seasonal / counts, np.nan)

    # Fill missing with seasonal pattern
    filled = data.copy()
    for i in range(n):
        if np.isnan(filled[i]) and not np.isnan(seasonal[i % period]):
            filled[i] = seasonal[i % period]

    return filled

25.9 Try It: Forward-Fill Gap Calculator

Read the Try It: Forward-Fill Gap Calculator material as a decision path rather than as isolated entries. First identify the operating condition in each entry and keep its units, timing, source, and assumed system state attached to it. Next compare the entries at the point where responsibility changes between device, gateway, network, analytic service, and operator; that hand-off is where apparently similar choices often produce different outcomes. Then follow the failure case: ask what becomes stale, delayed, unavailable, or unsafe, who detects it, and what evidence permits recovery. Finally connect the result to the chapter’s running design record by naming the selected behavior, the rejected alternative, the measurement that justifies the choice, and the condition that forces a recheck. That order turns the examples or comparison into an auditable engineering argument.

25.10 Noise Filtering Techniques

Once the gaps are accounted for, the next problem is different: readings are present, but the series is jagged, spiky, or too noisy for a reliable decision.

25.11 Moving Average Filter

Simple and effective for steady-state noise reduction:

class MovingAverageFilter:
    def __init__(self, window_size=5):
        self.window_size = window_size
        self.buffer = []

    def filter(self, value):
        self.buffer.append(value)
        if len(self.buffer) > self.window_size:
            self.buffer.pop(0)

        return sum(self.buffer) / len(self.buffer)

25.12 Median Filter

Moving averages reduce random variation, but a single bad spike still enters the average. The median filter is the next tool because it can ignore a spike when most nearby readings are sane.

Excellent for removing spike noise while preserving edges:

class MedianFilter:
    def __init__(self, window_size=5):
        self.window_size = window_size
        self.buffer = []

    def filter(self, value):
        self.buffer.append(value)
        if len(self.buffer) > self.window_size:
            self.buffer.pop(0)

        sorted_buffer = sorted(self.buffer)
        mid = len(sorted_buffer) // 2

        if len(sorted_buffer) % 2 == 0:
            return (sorted_buffer[mid - 1] + sorted_buffer[mid]) / 2
        return sorted_buffer[mid]

25.13 Exponential Smoothing

After windowed filters, exponential smoothing gives you a streaming option: each new reading nudges the estimate without waiting for a full centered window.

Provides weighted average with more weight on recent values:

class ExponentialSmoothingFilter:
    def __init__(self, alpha=0.3):
        """
        alpha: smoothing factor (0-1)
        Higher alpha = more weight on recent values = less smoothing
        Lower alpha = more weight on history = more smoothing
        """
        self.alpha = alpha
        self.smoothed = None

    def filter(self, value):
        if self.smoothed is None:
            self.smoothed = value
        else:
            self.smoothed = self.alpha * value + (1 - self.alpha) * self.smoothed

        return self.smoothed

25.14 Filter Comparison

Checkpoint: Filter Trade-Offs

You now know:

• Moving average noise falls as sigma divided by square root of N, but latency grows as (N-1)/2 samples.
• At N=10 and 1 Hz, that latency is 4.5 seconds; a 100 point moving average at 1 Hz later becomes a 50 second lag.
• Median filters handle sparse spikes; exponential smoothing is useful when a real-time stream cannot wait for a centered window.

25.19 Try It: Filter Window Size Calculator

25.20 Try It: Exponential Smoothing Explorer

25.21 Choosing the Right Filter

The examples above show the mechanics. This section turns them into an engineering choice: identify the noise, decide whether edges matter, then pick the simplest filter that preserves the signal you care about.

def choose_filter(signal_characteristics):
    """
    Guide for selecting appropriate noise filter based on signal characteristics.
    """
    recommendations = {
        'steady_state_with_gaussian_noise': {
            'filter': 'MovingAverage',
            'reason': 'Averages out random noise effectively',
            'window_size': 5  # Adjust based on noise frequency
        },
        'spiky_noise_impulse': {
            'filter': 'MedianFilter',
            'reason': 'Completely ignores outlier spikes',
            'window_size': 5  # Odd number works best
        },
        'real_time_tracking': {
            'filter': 'ExponentialSmoothing',
            'reason': 'No latency, responsive to changes',
            'alpha': 0.3  # Lower = smoother, higher = more responsive
        },
        'sensor_fusion_known_dynamics': {
            'filter': 'KalmanFilter',
            'reason': 'Optimal estimation with uncertainty tracking',
            'params': 'process_noise, measurement_noise'
        },
        'edge_preserving': {
            'filter': 'MedianFilter',
            'reason': 'Preserves sharp transitions in data',
            'window_size': 3
        }
    }
    return recommendations.get(signal_characteristics, recommendations['steady_state_with_gaussian_noise'])

25.22 Combining Filters

For robust noise removal, filters can be cascaded:

class CombinedFilter:
    """
    Two-stage filter: Median first (remove spikes), then exponential smooth.
    """
    def __init__(self, median_window=5, exp_alpha=0.3):
        self.median_filter = MedianFilter(median_window)
        self.exp_filter = ExponentialSmoothingFilter(exp_alpha)

    def filter(self, value):
        # Stage 1: Remove spikes with median
        despike = self.median_filter.filter(value)
        # Stage 2: Smooth remaining noise
        smooth = self.exp_filter.filter(despike)
        return smooth

# Usage
combined = CombinedFilter(median_window=5, exp_alpha=0.2)
for reading in sensor_stream:
    clean_value = combined.filter(reading)

median_filter = MedianFilter(window_size=5)
for reading in sensor_stream:
    clean_temp = median_filter.filter(reading)
    if clean_temp < -20:  # After filtering, threshold check is reliable
        trigger_high_temp_alarm()

def select_imputation_strategy(sensor_type, gap_duration_minutes):
    if sensor_type == "motion_detector":
        return "zero_fill"  # No motion during gap
    elif sensor_type == "temperature":
        if gap_duration_minutes < 30:
            return "forward_fill"
        elif gap_duration_minutes < 360:
            return "seasonal_fill"  # Use daily pattern
        else:
            return "mark_missing"  # Gap too long
    elif sensor_type == "energy_meter":
        return "zero_increment"  # No consumption
    elif sensor_type == "vibration":
        return "mark_missing"  # Cannot safely interpolate

# WRONG: Impute first, detect outliers second
readings = [22.1, 22.3, 99.9, None, None, None, 22.8]  # 99.9 is sensor fault

# Step 1: Forward-fill (MISTAKE - happens before outlier removal)
imputed = forward_fill(readings)
# Result: [22.1, 22.3, 99.9, 99.9, 99.9, 99.9, 22.8]

# Step 2: Outlier detection
cleaned = remove_outliers(imputed, threshold=3_sigma)
# Result: [22.1, 22.3, REMOVED, REMOVED, REMOVED, REMOVED, 22.8]
# Lost 4 data points! The None values became 99.9 outliers.

# CORRECT: Validate first, impute second
readings = [22.1, 22.3, 99.9, None, None, None, 22.8]

# Step 1: Outlier detection and removal (mark as None)
validated = remove_outliers(readings, threshold=3_sigma)
# Result: [22.1, 22.3, None, None, None, None, 22.8]

# Step 2: Forward-fill ONLY the validated stream
imputed = forward_fill(validated)
# Result: [22.1, 22.3, 22.3, 22.3, 22.3, 22.3, 22.8]
# Correctly preserved legitimate data!

Checkpoint: Pipeline Order

You now know:

• Validation must run before imputation, or a bad value can be copied into the gap.
• The warehouse example shows the cost: one 85C spike was forward-filled for 30 minutes and forced a $50,000 destruction decision.
• A defensible pipeline is range validation, rate-of-change validation, outlier detection, imputation, then filtering.

25.23 Knowledge Check

25.24 Quiz: Missing Data and Filtering

25.25 Interactive Quiz: Match Concepts

25.26 Interactive Quiz: Sequence the Steps

25.27 Label the Diagram

25.28 Code Challenge

25.29 Missing-Data Repair Contracts

The chapter above covers forward-fill, interpolation, seasonal fill, moving average filters, median filters, exponential smoothing, calculators, and practice quizzes. Continue to Missing-Data Repair and Filtering Contracts for the deeper L2 material: missingness mechanisms, repair quality flags, impulse-noise filters, held-out replay validation, and MCAR/MAR/MNAR bias boundaries.

25.30 Summary

Missing value imputation and noise filtering are essential for producing clean, complete sensor data:

• Forward Fill: Simple and effective for slowly-changing continuous values (temperature, humidity)
• Linear Interpolation: Better for trending data when you have values on both sides of the gap
• Seasonal Fill: Use when data has known periodic patterns (daily temperature cycles)
• Sensor-Specific Imputation: Motion sensors get zero, state sensors get “unknown”, counters get zero increment
• Moving Average: Good for steady-state Gaussian noise, but blurs edges
• Median Filter: Excellent for spike removal, preserves sharp transitions
• Exponential Smoothing: Real-time with no latency, tunable responsiveness

Critical Design Principle: Always match your imputation and filtering strategy to the sensor type and data characteristics. A one-size-fits-all approach will produce incorrect results for at least some of your sensors.

25.31 See Also

Data Quality Pipeline:

• Data Quality Overview - Full pipeline architecture
• Data Validation - Validation before imputation
• Anomaly Detection - Finding patterns in clean data

Filtering Techniques:

• Kalman Filters - Optimal filtering with fusion
• Complementary Filters - IMU signal conditioning
• Digital Signal Processing - Filter design

Applications:

• Edge Compute Patterns - Real-time filtering at edge
• Time Series Fundamentals - Handling temporal patterns
• Sensor Calibration - Bias correction

25.32 What’s Next