1356  Statistical Methods for Anomaly Detection

1356.1 Learning Objectives

By the end of this chapter, you will be able to:

  • Apply Z-Score Detection: Implement standard deviation-based anomaly detection for Gaussian data
  • Use IQR Method: Deploy robust outlier detection that works with any distribution
  • Build Adaptive Thresholds: Create moving statistics systems that handle concept drift
  • Select Appropriate Methods: Choose between statistical approaches based on data characteristics
TipMinimum Viable Understanding: Statistical Anomaly Detection

Core Concept: Statistical methods detect anomalies by measuring how far a value deviates from β€œnormal” - either in standard deviations (Z-score) or quartile ranges (IQR).

Why It Matters: Statistical methods are computationally lightweight, require no training data, and can run on resource-constrained edge devices. They catch 80% of anomalies with 10% of the complexity of ML approaches.

Key Takeaway: Use Z-score for Gaussian data, IQR for skewed or bounded data, and adaptive thresholds when β€œnormal” changes over time.

1356.2 Prerequisites

Before diving into this chapter, you should be familiar with:

~15 min | Intermediate | P10.C01.U02

1356.3 Introduction

Statistical approaches form the foundation of anomaly detection. They are computationally lightweight, suitable for edge deployment, and work well when data follows known distributions.

1356.4 Z-Score (Standard Deviation Method)

Core Concept: Measure how many standard deviations a data point is from the mean.

Formula:

Z = (x - mu) / sigma

Where:
- x = observed value
- mu = mean of dataset
- sigma = standard deviation
- |Z| > 3 typically indicates anomaly (99.7% confidence interval)

When It Works: - Data follows Gaussian (normal) distribution - You have sufficient historical data to calculate mu and sigma - Distribution is stable over time (no concept drift)

When It Fails: - Non-normal distributions (skewed, multi-modal) - Data with seasonal patterns - When β€œnormal” range changes dynamically

Implementation Example:

import numpy as np

class ZScoreDetector:
    def __init__(self, threshold=3.0, window_size=100):
        self.threshold = threshold
        self.window_size = window_size
        self.values = []

    def update(self, value):
        """Streaming Z-score anomaly detection"""
        self.values.append(value)

        # Keep only recent window
        if len(self.values) > self.window_size:
            self.values.pop(0)

        # Need minimum samples
        if len(self.values) < 30:
            return False, 0.0

        mean = np.mean(self.values)
        std = np.std(self.values)

        if std == 0:  # Avoid division by zero
            return False, 0.0

        z_score = abs((value - mean) / std)
        is_anomaly = z_score > self.threshold

        return is_anomaly, z_score

# Usage example: Temperature sensor monitoring
detector = ZScoreDetector(threshold=3.0)

# Normal readings
for temp in [22.1, 22.5, 21.8, 22.3, 22.0]:
    anomaly, score = detector.update(temp)
    print(f"Temp: {temp}C, Z-score: {score:.2f}, Anomaly: {anomaly}")

# Anomalous reading
anomaly, score = detector.update(-5.0)  # Sensor malfunction
print(f"Temp: -5.0C, Z-score: {score:.2f}, Anomaly: {anomaly}")
# Output: Temp: -5.0C, Z-score: 4.87, Anomaly: True

1356.5 Interquartile Range (IQR)

TipUnderstanding Outlier Detection

Core Concept: Outlier detection identifies data points that fall significantly outside the expected distribution of normal values - distinguishing genuine anomalies from sensor noise and measurement errors.

Why It Matters: Not all unusual values indicate real-world problems. A temperature spike from 22C to -40C is almost certainly a sensor malfunction, not actual weather. Conversely, a gradual drift from 22C to 28C over weeks might indicate genuine HVAC degradation. Proper outlier detection separates actionable anomalies from data quality issues, preventing both missed alerts and false alarms that erode operator trust.

Key Takeaway: Use IQR (Interquartile Range) when your data is skewed or has natural outliers - it is robust because it ignores extreme values. Use Z-score when data is normally distributed and you need speed (simpler calculation). For IoT sensors with physical constraints (humidity 0-100%, voltage 0-5V), combine statistical methods with hard bounds - any reading outside physical limits is immediately flagged as a sensor fault, not an anomaly.

Core Concept: Identify outliers based on the spread of the middle 50% of data.

Formula:

IQR = Q3 - Q1
Lower Bound = Q1 - 1.5 x IQR
Upper Bound = Q3 + 1.5 x IQR

Outlier if: x < Lower Bound OR x > Upper Bound

Where:
- Q1 = 25th percentile
- Q3 = 75th percentile

Advantages over Z-Score: - No distribution assumptions (works with skewed data) - Robust to extreme outliers - Good for bounded sensor ranges

IoT Use Cases: - Battery voltage monitoring (naturally bounded 0-4.2V) - Humidity sensors (0-100% RH) - Any sensor with physical constraints on range

Implementation:

import numpy as np

def iqr_anomaly_detection(data, value):
    """
    Detect if a new value is anomalous using IQR method
    """
    q1 = np.percentile(data, 25)
    q3 = np.percentile(data, 75)
    iqr = q3 - q1

    lower_bound = q1 - 1.5 * iqr
    upper_bound = q3 + 1.5 * iqr

    is_anomaly = (value < lower_bound) or (value > upper_bound)

    return is_anomaly, lower_bound, upper_bound

# Example: Battery voltage monitoring (normal: 3.3-4.1V)
battery_readings = [3.85, 3.92, 3.78, 3.88, 3.95, 3.82, 3.90,
                   3.87, 3.93, 3.81, 3.89, 3.86]

# Check new reading
new_reading = 2.1  # Low battery or sensor fault
anomaly, lower, upper = iqr_anomaly_detection(battery_readings, new_reading)

print(f"Battery: {new_reading}V")
print(f"Expected range: {lower:.2f}V - {upper:.2f}V")
print(f"Anomaly: {anomaly}")
# Output: Battery: 2.1V, Expected range: 3.57V - 4.16V, Anomaly: True

1356.6 Moving Statistics (Adaptive Thresholds)

Core Concept: Calculate mean and standard deviation over a sliding window to adapt to changing conditions.

Why It Matters for IoT: - Sensor characteristics drift over time (aging, temperature effects) - Environmental conditions change (seasons, occupancy patterns) - Fixed thresholds become obsolete

Implementation:

from collections import deque
import numpy as np

class AdaptiveThresholdDetector:
    def __init__(self, window_size=50, n_std=3.0):
        self.window = deque(maxlen=window_size)
        self.n_std = n_std

    def update(self, value):
        """
        Add new value and check if it's anomalous
        Returns: (is_anomaly, threshold_lower, threshold_upper)
        """
        # Calculate thresholds from current window
        if len(self.window) > 10:
            mean = np.mean(self.window)
            std = np.std(self.window)

            lower = mean - self.n_std * std
            upper = mean + self.n_std * std

            is_anomaly = (value < lower) or (value > upper)
        else:
            # Not enough data yet
            is_anomaly = False
            lower, upper = None, None

        # Add to window AFTER checking (don't contaminate with anomalies)
        if not is_anomaly:
            self.window.append(value)

        return is_anomaly, lower, upper

# Example: Indoor temperature monitoring with day/night cycles
detector = AdaptiveThresholdDetector(window_size=50, n_std=2.5)

# Simulate temperature readings over time
import random
for hour in range(24):
    # Normal pattern: warmer during day (8am-6pm)
    if 8 <= hour <= 18:
        base_temp = 23.0
    else:
        base_temp = 19.0

    # Add small random variation
    temp = base_temp + random.uniform(-0.5, 0.5)

    anomaly, lower, upper = detector.update(temp)
    if lower:
        print(f"Hour {hour}: {temp:.1f}C, "
              f"Range: [{lower:.1f}, {upper:.1f}], "
              f"Anomaly: {anomaly}")

Window Size Trade-Offs:

Window Size Responsiveness Stability Best For
Small (10-50) High - detects sudden changes Low - sensitive to noise Fast-changing environments
Medium (50-200) Balanced Balanced General IoT monitoring
Large (200+) Low - slow to adapt High - ignores noise Stable industrial processes
WarningTradeoff: Fixed Thresholds vs Adaptive Detection

Option A: Fixed statistical thresholds (Z-score > 3, IQR bounds) Option B: Adaptive thresholds with moving statistics Decision Factors: Fixed thresholds are simpler to implement and explain (auditable for compliance), work well when normal behavior is stable, and have lower computational overhead. Adaptive methods handle concept drift, seasonal patterns, and changing sensor characteristics, but require tuning window sizes and may temporarily miss anomalies during adaptation periods. Choose fixed for stable industrial processes; choose adaptive for environments with natural variation like building HVAC or outdoor sensors.

1356.7 Interactive Demo: Anomaly Detection Methods

Experiment with different anomaly detection methods on simulated IoT sensor data. Adjust the threshold and detection method to see how they affect true positive and false positive rates.

How to use this demo:

  1. Detection Method: Compare Z-score (global statistics), moving average (adaptive), and isolation concept (robust to outliers)
  2. Threshold: Lower thresholds catch more anomalies but increase false alarms
  3. Window Size: For moving average, controls how quickly the baseline adapts
  4. Anomaly Rate: Simulate different anomaly frequencies

Key observations:

  • Trade-off: Lowering the threshold improves recall (detection rate) but decreases precision (more false alarms)
  • Moving Average: Better for data with drift but needs tuning of window size
  • Isolation Concept: More robust to extreme outliers in the training data

1356.8 Method Comparison

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#16A085', 'secondaryColor': '#E67E22', 'tertiaryColor': '#7F8C8D'}}}%%

flowchart TB
    subgraph ZScore["Z-Score Method"]
        Z1[Compute Time: O(1)]
        Z2[Memory: mean, std only]
        Z3[Edge Friendly: Yes]
        Z4[Assumption: Gaussian]
        Z5[Drift Handling: Poor]
    end

    subgraph IQR["IQR Method"]
        I1[Compute Time: O(n log n)]
        I2[Memory: Full window]
        I3[Edge Friendly: Medium]
        I4[Assumption: None]
        I5[Drift Handling: Poor]
    end

    subgraph Adaptive["Moving Average"]
        A1[Compute Time: O(1)]
        A2[Memory: Window buffer]
        A3[Edge Friendly: Yes]
        A4[Assumption: None]
        A5[Drift Handling: Good]
    end

    subgraph Recommend["Deployment Recommendations"]
        R1[Edge/MCU: Z-Score<br/>Minimal memory]
        R2[Gateway: Moving Avg<br/>Handles drift]
        R3[Cloud: IQR + Ensemble<br/>Maximum robustness]
    end

    ZScore --> R1
    Adaptive --> R2
    IQR --> R3

    style ZScore fill:#16A085,color:#fff
    style IQR fill:#E67E22,color:#fff
    style Adaptive fill:#2C3E50,color:#fff
    style Recommend fill:#7F8C8D,color:#fff

Figure 1356.1: Comparison matrix showing computational characteristics and deployment recommendations for each statistical method. Z-score suits resource-constrained edge devices, moving average handles concept drift at gateways, and IQR combined with ensembles provides maximum robustness in cloud deployments.

1356.9 Summary

Statistical methods provide the foundation for anomaly detection in IoT:

  • Z-Score: Fast, simple, works for Gaussian data. Best for edge devices with minimal memory.
  • IQR: Robust to outliers and skewed data. Requires more memory for sorting.
  • Adaptive Thresholds: Handles concept drift. Requires tuning of window size.

Key Takeaway: Start with statistical methods. They catch 80% of anomalies with minimal resources. Only escalate to ML when statistical approaches fail.

1356.10 What’s Next

Continue learning about anomaly detection methods: