27 Feature Scaling Lab: Statistical Analysis

analytics-ml

data

quality

normalization

27.1 Start With the Decision

Wait for the statistics report (prints every 10 seconds) 2.

27.2 Route Overview

This is part 2 of 2. Review Feature Scaling Lab: Preprocessing Workflow for the preceding evidence.

27.3 Learning Objectives

Test step 7: analyze statistics with a concrete scenario and pass criteria.
Validate feature scaling leakage controls with a concrete scenario and pass criteria.

27.4 Chapter Roadmap

Step 7: Analyze Statistics
Checkpoint: Edge Pipeline
Challenge Exercises
MAD Filtering Challenge
Backoff for Missing Data
Challenge 3: Cross-Sensor Validation
Challenge 4: Implement Kalman Filter
Expected Outcomes
Try It: Exponential Smoothing Explorer
Normalize Sensors for Anomalies
Try It: Multi-Sensor Fusion Calculator
Checkpoint: Fusion Choices
Choose Normalization Method
Normalize After Train/Test Split
Checkpoint: Leakage Control
Try It: Data Leakage Visualizer
Interactive Quiz: Match Concepts
Interactive Quiz: Sequence the Steps
Common Pitfalls
Fit Normalizers After Splitting
Normalize by Sensor Type
3. Normalising labels (target variables)
Renormalize for New Sensors
Label the Diagram
Code Challenge
Checkpoint: Assessment Path
Feature Scaling Leakage Controls
Summary
Quiz: Data Normalization
Key Takeaway
Concept Relationships
What’s Next
See Also

27.5 Step 7: Analyze Statistics

Read the Step 7: Analyze Statistics material as a decision path rather than as isolated entries. First identify the operating condition in each entry and keep its units, timing, source, and assumed system state attached to it. Next compare the entries at the point where responsibility changes between device, gateway, network, analytic service, and operator; that hand-off is where apparently similar choices often produce different outcomes. Then follow the failure case: ask what becomes stale, delayed, unavailable, or unsafe, who detects it, and what evidence permits recovery. Finally connect the result to the chapter’s running design record by naming the selected behavior, the rejected alternative, the measurement that justifies the choice, and the condition that forces a recheck. That order turns the examples or comparison into an auditable engineering argument.

Wait for the statistics report (prints every 10 seconds)
Review the data quality percentages: valid, outliers, missing
Examine the sensor statistics: mean, standard deviation, range
Consider how these metrics would inform production monitoring

Checkpoint: Edge Pipeline

You now know:

The simulator samples every 200 ms, so the lab processes data at 5 Hz while maintaining rolling statistics.
Physical validation uses bounds such as -10 to 60 C for temperature and 0 to 100000 lux for light.
The pipeline marks range/rate violations, outliers, missing values, imputed values, and clean readings before it emits normalized output.

27.6 Challenge Exercises

MAD Filtering Challenge

Difficulty: Intermediate

Task: The code defines a MAD_THRESHOLD constant but does not implement MAD outlier detection. Implement a detectOutlierMAD() function and integrate it into the processTemperature() pipeline as the primary outlier detection method.

Hints:

MAD is more robust than Z-score for non-Gaussian data
You will need to implement a detectOutlierMAD() function (modified Z-score = 0.6745 * (x - median) / MAD)
Replace or complement the Z-score check with MAD

Expected Outcome: MAD should detect outliers even when extreme values skew the mean and standard deviation.

Backoff for Missing Data

Difficulty: Intermediate

Task: Currently, forward-fill uses a fixed limit (MAX_MISSING_SAMPLES). Implement an exponential decay on the confidence of imputed values.

Requirements:

Add a confidence field to SensorReading
Reduce confidence by 10% for each consecutive imputed value
Stop imputing when confidence drops below 50%
Display confidence in serial output

Expected Outcome: Imputed values should be flagged with decreasing confidence as gaps grow longer.

Challenge 3: Cross-Sensor Validation

Difficulty: Advanced

Task: Add plausibility checking between temperature and light sensors. If it is very bright (high light), temperature should be reasonable for daytime.

Requirements:

If light > 50000 lux and temperature < 10C, flag as suspicious
If light < 100 lux and temperature > 35C (outdoors), flag as suspicious
Add a new LED or serial indicator for cross-sensor anomalies

Expected Outcome: The system should detect when sensor readings are physically inconsistent with each other.

Challenge 4: Implement Kalman Filter

Difficulty: Advanced

Task: Replace the exponential smoothing filter with a simple Kalman filter for temperature.

Requirements:

Implement 1D Kalman filter with process noise and measurement noise
Estimate the Kalman gain dynamically
Output both the filtered value and the uncertainty estimate

Learning: Kalman filters provide optimal estimation when process and measurement noise characteristics are known.

27.7 Expected Outcomes

After completing this lab, you should be able to:

Understand validation trade-offs: Strict validation catches more errors but may reject valid extreme readings
Choose appropriate outlier methods: Z-score for Gaussian data, IQR/MAD for robust detection
Select imputation strategies: Forward-fill for slow-changing, interpolation for trending data
Apply noise filters correctly: Median for spikes, moving average for steady-state noise
Normalize for fusion: Understand when to use min-max vs Z-score normalization

Quality Metrics to Observe:

Valid sample rate should be >90% under normal conditions
Outlier rate should be <5% for stable sensors
Imputed values should maintain temporal continuity

Try It: Exponential Smoothing Explorer

Adjust the smoothing factor (alpha) and noise level to see how exponential smoothing filters noisy sensor data. A low alpha trusts the history more (smoother), while a high alpha trusts new readings more (more responsive).

Normalize Sensors for Anomalies

Scenario: You’re building an anomaly detection system for a server room with 3 sensors: temperature (15-35°C), CO2 (400-5000 ppm), and humidity (30-70%). A neural network needs all inputs on the same scale to detect abnormal conditions.

Given:

Temperature sensor: range 15-35°C, current reading 28°C
CO2 sensor: range 400-5000 ppm, current reading 1200 ppm
Humidity sensor: range 30-70%, current reading 55%
Neural network requires inputs in 0-1 range

Question: Normalize these readings and explain why proper normalization matters for the neural network.

Solution:

Step 1: Calculate min-max normalization for each sensor

Temperature normalization:

normalized_temp = (28 - 15) / (35 - 15)
               = 13 / 20
               = 0.65

CO2 normalization:

normalized_co2 = (1200 - 400) / (5000 - 400)
              = 800 / 4600
              = 0.174

Humidity normalization:

normalized_humidity = (55 - 30) / (70 - 30)
                   = 25 / 40
                   = 0.625

Step 2: Show the impact WITHOUT normalization

If we fed raw values to the neural network:

Input vector: [28, 1200, 55]
CO2 value is 40x larger than temperature
Neural network gradient updates would be dominated by CO2

Example gradient calculation (simplified):

Cost function: J = (pred - actual)²
Gradient w.r.t. weight: ∂J/∂w = 2 × (pred - actual) × input_value

For temperature: gradient ∝ 28
For CO2:         gradient ∝ 1200  (43x larger!)
For humidity:    gradient ∝ 55

The network would learn to minimize CO2 error while ignoring temperature and humidity!

Step 3: Verify normalized inputs have equal influence

Normalized input vector: [0.65, 0.174, 0.625]

Now all gradients are on similar scales:

For temperature: gradient ∝ 0.65
For CO2:         gradient ∝ 0.174
For humidity:    gradient ∝ 0.625

Each sensor contributes equally to gradient updates during training.

Step 4: Calculate percentage of each sensor’s range

This helps interpret the normalized values:

Temperature: 65% of its range (moderately warm)
CO2: 17.4% of its range (relatively low, good ventilation)
Humidity: 62.5% of its range (comfortable level)

Step 5: Detect an anomaly scenario

Normal condition: [0.65, 0.174, 0.625] Anomaly (AC failure): [0.95, 0.350, 0.825]

Temperature: 95% of range = 34°C (very hot!)
CO2: 35% of range = 2010 ppm (rising, poor ventilation)
Humidity: 82.5% of range = 63% (uncomfortable)

The neural network trained on normalized data can now detect this pattern as anomalous, because all three sensors contributed equally during training.

Key Insight: Without normalization, the neural network’s loss function is dominated by the largest-magnitude features. Min-max scaling ensures each sensor contributes proportionally to its information content, not its arbitrary measurement scale. This is why normalization is mandatory for neural networks, SVM, K-means, and any algorithm that uses distance metrics or gradient descent.

27.8 Try It: Multi-Sensor Fusion Calculator

The next question is selection: after you can normalize one stream, which scaler should each feature use when the data set mixes clean bounded channels with outlier-prone sensor readings?

Adjust the sensor readings below to see how min-max normalization brings different scales into alignment:

Run it: Before you commit to a branch of the decision tree below, operate the method on real numbers. Load the default temperature readings in the normalization workbench, switch between Min-Max, Z-score, and Robust-IQR, then flip the outlier toggle to drop an 85°C stuck-sensor spike into the set. Watch the Min-Max range jump from 7 to 67 while the same 22°C reading collapses from 0.571429 to 0.059701 — the 4/67 case from Ada’s audit — and the Robust-IQR denominator barely moves, holding the clean readings apart. Use what you see to justify each “outliers expected?” answer before you pick a scaler.

Checkpoint: Fusion Choices

You now know:

Raw CO2 at 1200 ppm can dominate temperature at 28 C and humidity at 55% unless the inputs are put on comparable scales.
The server-room example normalizes temperature to 0.65, CO2 to 0.174, and humidity to 0.625 before comparing them.
An 85 C outlier can collapse the 22 C min-max value from 0.571429 to 0.059701, which is why outlier-prone features need robust treatment.

Choose Normalization Method

Choose the appropriate normalization method based on your data characteristics and downstream algorithm:

Data Characteristic	Normalization Method	Output Range	Best For	Avoid When
Bounded range known (temperature, humidity)	Min-Max Scaling	0 to 1	Neural networks, image processing, bounded outputs	Outliers present (they compress valid range)
Outliers expected (sensor noise, network latency)	Robust Scaling	Median-centered	K-means, SVM, any distance-based method	Need exact 0-1 bounds
Gaussian distribution (many natural phenomena)	Z-Score Normalization	Mean=0, Std=1	Clustering, PCA, algorithms assuming normal distribution	Binary features (0/1)
Exponential/power-law (network traffic, wealth)	Log Transform then Z-Score	Variable	Right-skewed data, multiplicative relationships	Zero or negative values present
Mixed data types (some outliers + some bounded)	Hybrid: Robust for outlier features, Min-Max for clean	Variable per feature	Real-world messy datasets	Need uniform scaling method

Decision Tree:

are there extreme outliers (>5% of values beyond 3σ)?
- YES → Use Robust Scaling (median + IQR)
- NO → Continue to step 2
is your algorithm neural-network-based?
- YES → Use Min-Max Scaling (0-1) for activation function compatibility
- NO → Continue to step 3
does your algorithm assume Gaussian distribution?
- YES (PCA, LDA) → Use Z-Score Normalization
- NO → Continue to step 4
is your data heavily right-skewed (long tail)?
- YES → Log Transform + Z-Score
- NO → Default to Min-Max Scaling

Example Python Implementation:

def select_normalizer(data_characteristics):
    """
    Select appropriate normalization based on data characteristics.
    Returns: (normalizer_class, parameters)
    """
    has_outliers = data_characteristics['outlier_rate'] > 0.05
    is_neural_network = data_characteristics['model_type'] == 'neural_network'
    is_gaussian = data_characteristics['distribution'] == 'normal'
    is_skewed = data_characteristics['skewness'] > 2.0

    if has_outliers:
        return (RobustScaler, {})
    elif is_neural_network:
        return (MinMaxScaler, {'feature_range': (0, 1)})
    elif is_gaussian:
        return (ZScoreNormalizer, {})
    elif is_skewed:
        return (LogTransform, {'then_zscore': True})
    else:
        return (MinMaxScaler, {'feature_range': (0, 1)})

# Usage example
characteristics = {
    'outlier_rate': 0.08,  # 8% outliers
    'model_type': 'kmeans',
    'distribution': 'unknown',
    'skewness': 1.2
}

normalizer_class, params = select_normalizer(characteristics)
# Returns: RobustScaler (because outlier_rate > 0.05)

Warning Signs of Wrong Normalization:

Neural network accuracy plateaus at 60% → Inputs not normalized
Clustering groups all high-magnitude features together → Need Z-score instead of raw
Model ignores certain sensors → Their raw ranges are too small (need min-max)
Training loss explodes after first epoch → Gradients too large (need normalization)

Normalize After Train/Test Split

The Mistake: Calculating normalization parameters (min, max, mean, std) on the entire dataset before splitting into train and test sets. This causes data leakage, where the test set’s statistics influence the training process, leading to overly optimistic performance estimates.

Why It Happens: The normalization step feels like “data preparation” rather than “model training,” so developers apply it before the split. Many tutorials skip this detail. Scikit-learn’s fit_transform() makes it easy to accidentally normalize everything at once.

Example of the Problem:

# WRONG: Normalize first, split second
data = load_sensor_data()  # 10,000 samples
normalized = MinMaxScaler().fit_transform(data)  # Uses ALL data stats!
train, test = train_test_split(normalized, test_size=0.2)

# The test set's min/max values influenced the scaling parameters!
# Model evaluation is now too optimistic.

Why This Is Wrong:

Data Leakage: Test set statistics “leak” into training through normalization parameters
Overfitting: Model appears to generalize better than it actually does
Production Failure: Real-world data has different min/max than training data

Real-World Example:

Imagine temperature sensor data:

Training period (winter): 15-25°C
Test period (summer): 20-35°C

WRONG approach:

# Calculate on ALL data (winter + summer)
overall_min = 15°C, overall_max = 35°C

# Normalize training data using overall stats
train_normalized = (train - 15) / (35 - 15)
# Result: Training temps (15-25°C) map to 0.0-0.5

# Normalize test data using SAME overall stats
test_normalized = (test - 15) / (35 - 15)
# Result: Test temps (20-35°C) map to 0.25-1.0

# Model trained on 0.0-0.5 range, tested on 0.25-1.0 range
# Test performance appears good because model "saw" summer data during normalization!

The Fix: Fit normalization ONLY on training data, then apply to test:

# CORRECT: Split first, normalize second
data = load_sensor_data()
train, test = train_test_split(data, test_size=0.2)  # Split FIRST

# Fit normalization on training data ONLY
scaler = MinMaxScaler()
scaler.fit(train)  # Learns min=15, max=25 from training (winter)

# Apply fitted scaler to both train and test
train_normalized = scaler.transform(train)
test_normalized = scaler.transform(test)  # Summer data (20-35°C) may exceed [0,1]!

# This is CORRECT - test data reflecting real-world distribution

Correct Pipeline Order:

Split data into train/validation/test sets (e.g., 70/15/15)
Fit normalization parameters on TRAINING data only
Transform train, validation, and test sets using those parameters
Train model on normalized training data
Evaluate on normalized validation/test data

Code Template:

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler

# 1. Split FIRST
X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.3)
X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5)

# 2. Fit normalizer on TRAINING data only
scaler = MinMaxScaler()
scaler.fit(X_train)  # Only training data!

# 3. Transform ALL sets using training parameters
X_train_norm = scaler.transform(X_train)
X_val_norm = scaler.transform(X_val)
X_test_norm = scaler.transform(X_test)

# 4. Train model
model.fit(X_train_norm, y_train)

# 5. Evaluate (no data leakage!)
val_score = model.score(X_val_norm, y_val)
test_score = model.score(X_test_norm, y_test)

Warning Signs of This Mistake:

Test accuracy is suspiciously high (>95%) on first try
Model performance degrades significantly in production
Test set values occasionally exceed [0, 1] after normalization (this is actually GOOD - means no leakage!)
Reviewer asks “when did you fit the scaler?” and you can’t answer clearly

Real-World Impact: A smart building energy prediction model achieved 96% test accuracy during development but only 73% accuracy in production. Root cause: normalization was fit on the entire year’s data (including test set), so the model “saw” summer peak loads during training. In production, the next summer’s peak loads were outside the normalized range, causing poor predictions.

Checkpoint: Leakage Control

You now know:

Data leakage happens when min, max, mean, or standard deviation are fit on all data before the split.
A clean pipeline splits first, fits the scaler on training data only, then transforms train, validation, and test using those training parameters.
The chapter’s template uses a 70/15/15 split, so validation and test data can honestly reveal values outside the training range.

Try It: Data Leakage Visualizer

Explore how normalizing before vs. after the train/test split changes the scaling parameters. Adjust the test set distribution to see how leakage distorts the training normalization.

27.9 Interactive Quiz: Match Concepts

27.10 Interactive Quiz: Sequence the Steps

Common Pitfalls

Fit Normalizers After Splitting

Computing Min-Max bounds or Z-score mean/std on all data before splitting leaks test set information into training. Always fit normalisation parameters on the training set only and apply them to both train and test sets.

Normalize by Sensor Type

Binary status signals (0/1), count data, and continuous physical measurements require different normalisation strategies. Normalise each channel according to its distribution, not uniformly.

3. Normalising labels (target variables)

In regression tasks, some implementations accidentally normalise the prediction target along with features, causing the model to predict normalised units that require inverse transformation. Keep target variables in their original units unless the algorithm specifically requires otherwise.

Renormalize for New Sensors

Adding a new sensor channel to an existing normalised dataset requires recomputing normalisation parameters. Hardcoded normalisation bounds from the initial dataset will not accommodate the new sensor’s value range.

27.11 Label the Diagram

27.12 Code Challenge

Read the Code Challenge material as a decision path rather than as isolated entries. First identify the operating condition in each entry and keep its units, timing, source, and assumed system state attached to it. Next compare the entries at the point where responsibility changes between device, gateway, network, analytic service, and operator; that hand-off is where apparently similar choices often produce different outcomes. Then follow the failure case: ask what becomes stale, delayed, unavailable, or unsafe, who detects it, and what evidence permits recovery. Finally connect the result to the chapter’s running design record by naming the selected behavior, the rejected alternative, the measurement that justifies the choice, and the condition that forces a recheck. That order turns the examples or comparison into an auditable engineering argument.

Checkpoint: Assessment Path

You now know:

The matching quiz separates min-max, Z-score, robust scaling, log transform, and circular-buffer ideas.
The sequencing quiz keeps the processing order explicit: read ADC, validate, detect outliers, filter, then normalize.
The code challenge reinforces that validation comes before gap filling and smoothing.

27.13 Feature Scaling Leakage Controls

The lab above covers normalization formulas, ESP32 data-quality code, scaling calculators, leakage visualizers, and assessment practice. Continue to Feature Scaling and Leakage Controls for the deeper L2 material: method selection across min-max, z-score, and robust scaling; scaler-parameter records; train-only fitting; and production inference consistency.

27.14 Summary

Data normalization and scaling complete the data quality preprocessing pipeline:

Min-Max Scaling: Transforms data to 0-1 range, ideal for neural networks and bounded outputs
Z-Score Normalization: Centers data around mean with unit variance, best for clustering and SVM
Robust Scaling: Uses median and IQR, resistant to outliers
Log Transform: Compresses right-skewed data spanning orders of magnitude
Complete Pipeline: Validation -> Cleaning -> Transformation, all implementable on edge devices

Critical Design Principle: The complete “validate-clean-transform” pipeline should run at the edge. Catching data quality issues at the source costs 1% of fixing them in the cloud, and normalized data enables fair multi-sensor fusion.

27.15 Quiz: Data Normalization

Key Takeaway

Data normalization is essential before combining multi-sensor data or feeding it to machine learning models. Choose your method based on data characteristics: min-max for bounded outputs, Z-score for Gaussian assumptions, robust scaling when outliers are present. Always fit normalization parameters on training data only to avoid data leakage. Implement the complete validate-clean-transform pipeline at the edge to catch data quality issues at the source.

27.16 Concept Relationships

This chapter builds on validation and cleaning while introducing normalization as the final pipeline stage:

Prerequisites (Must understand first):

Data Validation and Outlier Detection - Range checks and outlier detection must occur before normalization to avoid skewing scaling parameters with invalid data
Missing Value Imputation and Noise Filtering - Cleaning must precede normalization; otherwise, gaps and noise corrupt min/max or mean/std calculations

Related Concepts (Enhance understanding):

Multi-Sensor Data Fusion - Normalization enables fair comparison when fusing sensors with vastly different measurement ranges (temperature vs light intensity)
Edge Data Acquisition - Edge devices normalize locally to reduce transmission bandwidth and prepare data for edge ML inference

Advanced Applications (Build on this):

Modeling and Inferencing - Neural networks require normalized inputs ([0,1] or mean=0, std=1) for stable gradient descent
Anomaly Detection - Z-score normalization makes distance-based anomaly detection work across features with different scales

Key Insight: Normalization is the final stage of the validate-clean-transform pipeline. Applying it before validation or cleaning causes incorrect scaling parameters (e.g., min-max uses outlier as max value, compressing all valid data into a tiny range).

27.17 What’s Next

If you want to…	Read this
Understand imputation techniques that precede normalisation	Data Quality Imputation and Filtering
Study the full preprocessing pipeline	Data Quality and Preprocessing
Dig deeper into scaler method selection and train/test leakage	Feature Scaling and Leakage Controls
Apply normalised data to ML model training	Modeling and Inferencing
Understand data validation before normalisation	Data Quality Validation
Return to the module overview	Big Data Overview

27.18 See Also

Data Quality Series:

Data Quality and Preprocessing - Overview and index
Data Validation and Outlier Detection - Validation and outliers
Missing Value Imputation and Noise Filtering - Handling gaps and noise

Practical Applications:

Multi-Sensor Data Fusion - Combining preprocessed data
Stream Processing - Real-time data pipelines
Edge Compute Patterns - Distributed preprocessing architectures

Advanced Topics:

Modeling and Inferencing - ML model deployment with quality data
Anomaly Detection - Finding meaningful outliers in clean data

27.19 Continue Your Route

This final part closes the route from Step 7: Analyze Statistics through See Also. Return to Feature Scaling Lab: Preprocessing Workflow or continue from the analytics-ml module index.