Analytics & ML · Study deck
Feature Scaling Lab: Statistical Analysis
Task: The code defines a MAD_THRESHOLD constant but does not implement MAD outlier detection.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Test step 7: analyze statistics with a concrete scenario and pass criteria.
- Validate feature scaling leakage controls with a concrete scenario and pass criteria.
- 'test step 7: analyze statistics with a concrete scenario and pass criteria'
- validate feature scaling leakage controls with a concrete scenario and pass criteria
Major section
MAD Filtering Challenge
Task: The code defines a MAD_THRESHOLD constant but does not implement MAD outlier detection.
- Expected Outcome: MAD should detect outliers even when extreme values skew the mean and standard deviation.
Major section
Challenge 3: Cross-Sensor Validation
If it is very bright (high light), temperature should be reasonable for daytime.
- Expected Outcome: The system should detect when sensor readings are physically inconsistent with each other.
Major section
Normalize Sensors for Anomalies
A neural network needs all inputs on the same scale to detect abnormal conditions.
- The network would learn to minimize CO2 error while ignoring temperature and humidity!
- Key Insight: Without normalization, the neural network's loss function is dominated by the largest-magnitude features.
Major section
Normalize After Train/Test Split
The Mistake: Calculating normalization parameters (min, max, mean, std) on the entire dataset before splitting into train and test sets.
- This causes data leakage, where the test set's statistics influence the training process, leading to overly optimistic performance estimates.
- Scikit-learn's fit_transform() makes it easy to accidentally normalize everything at once.
Major section
Renormalize for New Sensors
Adding a new sensor channel to an existing normalised dataset requires recomputing normalisation parameters.
- Hardcoded normalisation bounds from the initial dataset will not accommodate the new sensor's value range.
Major section
Summary
Catching data quality issues at the source costs 1% of fixing them in the cloud, and normalized data enables fair multi-sensor fusion.
- Critical Design Principle: The complete "validate-clean-transform" pipeline should run at the edge.
Major section
Concept Relationships
Data Validation and Outlier Detection: Range checks and outlier detection must occur before normalization to avoid skewing scaling parameters with invalid data.
- Multi-Sensor Data Fusion: Normalization enables fair comparison when fusing sensors with vastly different measurement ranges (temperature vs light intensity).
- Edge Data Acquisition: Edge devices normalize locally to reduce transmission bandwidth and prepare data for edge ML inference.
Major section
Concept Relationships (continued)
Modeling and Inferencing: Neural networks require normalized inputs ([0,1] or mean=0, std=1) for stable gradient descent.
- Anomaly Detection: Z-score normalization makes distance-based anomaly detection work across features with different scales.
- Key Insight:: Normalization is the final stage of the validate-clean-transform pipeline.
- Applying it before validation or cleaning causes incorrect scaling parameters (e.g., min-max uses outlier as max value, compressing all valid data into a tiny range).
Deck summary
Key takeaways
Task: The code defines a MAD_THRESHOLD constant but does not implement MAD outlier detection.
- If it is very bright (high light), temperature should be reasonable for daytime.
- A neural network needs all inputs on the same scale to detect abnormal conditions.
- The Mistake: Calculating normalization parameters (min, max, mean, std) on the entire dataset before splitting into train and test sets.
- Adding a new sensor channel to an existing normalised dataset requires recomputing normalisation parameters.
Retrieval practice
Recall check 1 of 3
Data Dora says: answer from memory, then check your reasoning.
Q1Place each normalization-lab responsibility where it lives so you can reproduce a scale transform and reverse it without leaking test information.
Show answer
Answer: A reproduce a scale transform and reverse it without leaking test information.
Q2Complete the sensor data quality pipeline:
Show answer
Answer: A Sensor data cleaning validates physical limits while preserving missing markers, then interpolates the gaps and smooths with an exponential moving average.
Retrieval practice
Recall check 2 of 3
Data Dora says: answer from memory, then check your reasoning.
Q3You are training a neural network to classify room occupancy using temperature (15-35C) and CO2 (400-5000 ppm) sensor data. Without normalization, what problem will the model likely exhibit?
Show answer
Answer: C Without normalization, CO2 values (400-5000) are roughly 200x larger than temperature values (15-35).
Retrieval practice
Recall check 3 of 3
Data Dora says: answer from memory, then check your reasoning.
Q4A humidity sensor dataset has the values [45, 48, 50, 52, 55, 99, 47, 51, 49, 50] where the value 99 is a stuck-sensor outlier. Which normalization method would be LEAST affected by this outlier?
Show answer
Answer: B Robust scaling uses the median (resistant to outliers) and interquartile range (25th to 75th percentile, also resistant).
Print reference
Answers
Answer key.
- A · reproduce a scale transform and reverse it without leaking test information.
- A · Sensor data cleaning validates physical limits while preserving missing markers, then interpolates the gaps and smooths with an exponential moving average.
- C · Without normalization, CO2 values (400-5000) are roughly 200x larger than temperature values (15-35).
- B · Robust scaling uses the median (resistant to outliers) and interquartile range (25th to 75th percentile, also resistant).