Analytics & ML · Study deck
Feature Scaling Lab: Statistical Analysis
Task: The code defines a MAD_THRESHOLD constant but does not implement MAD outlier detection.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Test step 7: analyze statistics with a concrete scenario and pass criteria.
- Validate feature scaling leakage controls with a concrete scenario and pass criteria.
- Explain: Task: The code defines a MAD_THRESHOLD constant but does not implement MAD outlier detection.
- Explain: If it is very bright (high light), temperature should be reasonable for daytime.
Major section
MAD Filtering Challenge
Task: The code defines a MAD_THRESHOLD constant but does not implement MAD outlier detection.
- Expected Outcome: MAD should detect outliers even when extreme values skew the mean and standard deviation.
Major section
Challenge 3: Cross-Sensor Validation
If it is very bright (high light), temperature should be reasonable for daytime.
- Expected Outcome: The system should detect when sensor readings are physically inconsistent with each other.
Major section
Normalize Sensors for Anomalies
A neural network needs all inputs on the same scale to detect abnormal conditions.
- Unequal scales can lead to disproportionately sized updates; training does not necessarily ignore temperature or humidity.
- Comparable input scales do not make each sensor contribute equally to gradient updates during training.
Major section
Normalize After Train/Test Split
The Mistake: Calculating normalization parameters (min, max, mean, std) on the entire dataset before splitting into train and test sets.
- This causes data leakage, where the test set's statistics influence the training process, leading to overly optimistic performance estimates.
- Scikit-learn's fit_transform() makes it easy to accidentally normalize everything at once.
Major section
Renormalize for New Sensors
Adding a new sensor channel requires fitting and storing scaling parameters for that channel on training data.
- Existing channels can retain their fitted parameters unless their own data or feature definitions change.
Major section
Summary
Catching data quality issues at the source can avoid downstream repair, and normalized data enables fair multi-sensor fusion.
- Critical Design Principle: The complete "validate-clean-transform" pipeline should run at the edge.
Major section
Concept Relationships
Data Validation and Outlier Detection: Range checks and outlier detection must occur before normalization to avoid skewing scaling parameters with invalid data.
- Multi-Sensor Data Fusion: Normalization enables fair comparison when fusing sensors with vastly different measurement ranges (temperature vs light intensity).
- Edge Data Acquisition: Edge devices normalize locally to reduce transmission bandwidth and prepare data for edge ML inference.
Major section
Concept Relationships (continued)
Missing Value Imputation and Noise Filtering: Cleaning must precede normalization; otherwise, gaps and noise corrupt min/max or mean/std calculations.
- Anomaly Detection: Z-score normalization makes distance-based anomaly detection work across features with different scales.
- Key Insight: Normalization is the final stage of the validate-clean-transform pipeline.
- Applying it before validation or cleaning causes incorrect scaling parameters (e.g., min-max uses outlier as max value, compressing all valid data into a tiny range).
Deck summary
Key takeaways
Task: The code defines a MAD_THRESHOLD constant but does not implement MAD outlier detection.
- If it is very bright (high light), temperature should be reasonable for daytime.
- A neural network needs all inputs on the same scale to detect abnormal conditions.
- The Mistake: Calculating normalization parameters (min, max, mean, std) on the entire dataset before splitting into train and test sets.
- Adding a new sensor channel requires fitting and storing scaling parameters for that channel on training data.
Retrieval practice
Recall check 1 of 2
Data Dora says: answer from memory, then check your reasoning.
Q1What training-set statistics must be versioned as model inputs for later scaling?
Show answer
Answer: D Training-set minima, maxima, means, and standard deviations are versioned model inputs, not disposable calculations.
Q2Complete the sensor data quality pipeline:
Show answer
Answer: A Sensor data cleaning validates physical limits while preserving missing markers, then interpolates the gaps and smooths with an exponential moving average.
Retrieval practice
Recall check 2 of 2
Data Dora says: answer from memory, then check your reasoning.
Q3A scaler is being prepared for a training set and a held-out summer test set. When should its min, max, mean, or spread be fitted?
Show answer
Answer: B Split first, fit scaling parameters only on training data, and transform held-out data with that fitted scaler.
Print reference
Answers
Answer key.
- D · Training-set minima, maxima, means, and standard deviations are versioned model inputs, not disposable calculations.
- A · Sensor data cleaning validates physical limits while preserving missing markers, then interpolates the gaps and smooths with an exponential moving average.
- B · Split first, fit scaling parameters only on training data, and transform held-out data with that fitted scaler.