Analytics & ML · Study deck

Feature Scaling Lab: Preprocessing Workflow

Start with one saved copy of the raw data.

Data Dora is your guide for this deck.

dataqualitynormalization
Data Dora, the module guide, in a scene from this chapter.
iotclass.org

After studying this chapter

Learning objectives

You will be able to:

  • Apply Normalization Techniques: Implement min-max scaling, Z-score normalization, and robust scaling for multi-sensor data fusion
  • Compare Normalization Methods: Evaluate which scaling approach suits each downstream use case (neural networks, clustering, visualization)
  • Implement a Complete Pipeline: Build and test an end-to-end data quality system on an ESP32 microcontroller
  • Assess Data Quality Metrics: Calculate and interpret validation rates, outlier counts, and imputation statistics
iotclass.org

Major section

Start With the Story

The values use different units and ranges.

  • If the team scales all records before it separates training data from test data, the test set quietly shapes the model.
  • Scaling cannot repair a bad sensor or prove that an alert is useful.
  • It only changes how values enter the next step.
iotclass.org

Major section

In 60 Seconds

You will compare Min-Max normalisation, Z-score standardisation, and robust scaling on data with outliers, and see the impact on a simple classifier.

  • The mathematical gist: The simulator's 10000 × (4095/adc: 1) is the voltage-divider equation solved backward.

This hands-on lab guides you through implementing data normalisation and standardisation on real IoT sensor datasets, demonstrating concretely how improper scaling causes ML models to ignore low-magnitude sensors and how to prevent it.

iotclass.org

Major section

For Beginners: Why Normalize Data?

For distance-based models, the larger lux numbers can dominate a distance calculation unless the features are scaled.

  • Key question this chapter answers: "How do I prepare multi-sensor data so it can be combined and analyzed fairly?".
Min-Max

Need the fitted training range mapped to 0-1 (neural networks)

Training range 0 to 1

Z-Score

Scale-sensitive methods such as K-means/SVM/PCA

Mean=0, StdDev=1

Robust

Many outliers you want to ignore

Median-centered

iotclass.org

Major section

Key Concepts

Min-Max normalisation: Scaling the fitted training range to [0, 1] by subtracting its minimum and dividing by its range; later values outside that range can transform below 0 or above 1.

  • Feature scaling: Transforming sensor channels to comparable numeric ranges so scale-sensitive methods are not dominated solely by engineering units.

- Normalisation artefact: A distortion introduced by incorrect normalisation — for example, scaling based on training set statistics applied incorrectly to the test set, or using the wrong normalisation for the downstream algorithm.

iotclass.org

Major section

Lab Overview

In this hands-on lab, you will implement a complete data quality preprocessing pipeline on an ESP32 microcontroller.

  • The simulation demonstrates real-world techniques for handling sensor data problems including outliers, missing values, noise, and the need for normalization.
  • Sensor Data Validation: Implementing range checks and rate-of-change validation.

Try it: Lab Overview in the chapter

iotclass.org

Major section

Step-by-Step Instructions

The Wokwi Serial Monitor is the source of evidence for the ESP32 sketch.

  • The local Python calculation below checks only scaling arithmetic; it is not a Wokwi run or a substitute for the monitor trace.

Record your own Wokwi readings when you run the browser simulation.

iotclass.org

Major section

Step 3: Trigger Range Violations

In the Wokwi sensor control, move the input beyond the sketch's valid range or change it fast enough to exceed its rate limit.

  • If the available sensor control cannot reach the configured limit, record that limitation instead of inventing a flag.

- In the Wokwi sensor control, move the input beyond the sketch's valid range or change it fast enough to exceed its rate limit.

iotclass.org

Major section

Step 5: Observe Missing Data Handling

Leave the Wokwi simulation running; the sketch's missing-data branch is random, so a short trace may show none.

  • If consecutive missing samples exceed the sketch's limit, record MISSING_NO_IMPUTE and the absent normalized value.

- Leave the Wokwi simulation running; the sketch's missing-data branch is random, so a short trace may show none.

iotclass.org

Deck summary

Key takeaways

The values use different units and ranges.

  • You will compare Min-Max normalisation, Z-score standardisation, and robust scaling on data with outliers, and see the impact on a simple classifier.
  • For distance-based models, the larger lux numbers can dominate a distance calculation unless the features are scaled.
  • Min-Max normalisation: Scaling the fitted training range to [0, 1] by subtracting its minimum and dividing by its range; later values outside that range can transform below 0 or above 1.
  • In this hands-on lab, you will implement a complete data quality preprocessing pipeline on an ESP32 microcontroller.
iotclass.org

Retrieval practice

Recall check

Data Dora says: answer from memory, then check your reasoning.

Q1A distance-based model combines humidity with a much larger-magnitude vibration feature, so scaling is required. Humidity readings [45,48,50,52,55,99,47,51,49,50] include a confirmed stuck-sensor outlier of 99. Which scaling method is least distorted by it?

ARobust scaling, which centers on the median and scales by the IQR rather than the mean
BMin-max scaling, because it stretches the data to a fixed 0-to-1 range
CZ-score standardization, since subtracting the mean removes any outlier effect
DNo scaling at all, because raw humidity values are already comparable
Show answer

Answer: A Robust scaling uses median and IQR, so a single stuck-sensor outlier barely affects the transformed values.

iotclass.org

Print reference

Answers

Answer key.

  1. A · Robust scaling uses median and IQR, so a single stuck-sensor outlier barely affects the transformed values.
iotclass.org