Analytics & ML · Study deck
Signal Preprocessing Contracts
Picture three vibration sensors watching the same motor.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain why preprocessing order changes the statistical meaning of a sensor stream.
- Choose alignment, resampling, smoothing, and detrending steps based on the physical signal you must preserve.
- Identify when causal filters, centered filters, interpolation, or downsampling can mislead real-time IoT decisions.
- Record preprocessing parameters and quality flags so dashboards, models, and incident reviews can reproduce the cleaned stream.
Major section
Start With the Story
Their clocks differ, one misses samples, and one reports a brief spike.
- The analyst can make a smooth chart, but the maintenance lead needs to know whether the motor changed or the cleaning steps invented the change.
- Shift one clock, remove a short run, add a real step change, repeat a reading, and place a spike at the edge of a window.
- This simple record does not select the best mathematical method.
Major section
Raw Data Needs Preprocessing
Between validated raw readings and a model that can learn from them sits preprocessing: aligning sensors that sample at different rates, filling or marking gaps, smoothing noise, and removing trends.
- Preprocessing is a workflow, not a single step, and the order of operations matters.
- Getting the sequence right is as important as choosing the techniques.
Major section
Raw Data Needs Preprocessing (continued)
For example, a cold-room monitor might collect temperature every 10 seconds, door state every event, and compressor current once a minute.
- Intuition: preprocessing is preparing ingredients before cooking.
- You wash and chop in a sensible order — and if you blend before removing the stone, the whole dish is ruined no matter how good the recipe.
- The important distinction is that preprocessing preserves evidence while making the signal usable.
Major section
Raw Data Needs Preprocessing (continued)
The sequence carries the chapter's narrative forward: every stage needs an input contract, a versioned action, and an output-quality record so downstream models can distinguish measured data from repaired or derived evidence.
- A removed point should leave a quality flag; an imputed value should be distinguishable from a measured value; a resampled series should record the original sampling interval.
- Those records let later troubleshooting answer whether an alarm came from the physical process or from the cleaning choice.
- That discipline gives downstream analytics a stable contract instead of a moving target.
Major section
Align, Smooth, Resample
The conversion might be mean vibration energy, maximum temperature, door-open seconds, and the count of validation failures.
- A practical preprocessing run starts by writing down the target analysis grain.
- Without that target grain, teams often resample blindly and create features that cannot be compared.
- The filter is part of the measurement design, not cosmetic cleanup.
Major section
Align, Smooth, Resample (continued)
If the model makes one prediction per minute, a 100 Hz vibration stream, a 1 Hz temperature stream, and event-driven door contacts all need rules that convert them to that one-minute grain.
- For those cases, a median filter may remove one-sample spikes, or a Savitzky-Golay filter may reduce noise while preserving peak shape.
- Savitzky-Golay (window 11, order 3): fits a cubic in each window, so it removes noise while keeping the peak's height and the slopes on either side.
- Each step assumes the previous one has already run.
Major section
How Preprocessing Misleads
Preprocessing changes the statistical meaning of the data.
- A moving average reduces variance, interpolation creates synthetic samples, detrending removes low-frequency content, and downsampling discards information above the new sampling limit.
- The timing model is just as important as the math.
- Aliasing is the other hidden trap.
- Downsample only after filtering out content the new sample rate cannot represent.
Major section
How Preprocessing Misleads (continued)
Over-smoothing hides signal Too wide a window erases the very features you are trying to detect.
- A live edge alarm cannot use future samples without delaying the alert or leaking future information into the decision.
- The pipeline may appear smoother and cheaper, while the feature has become false evidence.
- Neither creates information that was not sampled.
Major section
How Preprocessing Misleads (continued)
If a filter needs five future samples to look clean in a notebook, it is not the same filter that can run in production at the event boundary.
- If a 100 Hz vibration component is downsampled to 20 Hz without a low-pass filter, high-frequency energy folds into lower frequencies and can look like a real slow oscillation.
- Changing an imputation window, a smoothing alpha, or a normalization range can shift model inputs enough to invalidate comparisons with previous metrics.
- The window should be matched to the timescale of the phenomenon, not set to "as smooth as possible".
Major section
How Preprocessing Misleads (continued)
That record also lets teams rerun old raw data through the new pipeline and measure whether the change improved the signal or only moved the baseline.
- For real-time alarms that lag can be the difference between catching an event and missing its onset; account for it or use a zero-phase filter offline.
- Resampling is not free Upsampling invents points by interpolation (which can imply detail that was never measured); downsampling must respect the sampling limit to avoid aliasing.
- Retain the validated raw data so you can reprocess with different parameters instead of being stuck with one irreversible pipeline.
Deck summary
Key takeaways
Their clocks differ, one misses samples, and one reports a brief spike.
- Between validated raw readings and a model that can learn from them sits preprocessing: aligning sensors that sample at different rates, filling or marking gaps, smoothing noise, and removing trends.
- For example, a cold-room monitor might collect temperature every 10 seconds, door state every event, and compressor current once a minute.
- The sequence carries the chapter's narrative forward: every stage needs an input contract, a versioned action, and an output-quality record so downstream models can distinguish measured data from repaired or derived evidence.
- Preprocessing changes the statistical meaning of the data.
Retrieval practice
Recall check 1 of 3
Data Dora says: answer from memory, then check your reasoning.
Q1Why does the ORDER of preprocessing steps matter?
Show answer
Answer: B Preprocessing steps interact, so sequence changes the result.
Retrieval practice
Recall check 2 of 3
Data Dora says: answer from memory, then check your reasoning.
Q2You must smooth noise but preserve the height and shape of genuine sharp peaks. Which filter is better suited?
Show answer
Answer: C Savitzky-Golay preserves peak shape and derivatives that a moving average flattens.
Retrieval practice
Recall check 3 of 3
Data Dora says: answer from memory, then check your reasoning.
Q3A real-time alarm reacts late after a wide moving-average smoother was added. Why?
Show answer
Answer: A Smoothing introduces lag proportional to window width, delaying real-time detection.
Print reference
Answers
Answer key.
- B · Preprocessing steps interact, so sequence changes the result.
- C · Savitzky-Golay preserves peak shape and derivatives that a moving average flattens.
- A · Smoothing introduces lag proportional to window width, delaying real-time detection.