Analytics & ML · Study deck
Missing-Data Repair Filter Contracts
Picture an IoT team using the ideas in Missing-Data Repair Filter Contracts during a live operations review.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain why the cause of missing data changes whether imputation is safe.
- Choose forward-fill, interpolation, model-based repair, median filtering, or Hampel filtering based on sensor semantics.
- Preserve quality flags, gap metadata, and repair methods alongside cleaned values.
- Distinguish MCAR, MAR, and MNAR missingness when approving automated repairs.
Major section
Gaps and Spikes as Data Problems
Real sensor streams have holes (dropped packets, sleeping nodes, failed reads) and spikes (impulse noise, glitches).
- Imputation fills the holes with plausible estimates; filtering removes the spikes.
- The single most important idea is that how data went missing changes whether filling it is safe.
- Filling it in can delete the very event you care about.
Major section
Gaps and Spikes as Data Problems (continued)
Noise has the same trap.
- If a sensor fails precisely when values get extreme, naively filling the gap with an average erases exactly the information you needed and biases every downstream result.
- Intuition: a missing value is not always a random blank to paint over.
- A good cleaned dataset also keeps provenance.
Major section
Gaps and Spikes as Data Problems (continued)
The output explicitly includes a quality flag, repair provenance, and decision check, keeping later analytics honest about what was measured, inferred, or deliberately left missing.
- If five samples vanish during a Wi-Fi retry, a short forward-fill or interpolation may preserve enough continuity for a dashboard.
- If the same five samples vanish every time the freezer door opens and the transmitter browns out, the gap is not a harmless hole; it is evidence that the physical process exceeded a boundary.
- Treating both gaps identically would make the chart look smoother while making the inference worse.
Major section
Imputation and Impulse Filters
For a slow temperature probe, a two-minute gap might be forward-filled and marked.
- For a binary door or motion event, interpolation is usually wrong because there is no meaningful value halfway between closed and open.
- Dashboards often need continuity, so short forward-fill or interpolation can be enough.
- Those thresholds are not universal constants.
Major section
Under the Hood: MCAR, MAR, MNAR
Model-based imputation using those variables can fill it with little bias.
- MNAR means the missingness depends on the unobserved value itself, such as a pressure sensor that saturates and stops reporting exactly when pressure is high.
- In a MAR case, hide samples only when the gateway queue length is high; a model that includes queue length and neighbouring sensors can still recover useful estimates.
- So imputation and filtering are not mechanical clean-up.
Major section
Under the Hood: MCAR, MAR, MNAR (continued)
In an MNAR case, hide samples only when temperature exceeds 39 degrees C; interpolation from normal neighbouring points systematically replaces overheating evidence with ordinary values.
- A downstream alarm can choose to fire on long MNAR-suspect gaps, while a trend dashboard can show a dashed segment for imputed values.
- The pipeline stays honest because uncertainty is carried forward instead of being hidden behind a neat number.
- If the cleaner reduces variance, removes rare extremes, or shifts event timing, there must be a physical explanation.
- Without one, keep the raw value, mark the quality issue, and investigate the sensor or transport path.
Major section
Under the Hood: MCAR, MAR, MNAR (continued)
MCAR Missing completely at random: the gap is unrelated to any value (a random dropped packet).
- MAR Missing at random: missingness depends on other observed variables (a sensor that fails when the gateway is busy).
- MNAR Missing not at random: missingness depends on the missing value itself (a sensor that saturates and drops out at extremes).
- A moving average assumes neighbouring samples should look similar and reduces high-frequency variation, which is helpful for Gaussian measurement noise but harmful for sharp transitions.
- The statistical name for the gap mechanism matters because it states what information is allowed to explain the missing sample.
Deck summary
Key takeaways
Real sensor streams have holes (dropped packets, sleeping nodes, failed reads) and spikes (impulse noise, glitches).
- Noise has the same trap.
- The output explicitly includes a quality flag, repair provenance, and decision check, keeping later analytics honest about what was measured, inferred, or deliberately left missing.
- For a slow temperature probe, a two-minute gap might be forward-filled and marked.
- Model-based imputation using those variables can fill it with little bias.
Retrieval practice
Recall check 1 of 3
Data Dora says: answer from memory, then check your reasoning.
Q1Why does the REASON data is missing matter before choosing an imputation method?
Show answer
Answer: A A value-dependent outage can be the evidence, so a naive fill can hide the event instead of repairing the record.
Retrieval practice
Recall check 2 of 3
Data Dora says: answer from memory, then check your reasoning.
Q2Why is a median filter better than a mean filter for removing occasional impulse spikes?
Show answer
Answer: A A lone spike does not move the median much, while the mean is dragged toward the extreme value.
Retrieval practice
Recall check 3 of 3
Data Dora says: answer from memory, then check your reasoning.
Q3A sensor saturates and stops reporting whenever the true value is very high (missing not at random). Why is mean imputation especially harmful here?
Show answer
Answer: A Replacing those hidden highs with an average removes the extremes and biases statistics and models downward.
Print reference
Answers
Answer key.
- A · A value-dependent outage can be the evidence, so a naive fill can hide the event instead of repairing the record.
- A · A lone spike does not move the median much, while the mean is dragged toward the extreme value.
- A · Replacing those hidden highs with an average removes the extremes and biases statistics and models downward.