Analytics & ML · Study deck
Time-Series Anomalies: Statistical Methods
A cold-room trace can jump for one sample or drift for an hour.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Compare z-score, IQR, moving-average, and EWMA anomaly rules.
- Set a threshold from residual evidence and expected false alarms.
- Explain: The value may mark a door left open, a defrost cycle, a moved probe, or a faulty sensor.
- Explain: For each alert, keep the normal period, limit, repeat rule, and gap beside the choice.
Major section
Statistical Anomaly Methods
The value may mark a door left open, a defrost cycle, a moved probe, or a faulty sensor.
- A high score only says the pattern differs from the chosen reference.
- This review cannot prove the cause from the score alone.
- It makes the limit visible.
- The deeper sections explain thresholds, robust summaries, windows, and change tests.
Major section
Statistical Anomaly Methods (continued)
For each alert, keep the normal period, limit, repeat rule, and gap beside the choice.
- Statistical anomaly methods score how far a new reading is from a reference baseline.
- The tradeoff is that each method carries an assumption about the data: symmetry, spread, sample window, stationarity, or context.
- Without that evidence, the alert cannot be tuned or defended.
Major section
Statistical Anomaly Methods (continued)
The alert rule decides whether the unusual point is operationally important.
- If normal behavior depends on time, season, operating mode, or related sensors, the statistical score must be computed against the matching context or residual, not against a global baseline.
- The important step is to record why the statistic was chosen.
- Statistical detectors also fail in predictable ways.
Major section
Statistical Anomaly Methods (continued)
IQR or MADUses robust spread estimates so skewed data or existing outliers do not dominate the threshold.
- A statistical alert should preserve the baseline window, statistic, threshold, score, sample quality, missing-data handling, and sensor-health state.
- EWMASmooths recent values to track gradual changes while still exposing sudden departures.
- A tiny baseline window gives unstable estimates.
Major section
Statistical Anomaly Methods (continued)
Control LimitsMonitors a process against expected variation and investigation thresholds, often with persistence rules.
- This connects the method map to the running narrative: an explainable score is only defensible when its baseline window, assumptions, threshold, and sensor-health evidence travel with the alert.
- Z-score, standard control limits, or EWMA residual.
- Adaptive baselines need their own guardrail.
Major section
Statistical Anomaly Methods (continued)
If the signal is stable and roughly symmetric, a z-score or control limit can be enough.
- Worked example: a cold-room sensor may have 200 recent normal readings with a stable mean and spread, so a z-score check is cheap enough to run on the gateway.
- A battery-voltage signal from an energy-harvesting node may be bounded and skewed, so an IQR fence is easier to defend.
- A zero-variance window makes z-score undefined.
Major section
Statistical Anomaly Methods (continued)
Small baseline windows, zero variance, unit changes, and sensor recalibration.
- A pump current signal that changes after maintenance may need an adaptive baseline with a holdout review before new limits become trusted.
- If the signal is bounded, skewed, or already contains outliers, use IQR or median absolute deviation.
- IQR, median absolute deviation, percentile fences, or domain-specific bounds.
Major section
Statistical Anomaly Methods (continued)
A later reviewer should be able to see whether the baseline was local to the asset, shared across a fleet, segmented by operating mode, or rebuilt after drift.
- If normal drifts slowly, use a rolling baseline, EWMA, or residual from a context model.
- If the baseline has a few old excursions, compare the z-score decision with an IQR or MAD decision and store both scores during review.
- A threshold is a cost decision, not just a formula.
Major section
Statistical Anomaly Methods (continued)
Normal drift being learned too fast and real change being absorbed.
- That may be acceptable for one sensor sampled hourly and unacceptable for a fleet of thousands sampled every second.
- False alerts, missed events, latency, and investigation capacity must all shape the threshold.
- Fast adaptation can learn an emerging fault as normal.
Major section
Statistical Anomaly Methods (continued)
Missing or stale samples can make a process look stable when the sensor is failing.
- Production code should handle these cases explicitly and report them as evidence.
- The fleet-scale arithmetic is the usual surprise.
- Guard cases should be explicit in the implementation.
- During a pump bearing fault, vibration may rise gradually for several hours.
Major section
Statistical Anomaly Methods (continued)
Adaptive update rate is too fast or has no holdout review.
- A normal-data false-positive rate of about 0.27 percent is still roughly 1,440,000 x 0.0027 = 3,888 flagged normal samples per day before persistence, grouping, and operator triage.
- If a baseline window contains 30 identical values after a sensor freeze, the standard deviation is zero and a z-score cannot be computed honestly.
- Health checks are missing or run after alert escalation.
Major section
Statistical Anomaly Methods (continued)
The detector should switch to a sensor-health alert or domain-bound check, keep the zero-variance state with the incident, and avoid presenting the result as a normal statistical anomaly.
- A safer design freezes updates during open incidents, stores the held baseline id, and resumes learning only after a reviewed disposition.
- Warm-UpDo not alert from a baseline until the sample count and coverage are sufficient for the selected statistic.
- Variance GuardIf spread is zero or near zero, use domain bounds, robust fences, or a sensor-health state instead of dividing by zero.
Major section
Statistical Anomaly Methods (continued)
Multiple SensorsFleet-scale alerts multiply false-positive burden, so thresholds need fleet-level review, not only per-sensor math.
- Retest TriggerMaintenance, firmware changes, placement changes, and seasonal shifts should trigger baseline review.
- Statistical anomaly methods are the first tools to try when an IoT alert can be explained by a baseline, spread estimate, score, and threshold.
- The method is only trustworthy when its baseline, assumptions, sensor-health checks, threshold, and alert burden are visible.
Deck summary
Key takeaways
The value may mark a door left open, a defrost cycle, a moved probe, or a faulty sensor.
- For each alert, keep the normal period, limit, repeat rule, and gap beside the choice.
- The alert rule decides whether the unusual point is operationally important.
- IQR or MADUses robust spread estimates so skewed data or existing outliers do not dominate the threshold.
- Control LimitsMonitors a process against expected variation and investigation thresholds, often with persistence rules.
Retrieval practice
Recall check 1 of 3
Data Dora says: answer from memory, then check your reasoning.
Q1What evidence should a statistical IoT anomaly alert keep for later review?
Show answer
Answer: D Statistical methods are useful because their evidence can be recorded and reviewed.
Retrieval practice
Recall check 2 of 3
Data Dora says: answer from memory, then check your reasoning.
Q2A battery-voltage stream is bounded and skewed, with occasional low dips during cloudy periods. Which statistical method is usually a better first choice than a plain z-score?
Show answer
Answer: B The statistic should match the distribution shape and operational context.
Retrieval practice
Recall check 3 of 3
Data Dora says: answer from memory, then check your reasoning.
Q3Why can a mathematically valid 3-sigma threshold still be operationally poor for an IoT fleet?
Show answer
Answer: A Statistical thresholds must be validated against both assumptions and operational costs.
Print reference
Answers
Answer key.
- D · Statistical methods are useful because their evidence can be recorded and reviewed.
- B · The statistic should match the distribution shape and operational context.
- A · Statistical thresholds must be validated against both assumptions and operational costs.