Analytics & ML · Study deck
Time-Series Anomalies: Statistical Methods
A cold-room trace can jump for one sample or drift for an hour.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: A normal-data false-positive rate of about 0.27 percent is still roughly 1,440,000 x 0.0027 = 3,888 flagged normal samples per day before persistence, grouping, and operator triage.
- Explain: The detector should switch to a sensor-health alert or domain-bound check, keep the zero-variance state with the incident, and avoid presenting the result as a normal statistical anomaly.
- Explain: Worked example: a cold-room sensor may have 200 recent normal readings with a stable mean and spread, so a z-score check is cheap enough to run on the gateway.
Major section
Statistical Anomaly Methods
The value may mark a door left open, a defrost cycle, a moved probe, or a faulty sensor.
- A high score only says the pattern differs from the chosen reference.
- This review cannot prove the cause from the score alone.
- It makes the limit visible.
- The deeper sections explain thresholds, robust summaries, windows, and change tests.
Major section
Statistical Anomaly Methods (continued)
Z-score, standard control limits, or EWMA residual.
- A device has produced messy evidence, an analytic step is about to change an alert or control decision, and someone has to explain why the result should be trusted.
- Statistical anomaly methods score how far a new reading is from a reference baseline.
- Statistical detectors also fail in predictable ways.
Major section
Statistical Anomaly Methods (continued)
A statistical alert should preserve the baseline window, statistic, threshold, score, sample quality, missing-data handling, and sensor-health state.
- The tradeoff is that each method carries an assumption about the data: symmetry, spread, sample window, stationarity, or context.
- Without that evidence, the alert cannot be tuned or defended.
- A tiny baseline window gives unstable estimates.
Major section
Statistical Anomaly Methods (continued)
The alert rule decides whether the unusual point is operationally important.
- If normal behavior depends on time, season, operating mode, or related sensors, the statistical score must be computed against the matching context or residual, not against a global baseline.
- The important step is to record why the statistic was chosen.
- Adaptive baselines need their own guardrail.
Major section
Statistical Anomaly Methods (continued)
Small baseline windows, zero variance, unit changes, and sensor recalibration.
- This connects the method map to the running narrative: an explainable score is only defensible when its baseline window, assumptions, threshold, and sensor-health evidence travel with the alert.
- Worked example: a cold-room sensor may have 200 recent normal readings with a stable mean and spread, so a z-score check is cheap enough to run on the gateway.
- A zero-variance window makes z-score undefined.
Major section
Statistical Anomaly Methods (continued)
IQR or MAD Uses robust spread estimates so skewed data or existing outliers do not dominate the threshold.
- A battery-voltage signal from an energy-harvesting node may be bounded and skewed, so an IQR fence is easier to defend.
- A pump current signal that changes after maintenance may need an adaptive baseline with a holdout review before new limits become trusted.
- EWMA Smooths recent values to track gradual changes while still exposing sudden departures.
Major section
Statistical Anomaly Methods (continued)
A later reviewer should be able to see whether the baseline was local to the asset, shared across a fleet, segmented by operating mode, or rebuilt after drift.
- Control Limits Monitors a process against expected variation and investigation thresholds, often with persistence rules.
- If the signal is stable and roughly symmetric, a z-score or control limit can be enough.
- If the signal is bounded, skewed, or already contains outliers, use IQR or median absolute deviation.
Major section
Statistical Anomaly Methods (continued)
If normal drifts slowly, use a rolling baseline, EWMA, or residual from a context model.
- If the baseline has a few old excursions, compare the z-score decision with an IQR or MAD decision and store both scores during review.
- IQR, median absolute deviation, percentile fences, or domain-specific bounds.
- The fleet-scale arithmetic is the usual surprise.
Major section
Statistical Anomaly Methods (continued)
Normal drift being learned too fast and real change being absorbed.
- A threshold is a cost decision, not just a formula.
- That may be acceptable for one sensor sampled hourly and unacceptable for a fleet of thousands sampled every second.
- False alerts, missed events, latency, and investigation capacity must all shape the threshold.
Major section
Statistical Anomaly Methods (continued)
Fast adaptation can learn an emerging fault as normal.
- Missing or stale samples can make a process look stable when the sensor is failing.
- Production code should handle these cases explicitly and report them as evidence.
- Guard cases should be explicit in the implementation.
- During a pump bearing fault, vibration may rise gradually for several hours.
Major section
Statistical Anomaly Methods (continued)
Adaptive update rate is too fast or has no holdout review.
- A normal-data false-positive rate of about 0.27 percent is still roughly 1,440,000 x 0.0027 = 3,888 flagged normal samples per day before persistence, grouping, and operator triage.
- If a baseline window contains 30 identical values after a sensor freeze, the standard deviation is zero and a z-score cannot be computed honestly.
- Health checks are missing or run after alert escalation.
Major section
Statistical Anomaly Methods (continued)
The detector should switch to a sensor-health alert or domain-bound check, keep the zero-variance state with the incident, and avoid presenting the result as a normal statistical anomaly.
- A safer design freezes updates during open incidents, stores the held baseline id, and resumes learning only after a reviewed disposition.
- Warm-Up Do not alert from a baseline until the sample count and coverage are sufficient for the selected statistic.
- Variance Guard If spread is zero or near zero, use domain bounds, robust fences, or a sensor-health state instead of dividing by zero.
Major section
Statistical Anomaly Methods (continued)
Multiple Sensors Fleet-scale alerts multiply false-positive burden, so thresholds need fleet-level review, not only per-sensor math.
- Retest Trigger Maintenance, firmware changes, placement changes, and seasonal shifts should trigger baseline review.
- Statistical anomaly methods are the first tools to try when an IoT alert can be explained by a baseline, spread estimate, score, and threshold.
- The method is only trustworthy when its baseline, assumptions, sensor-health checks, threshold, and alert burden are visible.
Deck summary
Key takeaways
The value may mark a door left open, a defrost cycle, a moved probe, or a faulty sensor.
- Z-score, standard control limits, or EWMA residual.
- A statistical alert should preserve the baseline window, statistic, threshold, score, sample quality, missing-data handling, and sensor-health state.
- The alert rule decides whether the unusual point is operationally important.
- Small baseline windows, zero variance, unit changes, and sensor recalibration.
Retrieval practice
Recall check 1 of 3
Data Dora says: answer from memory, then check your reasoning.
Q1What evidence should a statistical IoT anomaly alert keep for later review?
Show answer
Answer: C Statistical methods are useful because their evidence can be recorded and reviewed.
Retrieval practice
Recall check 2 of 3
Data Dora says: answer from memory, then check your reasoning.
Q2A battery-voltage stream is bounded and skewed, with occasional low dips during cloudy periods. Which statistical method is usually a better first choice than a plain z-score?
Show answer
Answer: A The statistic should match the distribution shape and operational context.
Retrieval practice
Recall check 3 of 3
Data Dora says: answer from memory, then check your reasoning.
Q3Why can a mathematically valid 3-sigma threshold still be operationally poor for an IoT fleet?
Show answer
Answer: D Statistical thresholds must be validated against both assumptions and operational costs.
Print reference
Answers
Answer key.
- C · Statistical methods are useful because their evidence can be recorded and reviewed.
- A · The statistic should match the distribution shape and operational context.
- D · Statistical thresholds must be validated against both assumptions and operational costs.