40 Time-Series Anomalies: Statistical Methods

40.5.2 Statistical Baseline Detectors

Statistical anomaly methods score how far a new reading is from a reference baseline. They are useful for IoT because they are explainable, cheap to run, and easy to review. The tradeoff is that each method carries an assumption about the data: symmetry, spread, sample window, stationarity, or context.

Use statistical methods first for point anomalies and simple residual checks. Do not treat them as universal anomaly detection. If normal behavior depends on time, season, operating mode, or related sensors, the statistical score must be computed against the matching context or residual, not against a global baseline.

Use the method map in Figure 40.1 before selecting a threshold so the signal's distribution and drift behavior determine the method, rather than familiarity with one statistic.

Read Figure 40.1 from the signal assumption toward the deployment consequence. Z-score follows the stable, roughly Gaussian branch; IQR follows the skewed or outlier-prone branch; adaptive thresholds follow changing baselines and seasonal context. Notice that each branch names both a strength and a condition that can invalidate it. This connects the method map to the running narrative: an explainable score is only defensible when its baseline window, assumptions, threshold, and sensor-health evidence travel with the alert.

Worked example: a cold-room sensor may have 200 recent normal readings with a stable mean and spread, so a z-score check is cheap enough to run on the gateway. A battery-voltage signal from an energy-harvesting node may be bounded and skewed, so an IQR fence is easier to defend. A pump current signal that changes after maintenance may need an adaptive baseline with a holdout review before new limits become trusted.

The important step is to record why the statistic was chosen. A later reviewer should be able to see whether the baseline was local to the asset, shared across a fleet, segmented by operating mode, or rebuilt after drift. Otherwise teams can mistake a method mismatch for a process fault and tune the wrong threshold.

Overview Knowledge Check

40.5.3 Choose Statistics by Signal

Start by plotting or summarizing the recent normal window. If the signal is stable and roughly symmetric, a z-score or control limit can be enough. If the signal is bounded, skewed, or already contains outliers, use IQR or median absolute deviation. If normal drifts slowly, use a rolling baseline, EWMA, or residual from a context model.

Worked example: z-score candidate
baseline mean: 21.0 deg C
baseline standard deviation: 0.5 deg C
new reading: 22.8 deg C

z = (22.8 - 21.0) / 0.5 = 3.6

candidate rule:
flag when abs(z) >= 3.0

review rule:
raise an alert only after persistence, confirmation,
or an operator-approved safety policy.

why this matters:
The score says the point is unusual under this baseline.
The alert rule decides whether the unusual point is operationally important.

Run the same reasoning against the signal shape before choosing the method. If the baseline has a few old excursions, compare the z-score decision with an IQR or MAD decision and store both scores during review. Disagreement is useful evidence: it often points to skew, stale calibration, or a baseline window contaminated by previous incidents.

Practitioner Knowledge Check

40.5.4 Thresholds Encode Costs

A threshold is a cost decision, not just a formula. Under a Gaussian assumption, a two-sided 3-sigma z-score rule flags roughly 0.27 percent of normal samples. That may be acceptable for one sensor sampled hourly and unacceptable for a fleet of thousands sampled every second. False alerts, missed events, latency, and investigation capacity must all shape the threshold.

Statistical detectors also fail in predictable ways. A tiny baseline window gives unstable estimates. A zero-variance window makes z-score undefined. Fast adaptation can learn an emerging fault as normal. Missing or stale samples can make a process look stable when the sensor is failing. Production code should handle these cases explicitly and report them as evidence.

The fleet-scale arithmetic is the usual surprise. If 1,000 sensors produce one scored sample per minute, the fleet produces 1,000 x 1,440 = 1,440,000 scored samples per day. A normal-data false-positive rate of about 0.27 percent is still roughly 1,440,000 x 0.0027 = 3,888 flagged normal samples per day before persistence, grouping, and operator triage. That is why a statistically rare event can still be operationally common.

Guard cases should be explicit in the implementation. If a baseline window contains 30 identical values after a sensor freeze, the standard deviation is zero and a z-score cannot be computed honestly. The detector should switch to a sensor-health alert or domain-bound check, keep the zero-variance state with the incident, and avoid presenting the result as a normal statistical anomaly.

Adaptive baselines need their own guardrail. During a pump bearing fault, vibration may rise gradually for several hours. If the rolling baseline updates every minute with no freeze rule, the abnormal trend becomes the new normal before anyone reviews it. A safer design freezes updates during open incidents, stores the held baseline id, and resumes learning only after a reviewed disposition.

Under-the-Hood Knowledge Check