Analytics & ML · Study deck
Time-Series Anomalies: Baselines and Residuals
A freezer can warm during defrost and still be healthy.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: A time-aware detector compares the reading with the expected value for that minute, operating mode, and recent history, then stores the residual and context that made the decision reviewable.
- Explain: This makes time-series methods a natural fit for contextual anomalies in IoT: load profiles, temperature cycles, vibration windows, occupancy patterns, battery discharge, and watermarks in stream processing.
- Explain: That order advances the chapter's evidence chain: preserve the context, forecast or decomposition, residual, and decision threshold together so two identical readings can legitimately receive different decisions.
- choose a temporal baseline that preserves seasonality and drift
Major section
Start With the Story
A baseline is the normal pattern used for comparison.
- An anomaly is a reading that does not fit that pattern.
- A detector compares new readings with the baseline; operating state tells it what the machine is doing.
- A fixed alarm line may flag every warm afternoon, even though the pattern is normal.
Major section
Start With the Story (continued)
The useful question is whether today differs from the pattern expected at this time.
- An unusual value is not proof of a fault.
- A normal-looking value is not proof of health if the baseline or sampling has changed.
- Under the Hood shows how sample rate, delay, and drift can change the result.
Major section
Time-Series Residual Scoring
A value can be normal at one hour, season, operating mode, or cycle stage and abnormal at another.
- The usual pattern is to forecast or decompose the expected value, compute a residual, and alert only when the residual is large enough to matter.
- A raw threshold cannot tell those cases apart.
- This makes time-series methods a natural fit for contextual anomalies in IoT: load profiles, temperature cycles, vibration windows, occupancy patterns, battery discharge, and watermarks in stream processing.
- The detector must record the forecast window, seasonal context, residual, threshold, missing-data handling, and latency budget.
Major section
Time-Series Residual Scoring (continued)
ARIMA-style forecasting models trend and repeating cycles, exponential smoothing adapts the recent level, and STL separates trend and seasonality explicitly.
- That order advances the chapter's evidence chain: preserve the context, forecast or decomposition, residual, and decision threshold together so two identical readings can legitimately receive different decisions.
- Worked example: a freezer temperature of 6 deg C may be normal during a scheduled defrost cycle and abnormal during a steady cooling period.
- A time-aware detector compares the reading with the expected value for that minute, operating mode, and recent history, then stores the residual and context that made the decision reviewable.
Major section
Choose Temporal Baseline First
If the signal has daily or weekly cycles, use a seasonal baseline or STL residual.
- If it changes slowly but has sudden level shifts, use EWMA or Holt-Winters-style smoothing.
- If anomalies are window shapes, score windows rather than single points.
- Missing timestamps, daylight-saving changes, holiday modes, and new schedules.
Major section
Choose Temporal Baseline First (continued)
If samples arrive every minute but the operations team can wait five minutes, a short persistence rule can suppress one-off meter noise.
- If a motor trip must be caught within one sample, use an edge EWMA or threshold residual and send a richer window to the gateway for review.
- The design should state which layer owns the first alert and which layer owns the later explanation.
- Non-stationary drift, abrupt maintenance changes, and unstable sampling intervals.
Major section
Sampling, Drift, and Latency
Drift control is equally important.
- Irregular sampling, delayed packets, clock drift, duplicated readings, and gaps change the residual.
- Stream deployments also need explicit watermarks or late-data policies: a detector cannot safely score a window if the window may still receive late readings.
- Seasonal baselines become stale when equipment, firmware, occupancy, weather, or operating policy changes.
Major section
Sampling, Drift, and Latency (continued)
The alert record should say whether the data was complete, late, imputed, or scored in degraded mode.
- Production systems should separate drift monitoring from incident scoring and require baseline review after known site changes.
- The chosen policy changes both alert quality and response time, so it belongs in the incident record.
- Gap handling needs the same precision.
Major section
Sampling, Drift, and Latency (continued)
Reviewers can then distinguish a true contextual anomaly from an artifact of timing or missing evidence.
- Carrying the last value forward through six missing samples may hide a pump shutdown, while dropping the window may hide a short overload.
- Latency budgets should be checked against the method.
- Temporal context is missing or the seasonal period is wrong.
Major section
Sampling, Drift, and Latency (continued)
WatermarkDefines when a time window is complete enough to score despite late or out-of-order events.
- A robust implementation records the gap count, imputation method, clock source, model version, and baseline id beside the residual.
- A 30-minute STL window cannot support a two-minute safety response unless an edge rule raises a provisional alert first.
- Gap PolicyRecords whether missing samples are dropped, imputed, held, or treated as sensor-health evidence.
Major section
Sampling, Drift, and Latency (continued)
Conversely, a one-sample edge residual may be fast but weak, so the gateway or cloud path should attach the later context review to the same incident id instead of creating a second, disconnected alert.
- Baseline ReviewTriggers retesting after maintenance, firmware updates, schedule changes, or sustained drift.
- A partial window looks abnormal, then becomes normal after delayed readings arrive.
- A slow degradation stops alerting after the adaptive baseline catches up.
Deck summary
Key takeaways
A baseline is the normal pattern used for comparison.
- The useful question is whether today differs from the pattern expected at this time.
- A value can be normal at one hour, season, operating mode, or cycle stage and abnormal at another.
- ARIMA-style forecasting models trend and repeating cycles, exponential smoothing adapts the recent level, and STL separates trend and seasonality explicitly.
- If the signal has daily or weekly cycles, use a seasonal baseline or STL residual.
Retrieval practice
Recall check 1 of 4
Data Dora says: answer from memory, then check your reasoning.
Q1Why should a seasonal IoT signal usually be scored on residuals instead of raw readings?
Show answer
Answer: B Time-series anomaly detection compares observations with temporal expectations.
Retrieval practice
Recall check 2 of 4
Data Dora says: answer from memory, then check your reasoning.
Q2The matching seasonal baseline is 18 kW and the observed load is 31 kW. The alert band is ±4 kW, but the reading arrived 2 minutes after a 1-minute action deadline. What is the defensible result?
Show answer
Answer: A 31 kW − 18 kW = 13 kW, above ±4 kW; 2 minutes is later than the 1-minute deadline.
Retrieval practice
Recall check 3 of 4
Data Dora says: answer from memory, then check your reasoning.
Q3A site has strong weekday/weekend and hour-of-day load cycles. What is the safest first detector?
Show answer
Answer: C Seasonal and contextual signals should be scored against the matching temporal baseline.
Retrieval practice
Recall check 4 of 4
Data Dora says: answer from memory, then check your reasoning.
Q4Why does a streaming time-series detector need a late-data or watermark policy?
Show answer
Answer: A Time-series anomaly decisions depend on whether the relevant window and context are complete.
Print reference
Answers
Answer key.
- B · Time-series anomaly detection compares observations with temporal expectations.
- A · 31 kW − 18 kW = 13 kW, above ±4 kW; 2 minutes is later than the 1-minute deadline.
- C · Seasonal and contextual signals should be scored against the matching temporal baseline.
- A · Time-series anomaly decisions depend on whether the relevant window and context are complete.