42 Time-Series Anomalies: Evaluation and Types
42.1 Start With the Decision
A detector may catch nine faults and still waste the operator’s day with false alarms. Confusion counts and anomaly type show whether the model is useful.
42.2 Route Overview
This is part 4 of 5. Review Time-Series Anomalies: Machine Learning for the preceding evidence.
42.3 Learning Objectives
- Compute precision, recall, and F1 from detector confusion counts.
- Distinguish point, contextual, and collective anomalies.
42.4 Chapter Roadmap
- Anomaly Detector Evaluation
- Point, Contextual, Collective Anomalies
42.5 Anomaly Detector Evaluation
42.5.1 Start With the Story
Count the Work and the Misses Behind the Score
Picture a cold-store detector that raises 40 alerts in one week. Twelve point to real faults, while eight real fault windows are missed. Latency means the time from enough evidence existing to the reviewed action that matters. One high accuracy number can hide both the false work and the late or missed cases.
Write the review set before tuning. Name the event, normal window, label source, detector version, threshold, time window, alert owner, review outcome, action deadline, and cost of a false or missed alert.
Test a quiet week, a busy week, a changed season, a sensor fault, a late label, repeated windows from one incident, and a threshold change. Show counts and rates together. Check alert age and the final operator choice. A better score is not proof of a safer or workable alert service.
Keep urgent cold-store protection local when analysis is late. The detector can add warning and review evidence, but uncertainty should not delay a known safe step.
This opening does not choose one metric or threshold. Practitioner balances alert load and missed harm. Under the Hood examines labels, windows, imbalance, delay, drift, grouping, confidence, and why the same score can hide different field results.
Picture an IoT team using the ideas in Anomaly Detector Evaluation during a live operations review. A device has produced messy evidence, an analytic step is about to change an alert or control decision, and someone has to explain why the result should be trusted.
Read this page as that path from sensor evidence to accountable action. Start with what the system observes, keep the model or data treatment visible, and finish with the check that would convince an operator, maintainer, or auditor to act.
42.5.2 Alert Quality Metrics
Anomaly detection is usually a rare-event problem. A sensor fleet may produce thousands of normal windows for every real incident, so ordinary accuracy can look excellent even when the detector misses the events that matter. Evaluation must ask two operational questions: how many alerts were real, and how many real incidents were caught.
The core evidence is the confusion matrix. True positives are real incidents that were alerted. False positives are alerts on normal behavior. False negatives are missed incidents. True negatives are normal windows left alone. Precision, recall, F1, false positive rate, detection latency, and review outcome all come from that evidence.
In IoT operations, a metric is incomplete unless it can be tied back to a window, threshold, detector version, label source, and operator disposition. A dashboard number without review evidence is weak evidence.
Accuracy can look reassuring while operators are still buried in false alerts. Use Figure 42.1 to see why the threshold must be chosen from both detection evidence and operational cost before applying the fleet counts below.
At the top of Figure 42.1, Detection Algorithm names choices such as Z-Score, Isolation Forest, and LSTM. Their reviewed predictions populate the Confusion Matrix and the Precision, Recall, and F1 Score measures in Performance Metrics. The lower route adds False Positive Cost and False Negative Cost under Cost Analysis, then feeds both the metric evidence and Business Reqs into Threshold Selection. That convergence is the point: the cold-storage threshold below must balance missed incidents against the actual alert workload, not maximise one score in isolation.
Worked example: suppose a cold-storage fleet evaluates 10,000 five-minute temperature windows in one week. Twenty windows belong to reviewed refrigeration incidents, but the detector raises 40 alerts. If 12 alerts match real incidents and 28 are normal defrost cycles, the sample-level accuracy still looks high because most windows are normal. Operators, however, experience a 30% precision rate and eight missed incident windows, so the review should focus on alert quality, recall, and whether the threshold is tuned for the right operating cost.
From those same counts, recall is 12 / 20 = 60%, false negatives are 20 - 12 = 8, and the normal-window false-positive rate is 28 / 9,980 = 0.28%. The percentage looks small, but the operator still saw 28 false alerts in a week. That is why anomaly dashboards should show both rates and counts: rates compare detectors, while counts describe workload.
Precision
Of alerts raised, the share that were real incidents: TP / (TP + FP).
Recall
Of real incidents, the share that were caught: TP / (TP + FN).
False Positive Rate
Of normal windows, the share incorrectly alerted: FP / (FP + TN).
Latency
How long the system takes to detect, route, and review the event after enough evidence exists.
Overview Knowledge Check
42.5.3 Tune Thresholds to Review Cost
Threshold tuning is an operating decision. Lower thresholds usually catch more incidents but create more false alerts. Higher thresholds usually reduce alert volume but miss weaker or earlier incidents. The right threshold depends on the cost of investigation, the cost of missed events, staffing, escalation policy, and whether alerts can be grouped before reaching an operator.
Worked example: one week of scored sensor windows total windows: 10,000 reviewed real incidents: 20 alerts raised: 30 true positives: 12 false positives: 18 false negatives: 8 true negatives: 9,962 precision = 12 / (12 + 18) = 0.40 recall = 12 / (12 + 8) = 0.60 F1 = 2 * 0.40 * 0.60 / (0.40 + 0.60) = 0.48 FPR = 18 / (18 + 9,962) = 0.0018 = 0.18% Interpretation: The detector catches 60% of reviewed incidents, but only 40% of alerts are real. Operators see 18 false alerts for 12 true alerts, so alert review may become the limiting factor before the sample-level false positive rate looks large.
Precision-recall curves are often more useful than ROC curves for highly imbalanced anomaly tasks because they show what happens to real alert quality as the threshold moves. ROC curves still help compare ranking behavior, but a small false-positive rate can produce many alerts when normal windows are numerous.
Practitioner Knowledge Check
42.5.4 Metric Labels, Windows, Latency
Detector metrics depend on how events are matched to labels. A vibration spike, a battery brownout, or a connectivity outage may last many samples but represent one incident. Evaluation should define the event window, the allowed detection delay, duplicate-alert suppression, and whether a late alert counts as success, partial success, or a miss.
Streaming systems add timing constraints. Kafka, Spark, Flink, or a gateway pipeline may score event-time windows, and those windows may close before late packets arrive. Metrics should record whether an alert was based on complete data, a provisional early window, imputed samples, or degraded sensor-health evidence. Otherwise, late data and missing samples can change both the label and the score after the dashboard has already reported a metric.
Choose one metric grain before comparing detectors. If the grain is samples, a 20-minute outage with four five-minute windows can create four positives. If the grain is incidents, those four windows may map to one outage. A detector that alerts on all four windows looks strong at sample recall but noisy at operator workload. The incident-grain record should keep first-detection time, last-alert time, duplicate count, and final disposition so precision and recall do not change when someone re-aggregates the same evidence later.
Late data also needs a declared rule. If a gateway sends an alert after three minutes using partial data and the cloud receives the missing packet five minutes later, the metric should preserve both the early alert state and the completed-window state. Otherwise teams may tune a detector on corrected historical data while operators experience provisional edge decisions in production. Keep that distinction in the release note so historical replays do not overwrite what the operator actually saw.
Event Matching
Map many samples or alerts to one incident using a declared window and duplicate-suppression policy.
Label Source
Separate confirmed maintenance records, operator labels, synthetic fault tests, and uncertain weak labels.
Latency Budget
Measure when the detector had enough evidence, when it alerted, and when review completed.
Segment Drift
Track metrics by site, sensor type, firmware version, season, and operating mode before averaging.
Evaluation record for one alert alert_id: pump-17-2026-07-03T10:20Z detector: residual-ewma-v4 window: 10:15:00 to 10:20:00 event time score: 4.8 sigma-equivalent residual threshold: 4.0 data state: complete window, no imputed samples operator label: true positive incident id: pump-17-bearing-review-042 detection latency: 3 min after first threshold crossing review latency: 14 min after alert routing This record supports precision, recall, F1, false-alert workload, time-to-detection, and post-incident audit without changing the detector.
Worked example: if pump-17 raises four overlapping alerts between 10:17 and 10:20 for the same bearing review, the metric job should keep one incident id, one first-alert time, and the duplicate count. Precision then scores one reviewed true positive instead of four, while workload reporting can still show that the operator saw four routed notifications.
Under-the-Hood Knowledge Check
42.5.5 Summary
Anomaly-detector evaluation starts with a reviewed confusion matrix, not ordinary accuracy. Precision measures alert quality, recall measures missed-event coverage, F1 summarizes the tradeoff, false positive rate explains sample-level noise, and latency shows whether the detector is fast enough for the operating risk. A useful IoT metrics record preserves the event window, detector version, threshold, data-completeness state, label source, duplicate-suppression rule, operator disposition, and timing.
Use precision, recall, cost, and latency together. A low false-positive rate can still overload operators when normal windows are numerous, and high precision can still be unsafe if missed incidents are costly.
42.5.6 See Also
Anomaly Detection
Connect metric choices to detector evidence, persistence rules, and operator review.
Types of Anomalies
Map point, contextual, and collective anomalies to event windows and labels.
Statistical Methods
Review z-score, IQR, and EWMA thresholds before comparing alert metrics.
Anomaly Pipelines
Place metrics inside the edge, gateway, stream, and review pipeline that produces alerts.
42.6 Point, Contextual, Collective Anomalies
42.6.1 Begin With What Looks Wrong
Picture a cold room that should stay near its set range. One reading jumps high. That may be a bad point. Yet the same value may be normal while the door is open for loading. A slow rise across many readings may matter even when no single value crosses a limit. The first choice is therefore simple: decide whether the odd evidence is one point, one point in context, or a group pattern.
Write down the time, place, work state, nearby readings, and sensor health. Then try a check that can prove you wrong. Compare a second sensor. Inspect the door event. Look for a stuck or stale value. Keep the full group when the pattern spans time or devices. The alert should show why this evidence set was used.
A tight rule can catch a clear fault fast, but it may raise false alarms. A wide group can reveal drift, but it may react late. This first pass does not choose a maths method or prove the cause. Use the Practitioner layer to match a detector to the evidence need. Use the Under the Hood layer to trace how type errors become alert and response errors. The deeper routes add care without changing the first duty: state what is unusual before scoring it.
42.6.2 Start With the Story
Picture an IoT team using the ideas in Point, Contextual, Collective Anomalies during a live operations review. A device has produced messy evidence, an analytic step is about to change an alert or control decision, and someone has to explain why the result should be trusted.
Read this page as that path from sensor evidence to accountable action. Start with what the system observes, keep the model or data treatment visible, and finish with the check that would convince an operator, maintainer, or auditor to act.
42.6.3 Anomaly Type Sets Evidence
IoT anomaly work starts by classifying what is unusual. A point anomaly is a single reading that is unusual by itself. A contextual anomaly is unusual only under a specific time, place, operating mode, or related-sensor condition. A collective anomaly is a group or sequence that is unusual even when each individual reading looks acceptable.
This classification is not vocabulary trivia. It determines which baseline you need, which detector is reasonable, where the detector can run, and what evidence the alert must preserve for review. Many false alerts come from using a point threshold on a contextual or collective problem, or from treating a sensor-health fault as a process fault.
If you only remember one rule, remember this: classify the anomaly before selecting the algorithm. A threshold, residual, or model score is only meaningful when it matches the anomaly type and the operational context.
Before choosing a detector, decide how much evidence must be read together. Figure 42.2 contrasts a single reading, a context-dependent reading, and a multi-reading pattern so the cold-chain example does not escalate every suspicious number in the same way.
The Point Anomaly panel in Figure 42.2 marks -40 C against a Normal: 20-25 C band, so one value supplies the evidence. Contextual Anomaly puts 80 C in Fridge beside 80 C in Oven: the number is unchanged, but the operating context reverses its meaning. In Collective Anomaly, Sensor 1 through Sensor 4 are individually plausible while the labelled Pattern: Increasing variance is abnormal only as a group. The detector and alert record must therefore retain the matching boundary—sample, context, or window—because that choice controls the baseline, retest, and operator response used next.
For example, a cold-chain trailer may have a cargo target of 2 to 8 deg C. A single 15 deg C sample can be a point anomaly if the door stayed closed, the probe is fresh, and the previous samples were near 4 deg C. The same 15 deg C can be contextual if it occurs during a documented defrost or loading event. A 15-minute sequence where temperature, compressor current, and door events drift together is collective evidence, even if no single reading crosses the emergency limit.
The type label prevents a common escalation error: treating the first suspicious number as the whole story. A reviewable alert should say whether it came from one sample, a context-conditioned value, or a windowed group pattern, because each choice implies a different baseline, retest, and operator response.
Point
One value is outside the expected range or distribution, such as a pressure reading dropping to zero while the pipe is operating.
Contextual
The value is ordinary in one context and suspicious in another, such as high power demand during a closed overnight period.
Collective
A sequence or group forms the abnormal evidence, such as several vibration channels drifting together while each value remains in range.
Fault Boundary
Stuck, clipped, stale, or miscalibrated sensors must be separated from real process anomalies before escalation.
Overview Knowledge Check
42.6.4 Match Detector to Type
A useful implementation records the type decision before it records the detector. For point anomalies, start with thresholds, robust statistics, z-scores, IQR, EWMA, or control limits. For contextual anomalies, add the missing context: hour, day, season, duty cycle, operating state, location, or related-sensor state. For collective anomalies, score windows, feature vectors, correlations, residual sequences, or model reconstruction error.
Classification example: refrigerated asset monitor reading: 15 deg C sensor health: fresh sample, no clipping, battery ok point question: Is 15 deg C impossible or outside the asset limit? contextual question: Was the door open, was this a defrost cycle, or is this the wrong asset class? collective question: Are temperature, compressor current, and door events forming a failure pattern? action: Do not pick one detector first. Store the type decision, baseline, context fields, and sensor-health evidence, then choose the detector.
In practice, the first decision can be written as a short evidence route. If a pressure sensor jumps from 410 kPa to 40 kPa for one fresh sample, the point route stores the limit, unit, sample quality, and previous baseline. If freezer power is high only while the door-open flag is true, the contextual route stores the mode and selected baseline. If four vibration axes slowly rise together for 30 minutes, the collective route stores the window, features, and correlation evidence before escalation.
Practitioner Knowledge Check
42.6.5 Type Errors Become Alert Errors
The wrong anomaly type creates the wrong evaluation problem. A point detector can look good on isolated spikes while missing slow collective drift. A contextual detector can over-alert if context fields are missing or stale. A collective detector can create opaque alerts if it cannot explain which features or windows drove the score. The type label should therefore be part of the alert schema, not just a design note.
Production systems often use a cascade. Edge checks catch impossible values and sensor-health failures. Fog or gateway checks add mode and recent-history context. Cloud or fleet checks score longer windows and cross-device patterns. The cascade only works when each tier reports its evidence and does not overwrite a sensor fault as a process anomaly.
A minimal alert schema needs enough fields to reconstruct the evidence boundary: type, entity, timestamp, window length, baseline id, context version, feature set, sensor-health state, score, threshold, decision, and retest trigger. For a point alert, the window may be one sample and the retest may be immediate. For a contextual alert, the context version matters because a missing shift schedule can turn normal behavior into a false alarm. For a collective alert, the feature set and window are part of the claim, not optional metadata.
The cost of a type error can be estimated before deployment. If 500 assets each produce one false point alert per day, the team receives 500 reviews daily. Requiring a 15-minute collective pattern may cut review load, but it also delays detection and can miss short faults. Conversely, using only a collective detector may hide an impossible single value that should be quarantined at the edge. The production design should state which type is allowed to page humans, which type creates a candidate incident, and which type only updates model health.
Base Rate
Rare true events make false alerts expensive, so review burden matters as much as raw detection count.
Window Length
Collective patterns need enough history, but long windows increase detection latency.
Context Drift
Operating modes and seasonal baselines change, so contextual rules need explicit retest triggers.
Explanation
Operators need the value, context, window, related sensors, and type decision that produced the alert.
Under-the-Hood Knowledge Check
42.6.6 Summary
Point, contextual, and collective anomalies describe different evidence boundaries. Point anomalies are unusual single values, contextual anomalies are unusual only under specific conditions, and collective anomalies are unusual patterns across windows or groups. Sensor faults should remain a separate boundary so operations teams do not confuse measurement failure with process failure. Classify the anomaly type first, then select the detector, evidence record, deployment tier, and retest trigger.
Anomaly type is an engineering decision. Store the type, baseline, context, window, score, threshold, and sensor-health evidence with the alert so the detector can be reviewed, tuned, and retested.
42.6.7 See Also
Anomaly Detection
Review the baseline, score, threshold, and persistence evidence behind IoT anomaly alerts.
Statistical Methods
Use simple statistical scores for point anomalies and edge-friendly candidate alerts.
Time-Series Methods
Add time, season, operating mode, and residual context for contextual anomalies.
Machine Learning Methods
Score multivariate, windowed, and collective patterns when single-signal rules are not enough.
42.7 Continue to the Next Part
Carry this evidence into Time-Series Anomalies: Real-Time Pipelines, which begins with Real-Time Anomaly Pipelines.