Analytics & ML · Study deck
Time-Series Anomalies: Real-Time Pipelines
A good detector can fail when samples arrive late, out of order, or after the baseline has moved.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: A useful alert record therefore keeps the pipeline path as evidence: sensor id, event-time window, feature version, scoring location, detector version, threshold, score, incident key, routing decision, and reviewer label.
- Explain: If those three decisions are recorded separately, the team cannot tell whether an alert changed because the edge rule fired, the gateway feature vector shifted, or the cloud threshold was redeployed.
- Explain: The degraded-mode rule should say whether to shed noncritical assets, widen the stride, or fall back to a cheaper threshold while preserving the overload state in the audit record.
Major section
Real-Time Anomaly Pipelines
The score alone cannot tell the operator whether food is at risk, a door is open, a sensor is broken, or a late message arrived.
- Missing times can build the wrong window.
- Repeated alerts can tire staff.
- A quick local rule can protect safety, while a larger remote model can compare more signals but may respond later.
Major section
Real-Time Anomaly Pipelines (continued)
This cold-room story cannot choose one detector or prove every score.
- Those choices need known events and review outcomes.
- The deeper work makes the alert traceable; it does not turn a score into a fact.
- Retest after a change.
- The key design choice is where each step belongs.
Major section
Real-Time Anomaly Pipelines (continued)
Edge devices can catch immediate safety thresholds and buffer raw readings.
- A device has produced messy evidence, an analytic step is about to change an alert or control decision, and someone has to explain why the result should be trusted.
- Gateways can join nearby sensors, build sliding-window features, and run compact models.
- Backpressure is equally concrete.
Major section
Real-Time Anomaly Pipelines (continued)
Cloud or data-center systems can retrain models, compare sites, and review long-horizon drift.
- A real-time anomaly pipeline is the path from sensor evidence to an alert that can be trusted, reviewed, and improved.
- The pipeline should keep the event window, detector version, threshold, features, score, and review outcome together.
- That record also protects retesting.
Major section
Real-Time Anomaly Pipelines (continued)
If the anomaly depends on nearby sensors, score at a gateway.
- It ingests readings, validates timestamps, forms windows, extracts features, scores each window, routes alerts, records feedback, and triggers retests when the operating context changes.
- For anomaly detection, the pipeline is part of the detector.
- result: feature stream is about 75% smaller than raw transport.
Major section
Real-Time Anomaly Pipelines (continued)
If the process must stop immediately, run the safety rule at the edge and treat network delivery as a reporting path, not the control path.
- A strong model with missing timestamps, incomplete windows, duplicate alerts, or no feedback trail is not production-ready.
- Scoring partial windows as if they were complete.
- Reliability also needs explicit failure behavior.
Major section
Real-Time Anomaly Pipelines (continued)
Windowed detectors need watermarks or late-data rules so a stream job knows when a window is complete enough to score.
- If those three decisions are recorded separately, the team cannot tell whether an alert changed because the edge rule fired, the gateway feature vector shifted, or the cloud threshold was redeployed.
- The gateway still needs a short raw buffer so a reviewed alert can be replayed.
- Real-time anomaly pipelines usually run on event streams.
Major section
Real-Time Anomaly Pipelines (continued)
A useful alert record therefore keeps the pipeline path as evidence: sensor id, event-time window, feature version, scoring location, detector version, threshold, score, incident key, routing decision, and reviewer label.
- Bounded queues and backpressure protect downstream scoring jobs.
- Idempotent incident ids prevent retry storms from creating duplicate alerts.
- Window score changes after the first alert.
Major section
Real-Time Anomaly Pipelines (continued)
If it requires weeks of history, cross-site comparison, or retraining, keep that work in the cloud or data center and send back validated thresholds or model artifacts.
- If the pipeline silently replaces the first score, operators lose the fact that the original alert was made with incomplete evidence.
- If a model worker needs 25 ms to score one window, one worker can process about 40 windows per second.
- Feature values shift after firmware or payload changes.
Major section
Real-Time Anomaly Pipelines (continued)
Backpressure Uses bounded queues, shedding policy, or degraded mode when a stage cannot keep up.
- The degraded-mode rule should say whether to shed noncritical assets, widen the stride, or fall back to a cheaper threshold while preserving the overload state in the audit record.
- That state should travel with the incident so reviewers know whether a quiet period was genuinely normal or simply under-sampled.
- Backpressure, queue limits, lower-cost fallback rule, and degraded-mode reporting.
Major section
Real-Time Anomaly Pipelines (continued)
Watermark Defines how long the stream waits for late readings before scoring or marking a result provisional.
- Idempotency Gives each incident a stable key so retries do not create repeated pages for one event.
- A real-time anomaly pipeline connects sensor evidence to reviewable incidents.
- Reliability controls such as watermarks, backpressure, dead-letter queues, idempotent incident keys, and degraded-mode reporting are part of the detector's trust boundary.
Deck summary
Key takeaways
The score alone cannot tell the operator whether food is at risk, a door is open, a sensor is broken, or a late message arrived.
- This cold-room story cannot choose one detector or prove every score.
- Edge devices can catch immediate safety thresholds and buffer raw readings.
- Cloud or data-center systems can retrain models, compare sites, and review long-horizon drift.
- If the anomaly depends on nearby sensors, score at a gateway.
Retrieval practice
Recall check 1 of 3
Data Dora says: answer from memory, then check your reasoning.
Q1Why should an anomaly pipeline preserve the event window, features, score, threshold, and model version with each alert?
Show answer
Answer: A A production anomaly alert must carry enough pipeline evidence to be audited.
Retrieval practice
Recall check 2 of 3
Data Dora says: answer from memory, then check your reasoning.
Q2In the worked example, why should the gateway keep a short raw buffer even after emitting smaller feature summaries?
Show answer
Answer: A Summaries reduce bandwidth, while retained raw windows preserve evidence for alert review.
Retrieval practice
Recall check 3 of 3
Data Dora says: answer from memory, then check your reasoning.
Q3Why is a stable incident key useful in a real-time anomaly pipeline?
Show answer
Answer: A Stream retries are common, so alert records need idempotent identifiers.
Print reference
Answers
Answer key.
- A · A production anomaly alert must carry enough pipeline evidence to be audited.
- A · Summaries reduce bandwidth, while retained raw windows preserve evidence for alert review.
- A · Stream retries are common, so alert records need idempotent identifiers.