Analytics & ML · Study deck
Time-Series Anomalies: Real-Time Pipelines
A good detector can fail when samples arrive late, out of order, or after the baseline has moved.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Design a windowed anomaly pipeline for late and out-of-order samples.
- Define drift, retraining, and operator-review signals.
- Explain: The score alone cannot tell the operator whether food is at risk, a door is open, a sensor is broken, or a late message arrived.
- Explain: This cold-room story cannot choose one detector or prove every score.
Major section
Real-Time Anomaly Pipelines
The score alone cannot tell the operator whether food is at risk, a door is open, a sensor is broken, or a late message arrived.
- Missing times can build the wrong window.
- Repeated alerts can tire staff.
- A quick local rule can protect safety, while a larger remote model can compare more signals but may respond later.
Major section
Real-Time Anomaly Pipelines (continued)
This cold-room story cannot choose one detector or prove every score.
- Those choices need known events and review outcomes.
- The deeper work makes the alert traceable; it does not turn a score into a fact.
- Retest after a change.
- The key design choice is where each step belongs.
Major section
Real-Time Anomaly Pipelines (continued)
Gateways can join nearby sensors, build sliding-window features, and run compact models.
- A real-time anomaly pipeline is the path from sensor evidence to an alert that can be trusted, reviewed, and improved.
- Edge devices can catch immediate safety thresholds and buffer raw readings.
- For anomaly detection, the pipeline is part of the detector.
Major section
Real-Time Anomaly Pipelines (continued)
That record also protects retesting.
- It ingests readings, validates timestamps, forms windows, extracts features, scores each window, routes alerts, records feedback, and triggers retests when the operating context changes.
- Cloud or data-center systems can retrain models, compare sites, and review long-horizon drift.
- If the anomaly depends on nearby sensors, score at a gateway.
Major section
Real-Time Anomaly Pipelines (continued)
The pipeline should keep the event window, detector version, threshold, features, score, and review outcome together.
- A strong model with missing timestamps, incomplete windows, duplicate alerts, or no feedback trail is not production-ready.
- result: feature stream is about 75% smaller than raw transport.
- Scoring partial windows as if they were complete.
- Backpressure is equally concrete.
Major section
Real-Time Anomaly Pipelines (continued)
Windowed detectors need watermarks or late-data rules so a stream job knows when a window is complete enough to score.
- If those three decisions are not recorded separately, the team cannot tell whether an alert changed because the edge rule fired, the gateway feature vector shifted, or the cloud threshold was redeployed.
- The gateway still needs a short raw buffer so a reviewed alert can be replayed.
- Real-time anomaly pipelines usually run on event streams.
Major section
Real-Time Anomaly Pipelines (continued)
Reliability also needs explicit failure behavior.
- A useful alert record therefore keeps the pipeline path as evidence: sensor id, event-time window, feature version, scoring location, detector version, threshold, score, incident key, routing decision, and reviewer label.
- Bounded queues and backpressure protect downstream scoring jobs.
- Idempotent incident ids prevent retry storms from creating duplicate alerts.
Major section
Real-Time Anomaly Pipelines (continued)
WatermarkDefines how long the stream waits for late readings before scoring or marking a result provisional.
- If the process must stop immediately, run the safety rule at the edge and treat network delivery as a reporting path, not the control path.
- If it requires weeks of history, cross-site comparison, or retraining, keep that work in the cloud or data center and send back validated thresholds or model artifacts.
- Window score changes after the first alert.
Major section
Real-Time Anomaly Pipelines (continued)
That state should travel with the incident so reviewers know whether a quiet period was genuinely normal or simply under-sampled.
- If the pipeline silently replaces the first score, operators lose the fact that the original alert was made with incomplete evidence.
- If a model worker needs 25 ms to score one window, one worker can process about 40 windows per second.
- BackpressureUses bounded queues, shedding policy, or degraded mode when a stage cannot keep up.
Major section
Real-Time Anomaly Pipelines (continued)
The degraded-mode rule should say whether to shed noncritical assets, widen the stride, or fall back to a cheaper threshold while preserving the overload state in the audit record.
- IdempotencyGives each incident a stable key so retries do not create repeated pages for one event.
- Feature values shift after firmware or payload changes.
- Backpressure, queue limits, lower-cost fallback rule, and degraded-mode reporting.
- A real-time anomaly pipeline connects sensor evidence to reviewable incidents.
Major section
Real-Time Anomaly Pipelines (continued)
It should validate timestamps, form event-time windows, extract versioned features, score with an appropriate edge, gateway, or cloud method, route grouped alerts, capture feedback, and trigger retests when schemas, features, baselines, or operating modes change.
- Reliability controls such as watermarks, backpressure, dead-letter queues, idempotent incident keys, and degraded-mode reporting are part of the detector's trust boundary.
- The model score is only useful when the ingestion, windowing, feature, alert, and feedback stages preserve enough evidence to explain and retest the alert.
- Edge DeploymentPlace compact scoring and fallback rules near devices when response deadlines are tight.
Deck summary
Key takeaways
The score alone cannot tell the operator whether food is at risk, a door is open, a sensor is broken, or a late message arrived.
- This cold-room story cannot choose one detector or prove every score.
- Gateways can join nearby sensors, build sliding-window features, and run compact models.
- That record also protects retesting.
- The pipeline should keep the event window, detector version, threshold, features, score, and review outcome together.
Retrieval practice
Recall check 1 of 4
Data Dora says: answer from memory, then check your reasoning.
Q1Why should an anomaly pipeline preserve the event window, features, score, threshold, and model version with each alert?
Show answer
Answer: C A production anomaly alert must carry enough pipeline evidence to be audited.
Retrieval practice
Recall check 2 of 4
Data Dora says: answer from memory, then check your reasoning.
Q2In the worked example, why should the gateway keep a short raw buffer even after emitting smaller feature summaries?
Show answer
Answer: A Summaries reduce bandwidth, while retained raw windows preserve evidence for alert review.
Retrieval practice
Recall check 3 of 4
Data Dora says: answer from memory, then check your reasoning.
Q3A firmware update changes the meaning of a feature used by a live anomaly model. Alerts shift sharply. What should happen before normal scoring resumes?
Show answer
Answer: C Versioned feature evidence and operator review show whether the detector still means what it claims.
Retrieval practice
Recall check 4 of 4
Data Dora says: answer from memory, then check your reasoning.
Q4Why is a stable incident key useful in a real-time anomaly pipeline?
Show answer
Answer: A Stream retries are common, so alert records need idempotent identifiers.
Print reference
Answers
Answer key.
- C · A production anomaly alert must carry enough pipeline evidence to be audited.
- A · Summaries reduce bandwidth, while retained raw windows preserve evidence for alert review.
- C · Versioned feature evidence and operator review show whether the detector still means what it claims.
- A · Stream retries are common, so alert records need idempotent identifiers.