Analytics & ML · Study deck
Time-Series Anomalies: Real-Time Pipelines
A good detector can fail when samples arrive late, out of order, or after the baseline has moved.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Design a windowed anomaly pipeline for late and out-of-order samples.
- Define drift, retraining, and operator-review signals.
- Explain: The score alone cannot tell the operator whether food is at risk, a door is open, a sensor is broken, or a late message arrived.
- Explain: This cold-room story cannot choose one detector or prove every score.
Major section
Real-Time Anomaly Pipelines
The score alone cannot tell the operator whether food is at risk, a door is open, a sensor is broken, or a late message arrived.
- Missing times can build the wrong window.
- Repeated alerts can tire staff.
- A quick local rule can protect safety, while a larger remote model can compare more signals but may respond later.
Major section
Real-Time Anomaly Pipelines (continued)
This cold-room story cannot choose one detector or prove every score.
- Those choices need known events and review outcomes.
- The deeper work makes the alert traceable; it does not turn a score into a fact.
- Retest after a change.
- The key design choice is where each step belongs.
Major section
Real-Time Anomaly Pipelines (continued)
Gateways can join nearby sensors, build sliding-window features, and run compact models.
- A real-time anomaly pipeline is the path from sensor evidence to an alert that can be trusted, reviewed, and improved.
- Edge devices can catch immediate safety thresholds and buffer raw readings.
- For anomaly detection, the pipeline is part of the detector.
Major section
Real-Time Anomaly Pipelines (continued)
That record also protects retesting.
- It ingests readings, validates timestamps, forms windows, extracts features, scores each window, routes alerts, records feedback, and triggers retests when the operating context changes.
- Cloud or data-center systems can retrain models, compare sites, and review long-horizon drift.
- If the anomaly depends on nearby sensors, score at a gateway.
Major section
Real-Time Anomaly Pipelines (continued)
The pipeline should keep the event window, detector version, threshold, features, score, and review outcome together.
- A strong model with missing timestamps, incomplete windows, duplicate alerts, or no feedback trail is not production-ready.
- result: feature stream is about 75% smaller than raw transport.
- Scoring partial windows as if they were complete.
- Backpressure is equally concrete.
Major section
Real-Time Anomaly Pipelines (continued)
Windowed detectors need watermarks or late-data rules so a stream job knows when a window is complete enough to score.
- If those three decisions are not recorded separately, the team cannot tell whether an alert changed because the edge rule fired, the gateway feature vector shifted, or the cloud threshold was redeployed.
- The gateway still needs a short raw buffer so a reviewed alert can be replayed.
- Real-time anomaly pipelines usually run on event streams.
Major section
Real-Time Anomaly Pipelines (continued)
Reliability also needs explicit failure behavior.
- A useful alert record therefore keeps the pipeline path as evidence: sensor id, event-time window, feature version, scoring location, detector version, threshold, score, incident key, routing decision, and reviewer label.
- Bounded queues and backpressure protect downstream scoring jobs.
- Idempotent incident ids prevent retry storms from creating duplicate alerts.
Major section
Real-Time Anomaly Pipelines (continued)
WatermarkDefines how long the stream waits for late readings before scoring or marking a result provisional.
- If the process must stop immediately, run the safety rule at the edge and treat network delivery as a reporting path, not the control path.
- If it requires weeks of history, cross-site comparison, or retraining, keep that work in the cloud or data center and send back validated thresholds or model artifacts.
- Window score changes after the first alert.
Major section
Real-Time Anomaly Pipelines (continued)
That state should travel with the incident so reviewers know whether a quiet period was genuinely normal or simply under-sampled.
- If the pipeline silently replaces the first score, operators lose the fact that the original alert was made with incomplete evidence.
- If a model worker needs 25 ms to score one window, one worker can process about 40 windows per second.
- BackpressureUses bounded queues, shedding policy, or degraded mode when a stage cannot keep up.
Major section
Real-Time Anomaly Pipelines (continued)
The degraded-mode rule should say whether to shed noncritical assets, widen the stride, or fall back to a cheaper threshold while preserving the overload state in the audit record.
- IdempotencyGives each incident a stable key so retries do not create repeated pages for one event.
- Feature values shift after firmware or payload changes.
- Backpressure, queue limits, lower-cost fallback rule, and degraded-mode reporting.
- A real-time anomaly pipeline connects sensor evidence to reviewable incidents.
Major section
Real-Time Anomaly Pipelines (continued)
It should validate timestamps, form event-time windows, extract versioned features, score with an appropriate edge, gateway, or cloud method, route grouped alerts, capture feedback, and trigger retests when schemas, features, baselines, or operating modes change.
- Reliability controls such as watermarks, backpressure, dead-letter queues, idempotent incident keys, and degraded-mode reporting are part of the detector's trust boundary.
- The model score is only useful when the ingestion, windowing, feature, alert, and feedback stages preserve enough evidence to explain and retest the alert.
- Edge DeploymentPlace compact scoring and fallback rules near devices when response deadlines are tight.
Deck summary
Key takeaways
The score alone cannot tell the operator whether food is at risk, a door is open, a sensor is broken, or a late message arrived.
- This cold-room story cannot choose one detector or prove every score.
- Gateways can join nearby sensors, build sliding-window features, and run compact models.
- That record also protects retesting.
- The pipeline should keep the event window, detector version, threshold, features, score, and review outcome together.
Retrieval practice
Recall check 1 of 4
Data Dora says: answer from memory, then check your reasoning.
Q1A sensor event happened at 10:02 but reached the stream job at 10:04. Which timestamp should place it in an anomaly window?
Show answer
Answer: C Event-time windows use sensor timestamps; watermarks or late-data rules bound scoring and revisions.
Retrieval practice
Recall check 2 of 4
Data Dora says: answer from memory, then check your reasoning.
Q2In the 120-sensor worked example, raw traffic is 3,840 bytes per second and feature summaries use 960 bytes per second. What does that comparison establish?
Show answer
Answer: A In the worked example, 960 / 3,840 = 0.25, so feature transport is 75% smaller.
Retrieval practice
Recall check 3 of 4
Data Dora says: answer from memory, then check your reasoning.
Q3One model worker scores 40 windows per second, but a fleet sends 80. What should the pipeline do as its scoring queue grows?
Show answer
Answer: C At 80 arrivals and 40 scores per second, the queue grows by roughly 40 windows each second without capacity or a declared degraded mode.
Retrieval practice
Recall check 4 of 4
Data Dora says: answer from memory, then check your reasoning.
Q4A 60-second anomaly window has only 56 of 60 readings when its watermark expires. What should the first alert record say?
Show answer
Answer: A A partial event-time window needs a visible provisional state and a retained revision history.
Print reference
Answers
Answer key.
- C · Event-time windows use sensor timestamps; watermarks or late-data rules bound scoring and revisions.
- A · In the worked example, 960 / 3,840 = 0.25, so feature transport is 75% smaller.
- C · At 80 arrivals and 40 scores per second, the queue grows by roughly 40 windows each second without capacity or a declared degraded mode.
- A · A partial event-time window needs a visible provisional state and a retained revision history.