15 Real-Time Anomaly Pipelines

analytics-ml

anomaly

pipelines

15.1 Start With the Story

Picture an IoT team using the ideas in Real-Time Anomaly Pipelines during a live operations review. A device has produced messy evidence, an analytic step is about to change an alert or control decision, and someone has to explain why the result should be trusted.

Read this page as that path from sensor evidence to accountable action. Start with what the system observes, keep the model or data treatment visible, and finish with the check that would convince an operator, maintainer, or auditor to act.

15.2 Scores to Reviewable Incidents

A real-time anomaly pipeline is the path from sensor evidence to an alert that can be trusted, reviewed, and improved. It does more than run a model. It ingests readings, validates timestamps, forms windows, extracts features, scores each window, routes alerts, records feedback, and triggers retests when the operating context changes.

The key design choice is where each step belongs. Edge devices can catch immediate safety thresholds and buffer raw readings. Gateways can join nearby sensors, build sliding-window features, and run compact models. Cloud or data-center systems can retrain models, compare sites, and review long-horizon drift. The pipeline should keep the event window, detector version, threshold, features, score, and review outcome together.

For anomaly detection, the pipeline is part of the detector. A strong model with missing timestamps, incomplete windows, duplicate alerts, or no feedback trail is not production-ready.

Production anomaly pipelines place fast safety checks at the edge, multi-sensor scoring at the fog or gateway layer, and heavier training or drift review in the cloud.

Worked example: a refrigeration site may need an edge rule to stop a compressor within one control cycle, a gateway model to compare temperature, current, and door-open signals over a five-minute window, and a cloud job to decide whether winter baselines should differ from summer baselines. If those three decisions are recorded separately, the team cannot tell whether an alert changed because the edge rule fired, the gateway feature vector shifted, or the cloud threshold was redeployed.

A useful alert record therefore keeps the pipeline path as evidence: sensor id, event-time window, feature version, scoring location, detector version, threshold, score, incident key, routing decision, and reviewer label. That record also protects retesting. When a firmware update changes a payload field or a gateway starts imputing missing readings, the team can replay the same windows through the old and new stages and compare the alert count before changing production thresholds.

Ingest

Collect sensor readings through MQTT, HTTP, files, or a broker such as Kafka with schema and timestamp checks.

Window

Group readings by event time using tumbling, sliding, or session windows plus a late-data policy.

Score

Run a threshold, residual, Isolation Forest, autoencoder, or ensemble against the selected features.

Review

Route incidents, suppress duplicates, capture operator labels, and feed confirmed outcomes into retests.

Overview Knowledge Check

15.3 Place Work by Review Need

Start with the response deadline. If the process must stop immediately, run the safety rule at the edge and treat network delivery as a reporting path, not the control path. If the anomaly depends on nearby sensors, score at a gateway. If it requires weeks of history, cross-site comparison, or retraining, keep that work in the cloud or data center and send back validated thresholds or model artifacts.

Worked example: gateway feature pipeline
sensor count: 120
sample rate: 2 readings per second
payload size: 16 bytes per reading
raw rate: 120 * 2 * 16 = 3,840 bytes/s

window: 10 seconds, sliding every 5 seconds
raw bytes per 10-second window:
120 * 2 * 10 * 16 = 38,400 bytes

feature summary:
8 float features per sensor, 4 bytes each
feature bytes per window: 120 * 8 * 4 = 3,840 bytes
metadata budget: about 960 bytes
total summary: about 4,800 bytes every 5 seconds = 960 bytes/s

result:
feature stream is about 75% smaller than raw transport.
The gateway still needs a short raw buffer so a reviewed alert can be replayed.

Stage

Good Location

Evidence to Keep

Main Risk

Safety threshold

Edge device or local controller.

Raw value, threshold, timestamp, actuator action, and sensor-health state.

Relying on a remote network path for immediate control.

Window features

Gateway or stream job close to the sensor fleet.

Window start/end, stride, watermark, missing-data policy, and feature schema.

Scoring partial windows as if they were complete.

Model scoring

Edge for tiny rules, gateway for compact models, cloud for long context.

Model version, feature version, score, threshold, and deployment segment.

Changing features without retesting the deployed threshold.

Alert routing

Operations service, alert manager, or incident workflow.

Incident id, grouping rule, escalation state, operator label, and review notes.

Paging operators for duplicate alerts from one incident.

Practitioner Knowledge Check

15.4 Stream Semantics for Alerts

Real-time anomaly pipelines usually run on event streams. The difference between event time and processing time matters: an alert should normally be tied to when the sensor event happened, not merely when a server received it. Windowed detectors need watermarks or late-data rules so a stream job knows when a window is complete enough to score.

Reliability also needs explicit failure behavior. Bounded queues and backpressure protect downstream scoring jobs. Dead-letter queues preserve malformed readings for inspection. Idempotent incident ids prevent retry storms from creating duplicate alerts. Exactly-once processing can help in systems that support it, but many IoT pipelines still need idempotent writes and duplicate suppression because edge networks retry, reconnect, and resend.

Consider a 60-second event-time window with a 15-second watermark. If a gateway has received 56 of 60 expected readings when the watermark expires, it can score the window as provisional, attach data_state=partial, and update the same incident if the four late readings arrive before the recomputation deadline. If the pipeline silently replaces the first score, operators lose the fact that the original alert was made with incomplete evidence.

Backpressure is equally concrete. If a model worker needs 25 ms to score one window, one worker can process about 40 windows per second. A fleet that emits 80 windows per second needs at least two equivalent workers before overhead, or the queue grows by roughly 40 windows per second and alert latency rises even though the model itself has not changed. The degraded-mode rule should say whether to shed noncritical assets, widen the stride, or fall back to a cheaper threshold while preserving the overload state in the audit record. That state should travel with the incident so reviewers know whether a quiet period was genuinely normal or simply under-sampled.

Event Time

Scores the window based on sensor timestamps rather than arrival order alone.

Watermark

Defines how long the stream waits for late readings before scoring or marking a result provisional.

Backpressure

Uses bounded queues, shedding policy, or degraded mode when a stage cannot keep up.

Idempotency

Gives each incident a stable key so retries do not create repeated pages for one event.

Incident key example
site: plant-4
asset: compressor-17
detector: residual-ewma-v4
window_start: 2026-07-03T10:15:00Z
window_end:   2026-07-03T10:20:00Z

incident key:
plant-4:compressor-17:residual-ewma-v4:2026-07-03T10:15:00Z

Why it matters:
If MQTT reconnects, Kafka retries, or the alert writer restarts,
the same incident key can update the existing alert instead of paging again.

Failure Mode

Symptom

Control

Audit Field

Late data

Window score changes after the first alert.

Watermark, provisional alert state, and recomputation policy.

data_state and watermark_delay.

Schema drift

Feature values shift after firmware or payload changes.

Schema registry, feature-version check, and blocked deployment until retest.

schema_version and feature_version.

Retry duplicate

One event creates repeated operator notifications.

Stable incident key, idempotent write, and alert grouping.

incident_id and duplicate_count.

Overload

Scoring lags behind event time and misses response deadlines.

Backpressure, queue limits, lower-cost fallback rule, and degraded-mode reporting.

queue_depth and scoring_latency.

Under-the-Hood Knowledge Check

15.5 Summary

A real-time anomaly pipeline connects sensor evidence to reviewable incidents. It should validate timestamps, form event-time windows, extract versioned features, score with an appropriate edge, gateway, or cloud method, route grouped alerts, capture feedback, and trigger retests when schemas, features, baselines, or operating modes change. Reliability controls such as watermarks, backpressure, dead-letter queues, idempotent incident keys, and degraded-mode reporting are part of the detector’s trust boundary.

Key Takeaway

Treat pipeline design as detector design. The model score is only useful when the ingestion, windowing, feature, alert, and feedback stages preserve enough evidence to explain and retest the alert.