16  Real-Time Anomaly Pipelines

In 60 Seconds

Production anomaly detection requires a three-tier pipeline architecture spanning edge, fog, and cloud – simple statistical checks at the edge respond in under 10ms for safety-critical alerts, ML models at fog gateways aggregate multi-sensor data in about 1 second, and deep learning in the cloud discovers long-term patterns across thousands of devices. This layered approach achieves 99% bandwidth reduction while preserving all anomalies, and feedback loops from operator decisions continuously improve detection accuracy across all tiers.

16.1 Learning Objectives

By the end of this chapter, you will be able to:

• Design Three-Tier Architecture: Build anomaly detection systems spanning edge, fog, and cloud
• Balance Latency and Accuracy: Choose where to deploy detection based on requirements
• Handle Concept Drift: Implement adaptive systems that evolve with changing data patterns
• Build Production Pipelines: Create end-to-end systems from data ingestion to alerting

For Beginners: Anomaly Detection Pipelines

An anomaly detection pipeline is like an assembly line for finding unusual events in sensor data. Raw readings enter at one end, get cleaned and analyzed step by step, and alerts come out the other end when something abnormal is detected. Building this as a pipeline means each step can be tested and improved independently.

Minimum Viable Understanding: Production Pipelines

Core Concept: Production anomaly detection requires a layered architecture - simple statistical methods at the edge for immediate response, ML models at gateways for deeper analysis, and cloud systems for training and pattern discovery.

Why It Matters: Edge-only detection is fast but misses complex patterns. Cloud-only detection has high latency and network dependency. The right architecture balances both.

Key Takeaway: Deploy threshold detection at the edge (<100ms), ML models at gateways (~1s), and deep learning in the cloud (~10s). Use feedback loops to continuously improve all layers.

16.2 Prerequisites

Before diving into this chapter, you should be familiar with:

• Statistical Methods: Methods suitable for edge deployment
• Machine Learning Methods: Methods requiring cloud or gateway resources
• Anomaly Types: Understanding which anomalies to detect where

~15 min | Advanced | P10.C01.U05

Key Concepts

• Three-tier pipeline: An architecture placing threshold detection at the edge (<100 ms), ML-based detection at fog gateways (~1 s), and deep-learning analysis in the cloud (~10 s).
• Concept drift: The gradual change in statistical properties of sensor data over time that causes a previously accurate model to degrade without retraining.
• Ensemble detection: Combining outputs from multiple algorithms and only raising an alert when a majority agree, reducing false positives without sacrificing recall.
• Alert fusion: Aggregating related alerts from multiple sensors or time windows into a single prioritised incident to reduce operator notification volume.
• Backpressure: A flow-control mechanism that prevents downstream pipeline stages from being overwhelmed by upstream data bursts, typically implemented with bounded queues.
• Feedback loop: A mechanism routing operator decisions (false positive/confirmed anomaly) back to the model training pipeline so the detector continuously improves.
• Dead-letter queue: A storage location for data packets that failed pipeline processing, preserving them for later analysis rather than silently dropping them.

16.3 Introduction

Moving from algorithms to production systems requires careful architecture design balancing latency, accuracy, and resource constraints. This chapter walks through a three-tier pipeline architecture – edge, fog, and cloud – showing what each tier does, how data flows between them, and how to keep the system accurate as conditions change over time.

How It Works

Production anomaly detection pipelines operate as multi-tier systems where each layer has distinct responsibilities:

Edge Tier (Device/Sensor Level):

• Runs simple statistical checks (Z-score, threshold) in <10ms
• Triggers immediate safety responses (motor shutdown, valve close)
• Filters 90% of data locally before transmission
• Example: ESP32 with 520KB RAM running Z-score on temperature readings

Fog Tier (Gateway/Concentrator Level):

• Aggregates data from 10-1000 sensors
• Runs ML models (Isolation Forest) on multi-sensor windows in ~1 second
• Provides local dashboards and alerts
• Example: Raspberry Pi 4 processing 100 sensors with trained Isolation Forest model

Cloud Tier (Data Center Level):

• Trains ML models on historical data from all devices
• Runs deep learning (LSTM autoencoders) for pattern discovery
• Provides cross-site analytics and long-term trends
• Example: AWS running Spark to train models on 6 months of data from 10,000 devices

The Flow: Raw data enters at the edge tier, gets filtered to summaries at the fog tier (typically 90%+ reduction), then aggregates to compact insights sent to the cloud. Alerts flow in reverse: cloud discovers gradual degradation pattern, updates fog model, which refines edge thresholds.

For Kids: Meet the Sensor Squad!

The Sensor Squad was protecting a big chocolate factory, and they had a clever three-layer defense system!

Sammy the Sensor was stationed right next to the chocolate mixer. “I am the FIRST line of defense!” he said proudly. “If the temperature shoots above 80 degrees, I sound the alarm in less than one second. I am fast but I only know simple rules.”

Lila the LED was at the factory gateway computer. “I am the SECOND line of defense! Every few seconds, I look at readings from ALL the sensors together – temperature, stirring speed, and chocolate thickness. If the combination looks weird, I alert the manager. I am smarter than Sammy but a bit slower.”

Max the Microcontroller was up in the cloud. “And I am the THIRD line of defense! I study weeks of data to spot slow changes. Last month, I noticed the mixer was getting slightly louder each day – a pattern nobody else could see. I predicted the motor would break in two weeks, and the factory fixed it before it failed!”

Bella the Battery summed it up: “It is like having three guards – a fast one at the door who checks your badge, a smart one in the hallway who checks your face, and a detective in the back room who studies patterns over time. Together, nothing gets past us!”

Key lesson: Production anomaly detection uses layers – fast simple checks at the edge, smarter analysis at gateways, and deep pattern detection in the cloud. Each layer catches different types of problems!

16.4 End-to-End Architecture

With the three-tier model established, the next question is how data actually flows through the pipeline – from raw sensor readings to actionable alerts:

Figure 16.1: Three-tier anomaly detection architecture balancing latency and accuracy

Pipeline Stages:

1. Data Ingestion
   • Protocols: MQTT (lightweight, edge), Kafka (high-throughput, cloud)
   • Buffering: Handle network interruptions with local queues
   • Sampling: Adaptive sampling - increase frequency when anomalies suspected
2. Feature Extraction
   • Windowing: Sliding windows (e.g., 10-second windows, 5-second overlap)
   • Aggregation: Mean, std, min, max, percentiles per window
   • Domain-specific: FFT for vibration, wavelets for signals
   • Normalization: Scale features to common range (0-1 or z-score)
3. Model Scoring
   • Edge: Simple statistical methods (<10ms per sample)
   • Gateway: Trained ML models (10-100ms per batch)
   • Cloud: Deep learning ensembles (1-10s per batch)

Putting Numbers to It

Data reduction through the three-tier pipeline follows geometric progression. For 280 sensors at 1 Hz each:

Raw Rate: \(280 \text{ sensors} \times 1 \text{ Hz} \times 100 \text{ bytes} = 28 \text{ KB/s}\)

Edge Tier (Z-score filter, keep anomalies + 10% samples): \[\text{Edge Out} = 28 \text{ KB/s} \times (0.01 \text{ anomaly rate} + 0.10 \text{ sampling}) = 3.08 \text{ KB/s}\]

Fog Tier (aggregate to 1-min summaries, forward anomalies): \[\text{Fog Out} = \frac{280 \times 50 \text{ bytes}}{60 \text{ s}} + (3.08 \text{ KB/s} \times 0.01) \approx 233 + 31 = 264 \text{ bytes/s}\]

Cloud receives \(264 \text{ bytes/s} = 22.8 \text{ MB/day}\) vs. \(28 \text{ KB/s} = 2.4 \text{ GB/day}\) raw.

Reduction: \(99.1\%\) bandwidth savings while preserving all anomalies and statistical trends.

Interactive: Pipeline Data Reduction Calculator

Explore how the three-tier pipeline reduces data volume from edge to cloud.

viewof pipelineSensors = Inputs.range([10, 1000], {value: 280, step: 10, label: "Number of sensors"})
viewof pipelineSampleHz = Inputs.range([0.1, 10], {value: 1.0, step: 0.1, label: "Sample rate (Hz)"})
viewof pipelineBytesPerSample = Inputs.range([10, 500], {value: 100, step: 10, label: "Bytes per sample"})
viewof pipelineAnomalyRate = Inputs.range([0.001, 0.10], {value: 0.01, step: 0.001, label: "Anomaly rate (fraction)"})
viewof pipelineSamplingRate = Inputs.range([0.01, 0.50], {value: 0.10, step: 0.01, label: "Edge sampling keep rate"})
viewof pipelineSummaryBytes = Inputs.range([10, 200], {value: 50, step: 10, label: "Fog summary bytes per sensor"})

{
  const rawRate = pipelineSensors * pipelineSampleHz * pipelineBytesPerSample;
  const edgeOut = rawRate * (pipelineAnomalyRate + pipelineSamplingRate);
  const fogSummary = (pipelineSensors * pipelineSummaryBytes) / 60;
  const fogAnomaly = edgeOut * pipelineAnomalyRate;
  const fogOut = fogSummary + fogAnomaly;
  const rawPerDay = (rawRate * 86400) / (1024 * 1024);
  const fogPerDay = (fogOut * 86400) / (1024 * 1024);
  const reduction = ((1 - fogOut / rawRate) * 100).toFixed(1);

  const fmtRate = (bytes) => {
    if (bytes >= 1024 * 1024) return (bytes / (1024 * 1024)).toFixed(2) + " MB/s";
    if (bytes >= 1024) return (bytes / 1024).toFixed(2) + " KB/s";
    return bytes.toFixed(0) + " bytes/s";
  };

  return html`<div style="background: var(--bs-light, #f8f9fa); padding: 1rem; border-radius: 6px; border-left: 4px solid #16A085; font-family: inherit;">
    <p style="margin: 0.3rem 0;"><strong>Raw sensor rate:</strong> ${fmtRate(rawRate)}</p>
    <p style="margin: 0.3rem 0;"><strong>After edge filtering:</strong> ${fmtRate(edgeOut)}</p>
    <p style="margin: 0.3rem 0;"><strong>After fog aggregation:</strong> ${fmtRate(fogOut)}</p>
    <p style="margin: 0.3rem 0;"><strong>Cloud receives:</strong> ${fogPerDay.toFixed(1)} MB/day (vs. ${rawPerDay.toFixed(1)} MB/day raw)</p>
    <p style="margin: 0.5rem 0 0 0; font-size: 1.1em;"><strong>Total bandwidth reduction: ${reduction}%</strong></p>
  </div>`;
}

4. Alert Generation
   • Thresholding: Confidence scores to binary decision
   • Hysteresis: Require N consecutive anomalies to reduce false alarms
   • Severity levels: Critical (act immediately) vs Warning (monitor)
5. Feedback Loop
   • Operator confirmation: Anomaly true/false?
   • Model updates: Retrain with confirmed labels
   • Threshold tuning: Adjust based on false positive rate

16.5 Edge vs Cloud Detection

Decision Framework:

Requirement	Deploy At Edge If…	Deploy In Cloud If…
Latency	<100ms required	1-10s acceptable
Criticality	Safety-critical (motor shutdown)	Informational (dashboards)
Complexity	Simple patterns (out-of-range)	Complex temporal/spatial
Connectivity	Intermittent or unreliable	Always-on, high bandwidth
Power	Battery-powered devices	Mains-powered or cloud

Hybrid Approach (Recommended):

• Edge: Detect critical anomalies (safety), filter obvious noise
• Cloud: Detect subtle patterns, perform root cause analysis, retrain models

Example: Factory Motor Monitoring

# Edge: Simple threshold detection (runs on ESP32 microcontroller)
def edge_detector(temperature, vibration):
    """
    Runs every 100ms on edge device
    Returns: (is_critical, should_alert)
    """
    critical = (temperature > 95) or (vibration > 5.0)
    if critical:
        # Immediate action: Stop motor
        stop_motor()
        return True, True

    # Send to gateway for advanced detection
    return False, False

# Gateway: ML-based detection (Raspberry Pi)
class GatewayDetector:
    def __init__(self):
        self.model = load_isolation_forest()  # Pre-trained
        self.window = deque(maxlen=100)  # 10 seconds at 10Hz
        self.sample_count = 0

    def update(self, sample):
        """
        Runs every 100ms, batches for ML every 1s
        sample: [temp, vibration, current, rpm]
        """
        self.window.append(sample)
        self.sample_count += 1

        if len(self.window) >= 100 and self.sample_count % 10 == 0:
            # Extract features from full window every 10 samples (~1s)
            features = self.extract_features(self.window)

            # ML prediction (decision_function returns anomaly scores)
            anomaly_score = self.model.decision_function([features])[0]

            if anomaly_score < -0.1:  # Threshold
                send_alert("Moderate anomaly detected")
                send_to_cloud(self.window)  # For deep analysis

            return anomaly_score < -0.1

        return False

    def extract_features(self, window):
        """Extract statistical features from time window"""
        window_array = np.array(window)
        return np.concatenate([
            np.mean(window_array, axis=0),  # Mean of each sensor
            np.std(window_array, axis=0),   # Std of each sensor
            np.max(window_array, axis=0),   # Max of each sensor
            np.min(window_array, axis=0),   # Min of each sensor
        ])

# Cloud: Deep learning pattern analysis (runs every 10s)
class CloudDetector:
    def __init__(self):
        self.lstm_model = load_lstm_model()  # Pre-trained

    def analyze_pattern(self, historical_windows):
        """
        Analyze long-term patterns (hours/days)
        historical_windows: Last 1000 windows (100s of data)
        """
        # LSTM prediction
        predicted_next = self.lstm_model.predict(historical_windows)
        actual_next = historical_windows[-1]

        error = np.mean((predicted_next - actual_next)**2)

        if error > THRESHOLD:
            # Pattern anomaly: gradual degradation detected
            create_maintenance_ticket("Bearing wear pattern detected")
            predict_time_to_failure()

        return error

16.6 Handling Concept Drift

The Problem: “Normal” changes over time: - Sensors degrade and drift - Seasonal patterns (summer vs winter temperatures) - Equipment wear (vibration baseline increases) - Process changes (new production schedules)

Strategies:

1. Continuous Retraining
   • Retrain models weekly/monthly on recent data
   • Use sliding window (last 30 days, not all historical data)
   • Automate via MLOps pipelines
2. Adaptive Thresholds
   • Dynamic thresholds that adjust to recent baseline
   • Example: “Flag if >3 sigma from last 7 days mean” (not all-time mean)
3. Ensemble Models
   • Combine short-term (catches sudden changes) and long-term (catches drift) models
   • Example: Z-score (1-day window) + ARIMA (30-day window)
4. Human-in-the-Loop
   • Operator labels false positives/negatives
   • Feedback trains model incrementally (online learning)

Implementation:

class AdaptiveDetector:
    def __init__(self, retrain_interval=86400):  # Retrain daily
        self.model = IsolationForest()
        self.training_buffer = deque(maxlen=10000)  # Keep last 10k samples
        self.last_retrain = time.time()
        self.retrain_interval = retrain_interval

    def update(self, sample):
        """Process new sample and periodically retrain"""
        # Predict with current model
        is_anomaly = self.model.predict([sample])[0] == -1

        # Add to training buffer (if confirmed normal)
        if not is_anomaly:
            self.training_buffer.append(sample)

        # Check if time to retrain
        if time.time() - self.last_retrain > self.retrain_interval:
            self.retrain()

        return is_anomaly

    def retrain(self):
        """Retrain model on recent data"""
        if len(self.training_buffer) > 1000:
            print(f"Retraining model on {len(self.training_buffer)} samples")
            self.model.fit(list(self.training_buffer))
            self.last_retrain = time.time()

Pitfall: Ignoring Concept Drift in Production Systems

The Mistake: Training anomaly detection models once on historical data and deploying them permanently, assuming the definition of “normal” remains constant over the system’s operational lifetime.

Why It Happens: Model training is resource-intensive and disrupts operations. “If it ain’t broke, don’t fix it” thinking prevails. Concept drift is gradual and invisible - performance degrades slowly without obvious failure points. Teams lack automated drift detection and retraining pipelines.

The Fix: Design for continuous learning from the start:

1. Drift detection: Monitor model performance indicators continuously - track the distribution of anomaly scores over time using Page-Hinkley test or ADWIN algorithm. If the score distribution shifts significantly (KL divergence > 0.1), trigger investigation.

2. Scheduled retraining: Establish periodic retraining cycles (weekly, monthly) using recent data. For seasonal systems (HVAC, agriculture), retrain at season boundaries.

3. Online learning: For fast-drifting environments, use incremental learning algorithms (online Isolation Forest, streaming autoencoders) that update continuously without full retraining.

4. Human-in-the-loop feedback: When operators dismiss alerts as false positives or confirm true anomalies, feed this labeled data back into the training pipeline. Over 6 months, this feedback can improve precision by 20-40%.

5. Model versioning with A/B testing: Deploy new models alongside old ones, compare performance on held-out data, automatically promote better performers.

Reality check: a predictive maintenance model trained in summer may have 30% higher false positive rate in winter due to thermal effects on vibration patterns.

16.7 Worked Example: Three-Tier Anomaly Detection for Water Treatment

Worked Example: Designing a Layered Anomaly Detection Pipeline for a Municipal Water Plant

Scenario: Thames Water operates the Beckton Sewage Treatment Works in east London, processing 900 million litres of wastewater daily. The plant monitors chlorine residual, turbidity, pH, flow rate, and dissolved oxygen across 45 treatment stages. A contamination event must be detected within 60 seconds (safety-critical), while gradual process degradation should be caught within hours.

Given:

• 280 sensors across 45 treatment stages
• Sampling rates: Chlorine/pH at 1 Hz, turbidity at 0.5 Hz, flow at 0.1 Hz
• Total data rate: 280 sensors x avg 0.5 Hz = 140 readings/second
• Safety requirement: Chlorine deviation >0.5 mg/L detected within 60 seconds
• Process optimization: Detect sludge thickener degradation within 4 hours
• Historical data: 3 years of sensor readings with 47 labeled contamination events

Step 1 – Deploy edge detection (safety-critical, <60 second response):

Each treatment stage has a PLC (Siemens S7-1500) performing edge detection:

Check	Method	Threshold	Response Time	Action
Chlorine out of range	Hard bounds	<0.2 or >4.0 mg/L	<1 second	Emergency shutdown of dosing system
pH extreme	Hard bounds	<5.0 or >10.0	<1 second	Divert flow to holding tank
Chlorine rate-of-change	Delta check	>0.3 mg/L in 30 seconds	30 seconds	Alert operator + increase sampling
Turbidity spike	Z-score (window=60)	Z > 4.0	60 seconds	Alert + trigger grab sample

Edge detection catches: instantaneous sensor failures, sudden contamination events, dosing equipment malfunctions.

Edge detection misses: gradual fouling, seasonal baseline shifts, multi-sensor correlated degradation.

Step 2 – Deploy gateway detection (process anomalies, ~5 minute response):

A gateway server (Dell PowerEdge, on-premises) runs Isolation Forest models:

• Input: 5-minute rolling features from 280 sensors (mean, std, min, max, trend slope)
• Model: 45 Isolation Forest models (one per treatment stage), trained on 2 years of normal operation
• Detection targets: Multi-sensor anomalies where individual readings are in-range but the combination is abnormal
• Example: Turbidity normal (2.1 NTU), pH normal (7.2), but dissolved oxygen dropping from 6.0 to 4.5 mg/L while flow is constant – indicates biological treatment failure

Detection Type	Feature Window	Model Update	False Positive Rate
Multi-sensor correlation	5-minute aggregates	Monthly retrain	2.1% (acceptable for operator review)
Diurnal pattern deviation	1-hour features vs time-of-day profile	Weekly profile update	1.8%
Inter-stage anomaly	Difference between stage N and stage N+1	Monthly	0.9%

Step 3 – Deploy cloud detection (long-term degradation, hours to days):

Azure cloud runs LSTM autoencoders on historical patterns:

• Training data: 3 years, 47 labeled events, ~10 billion data points
• Model: LSTM autoencoder per treatment stage, 168-hour (1-week) input window
• Detection: Reconstruction error exceeding 99th percentile of training distribution
• Targets: Membrane fouling (gradual over 2-4 weeks), sludge thickener degradation (5-10 day progression), seasonal algae bloom effects

Retraining: Quarterly, incorporating new labeled events. Model A/B testing before promotion to production.

Step 4 – Calculate detection coverage:

Anomaly Type	Edge	Gateway	Cloud	Overall
Sudden contamination (47 events in 3 years)	43/47 (91%)	46/47 (98%)	47/47 (100%)	47/47 (100%)
Gradual degradation (23 events)	2/23 (9%)	14/23 (61%)	21/23 (91%)	21/23 (91%)
Sensor malfunction (156 events)	148/156 (95%)	155/156 (99%)	156/156 (100%)	156/156 (100%)
Weighted detection rate	77%	90%	96%	98%

Cost of the pipeline:

Tier	Hardware	Software	Annual Cost
Edge (45 PLCs, existing)	GBP 0 (already installed)	Configuration only	GBP 5,000
Gateway (1 server)	GBP 4,000 (one-time)	Open-source ML stack	GBP 2,000
Cloud (Azure)	–	Azure ML + storage	GBP 14,000
Total annual			GBP 21,000

Value: A single undetected contamination event resulting in a Drinking Water Inspectorate prosecution costs GBP 250,000-500,000 in fines plus remediation. The pipeline has prevented 3 near-miss events in its first year of operation.

Result: The three-tier architecture achieves 98% weighted detection rate at GBP 21,000/year. Edge tier handles 77% of anomalies with <1 second response. Gateway adds 13% coverage for multi-sensor correlations. Cloud adds 6% for long-term degradation patterns. Each tier catches anomaly types that lower tiers miss.

Key Insight: The three tiers are not redundant – they detect fundamentally different anomaly types. Removing any tier creates blind spots. Edge catches sudden failures (critical for safety). Gateway catches cross-sensor correlations (critical for process quality). Cloud catches slow degradation (critical for maintenance planning). The layered cost also scales appropriately: edge detection is nearly free (reuses existing PLCs), gateway costs GBP 2,000/year, and cloud costs GBP 14,000/year.

Concept Relationships

How These Concepts Connect:

• Detection complexity increases from edge to cloud: Edge handles simple point anomalies, fog handles multi-sensor patterns, cloud handles complex temporal sequences
• Latency requirements drive deployment: Safety systems need edge (<100ms), predictive maintenance tolerates cloud (~10s)
• Feedback loops maintain accuracy: Cloud trains models → fog deploys models → edge applies thresholds → all send results back to cloud for retraining
• Concept drift requires all tiers: Edge detects immediate shifts, fog adapts thresholds, cloud retrains long-term models

See Also

Foundation Concepts:

• Anomaly Types - Understand which anomaly types require which tier (point at edge, collective in cloud)
• Statistical Methods - Methods suitable for edge deployment
• Machine Learning Methods - Methods requiring fog/cloud resources

Architecture Patterns:

• Edge Computing Patterns - When to process at edge vs cloud
• Edge and Fog Computing - Gateway-level processing and cloud integration

Data Pipeline Components:

• Stream Processing - Real-time data ingestion and processing
• Message Queuing - MQTT for edge-to-fog data transport
• Data Lakes - Storing historical data for model training

Model Operations:

• Online Learning - Continuous model updates for concept drift
• Model Deployment - Deploying trained models to edge/fog/cloud

Try It Yourself

Experiment 1: Measure Latency at Each Tier

Build a three-tier demo pipeline and measure end-to-end latency:

1. Edge: Python script simulating sensor with Z-score detection
2. Fog: Receive via MQTT, run Isolation Forest, publish alerts
3. Cloud: Store in database, display on dashboard

Inject an anomaly and measure: sensor detection time, MQTT transmission time, fog processing time, cloud visualization time. Which component dominates?

Experiment 2: Simulate Concept Drift

Generate temperature data with gradual baseline shift (22C → 26C over 30 days):

1. Fixed thresholds: Alert when >25C (fails as baseline drifts)
2. Adaptive thresholds: Z-score on 7-day rolling window
3. Model retraining: Retrain Isolation Forest weekly

Which approach maintains accuracy? What is the trade-off in responsiveness vs false positive rate?

Experiment 3: Edge Processing ROI

Calculate bandwidth and cost with and without edge filtering:

• 1,000 sensors at 10 Hz, 100 bytes/reading
• Cloud ingress: $0.09/GB
• Edge device: Raspberry Pi Zero ($10) running Z-score

How many days until edge processing pays for itself? What if sensors run at 1 kHz (vibration monitoring)?

Interactive: Edge Processing ROI Calculator

Adjust the parameters below to explore how edge filtering affects cloud bandwidth costs and payback period.

viewof numSensors = Inputs.range([10, 10000], {value: 1000, step: 10, label: "Number of sensors"})
viewof sampleRate = Inputs.range([1, 1000], {value: 10, step: 1, label: "Sample rate (Hz)"})
viewof bytesPerReading = Inputs.range([10, 500], {value: 100, step: 10, label: "Bytes per reading"})
viewof cloudCostPerGB = Inputs.range([0.01, 0.50], {value: 0.09, step: 0.01, label: "Cloud ingress ($/GB)"})
viewof edgeDeviceCost = Inputs.range([5, 100], {value: 10, step: 5, label: "Edge device cost ($)"})
viewof edgeFilterRate = Inputs.range([0.50, 0.99], {value: 0.90, step: 0.01, label: "Edge filter rate (fraction removed)"})

{
  const rawBytesPerSec = numSensors * sampleRate * bytesPerReading;
  const rawGBPerDay = (rawBytesPerSec * 86400) / (1024 * 1024 * 1024);
  const filteredGBPerDay = rawGBPerDay * (1 - edgeFilterRate);
  const dailySavingsGB = rawGBPerDay - filteredGBPerDay;
  const dailySavingsDollars = dailySavingsGB * cloudCostPerGB;
  const paybackDays = dailySavingsDollars > 0 ? Math.ceil(edgeDeviceCost / dailySavingsDollars) : Infinity;

  const fmt = (n) => n < 0.01 ? n.toExponential(2) : n.toFixed(2);

  return html`<div style="background: var(--bs-light, #f8f9fa); padding: 1rem; border-radius: 6px; border-left: 4px solid #3498DB; font-family: inherit;">
    <p style="margin: 0.3rem 0;"><strong>Raw data rate:</strong> ${fmt(rawBytesPerSec / (1024*1024))} MB/s = <strong>${fmt(rawGBPerDay)} GB/day</strong></p>
    <p style="margin: 0.3rem 0;"><strong>After edge filtering (${(edgeFilterRate * 100).toFixed(0)}% removed):</strong> ${fmt(filteredGBPerDay)} GB/day</p>
    <p style="margin: 0.3rem 0;"><strong>Daily bandwidth saved:</strong> ${fmt(dailySavingsGB)} GB/day</p>
    <p style="margin: 0.3rem 0;"><strong>Daily cost saved:</strong> $${fmt(dailySavingsDollars)}/day</p>
    <p style="margin: 0.5rem 0 0 0; font-size: 1.1em;"><strong>Payback period:</strong> ${paybackDays === Infinity ? "Never (no savings)" : paybackDays + " days"} for a $${edgeDeviceCost} edge device</p>
  </div>`;
}

Challenge: Build a Feedback Loop

Implement operator confirmation feedback:

1. Operator marks alerts as true positive or false positive
2. System recomputes precision/recall weekly
3. If precision <80%, automatically increase threshold
4. If recall <95%, automatically decrease threshold

Test with simulated operator feedback. Does the system converge to optimal thresholds?

Interactive Quiz: Match Concepts

Interactive Quiz: Sequence the Steps

Common Pitfalls

1. Deploying only at one tier

Edge-only detection misses complex multivariate patterns; cloud-only detection introduces unacceptable latency for safety shutdowns. Layer algorithms across all three tiers.

2. Ignoring backpressure

A sudden burst of anomalies can flood the pipeline. Without bounded queues, the system crashes exactly when accurate detection matters most.

3. Skipping the feedback loop

A pipeline without operator feedback never improves. Wire confirmed anomaly and false-positive labels back to retraining jobs so models adapt to changing conditions.

4. Treating alert volume as a success metric

Generating 10,000 alerts per day is not sensitivity — it is misconfiguration. Track the ratio of actionable alerts to total alerts; aim for > 60% actionable before declaring production-ready.

5. Not testing pipeline failure modes

Simulate network partitions, sensor dropouts, and clock skew before going live. A pipeline that silently stops detecting anomalies during a network outage is more dangerous than one that fails loudly.

Label the Diagram

Code Challenge

16.8 Summary

Production anomaly detection requires layered architecture:

• Edge: Fast threshold detection for safety-critical anomalies (<100ms)
• Gateway: ML models for pattern detection (~1s latency)
• Cloud: Deep learning and continuous retraining (~10s latency)

Key Takeaway: Design for concept drift from the start. Implement feedback loops and scheduled retraining to maintain detection accuracy over time.

16.9 What’s Next

If you want to…	Read this
Evaluate detection accuracy with correct metrics	Performance Metrics
Learn the anomaly types your pipeline must handle	Types of Anomalies
Apply statistical methods at the edge tier	Statistical Methods
Deploy ML models at the gateway tier	Machine Learning Approaches
Return to the module overview	Anomaly Detection Overview