By the end of this chapter, you will be able to:
Think of deploying an ML model like releasing a new employee into a factory. On day one, they perform well because training matched reality. But over weeks and months, things change – new equipment arrives, procedures shift, seasons alter conditions. Without regular check-ins (monitoring), you would not notice the employee struggling until something goes wrong. Production ML monitoring is that regular check-in: it watches how your model performs in the real world and alerts you when things start drifting from expectations.
This is part 7 (final) of the IoT Machine Learning series:
Deploying ML to IoT introduces unique challenges. Unlike cloud ML with direct server access, IoT models run on distributed, often disconnected edge devices.
Production ML monitoring creates a continuous feedback cycle to detect and respond to model degradation:
Stage 1: Baseline Establishment - During initial deployment, record expected ranges for key metrics (accuracy >90%, latency <50ms, feature distributions)
Stage 2: Continuous Telemetry - Edge devices stream inference logs to cloud: prediction confidence, feature values, latency, resource usage
Stage 3: Drift Detection - Compare live feature distributions vs training data using statistical tests (KL divergence, Kolmogorov-Smirnov)
Stage 4: Performance Tracking - Monitor accuracy proxy metrics (confidence distribution, rejection rate) since ground truth labels are rarely available in production
Stage 5: Alerting & Triage - Trigger alerts when metrics exceed thresholds (latency P99 > 200ms, confidence drops 10%, feature drift > 0.3)
Stage 6: Root Cause Analysis - Investigate alerts—sensor degradation? Seasonal shift? New failure mode not in training data?
Stage 7: Remediation - Options include: retrain with recent data, rollback to previous model version, update feature normalization, add new sensor calibration
The cycle repeats continuously—production is not a final state but an ongoing process of adaptation. IoT environments drift (sensors age, usage patterns shift, seasons change), requiring models to evolve.
| Metric Category | Critical Metrics | Alert Threshold |
|---|---|---|
| Model Performance | Inference accuracy, confidence distribution | < 80% baseline, KL divergence > 0.3 |
| Inference Latency | P50/P99 latency, timeout rate | > 100ms / > 500ms, > 1% |
| Resource Usage | CPU, memory, battery drain | > 80% sustained, > 5mW avg |
| Data Quality | Missing features, out-of-range values | > 5% devices, > 1% readings |
Problem: Fall detection achieves 95% accuracy and 99.9% specificity, but generates thousands of false alarms per user per year.
Root Cause: Class imbalance. Falls are rare (~1 per year), but the system evaluates activity events every 10 seconds during 16 waking hours. That yields 5,760 events/day x 365 days = ~2.1M non-fall events per year. With a 0.1% false positive rate (99.9% specificity): 2.1M x 0.001 = 2,100 false alarms per user per year!
Fall Detection Model Performance Metrics:
Quantifying real-world accuracy for elderly care fall detection.
Confusion matrix from 1000 test samples:
Precision (when model predicts fall, how often is it correct?): \[ \text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}} = \frac{45}{45 + 20} = \frac{45}{65} = 0.692 = 69.2\% \]
Recall (of all actual falls, how many are detected?): \[ \text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}} = \frac{45}{45 + 5} = \frac{45}{50} = 0.90 = 90\% \]
F1-score (harmonic mean balancing precision and recall): \[ F_1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} = 2 \times \frac{0.692 \times 0.90}{0.692 + 0.90} = 0.783 = 78.3\% \]
Specificity (of all non-fall events, how many are correctly classified?): \[ \text{Specificity} = \frac{\text{TN}}{\text{TN} + \text{FP}} = \frac{930}{930 + 20} = \frac{930}{950} = 0.979 = 97.9\% \]
Note: This test-set specificity of 97.9% appears strong, but at scale the 2.1% false positive rate produces unacceptable alarm volumes. Production deployment requires specificity above 99.9%. For safety-critical systems, minimizing False Negatives (FN) is also paramount – a 90% recall means 10% of falls go undetected, which may be unacceptable for high-risk patients.
Adjust the parameters below to see how specificity, check frequency, and waking hours affect false alarm volume. This demonstrates why even 99.9% specificity produces unacceptable alarm rates for high-frequency monitoring.
Results: False positives reduced from ~2,100 to ~1.5 per user per year (acceptable!)
| Tool | Best For | IoT Support | Key Features |
|---|---|---|---|
| TensorBoard | Training metrics | Limited | Visualization, profiling |
| MLflow | Experiment tracking | Good | Model versioning |
| Prometheus + Grafana | Infrastructure | Excellent | Time-series, alerting |
| Edge Impulse | TinyML | Native | Device profiling, OTA |
Before deploying ML models to IoT devices:
Scenario: A water utility operates 200 industrial pumps with vibration sensors (4 kHz accelerometer). Goal: Detect failures 2-4 weeks before they occur.
import numpy as np
from scipy import stats
from scipy.fftpack import fft
def extract_vibration_features(raw_signal, sample_rate=4000):
"""
Extract predictive maintenance features from vibration.
Input: 1 second of raw accelerometer (4000 samples)
Output: 7 key features capturing mechanical health
(full pipeline extracts 18 features with additional bands)
"""
features = {}
# Time-domain features
features['rms'] = np.sqrt(np.mean(raw_signal**2))
features['peak'] = np.max(np.abs(raw_signal))
features['crest_factor'] = features['peak'] / features['rms']
features['kurtosis'] = stats.kurtosis(raw_signal)
# Frequency-domain features
fft_vals = np.abs(fft(raw_signal))[:len(raw_signal)//2]
freqs = np.fft.fftfreq(len(raw_signal), 1/sample_rate)[:len(raw_signal)//2]
# Spectral energy in bands
features['energy_0_500hz'] = np.sum(fft_vals[(freqs < 500)]**2)
features['energy_500_1000hz'] = np.sum(fft_vals[(freqs >= 500) & (freqs < 1000)]**2)
# Bearing fault frequencies
rpm = 1800
bpfo = 0.4 * (rpm/60) * 9 # Ball Pass Frequency Outer
features['bpfo_energy'] = np.sum(fft_vals[(freqs >= bpfo-5) & (freqs <= bpfo+5)]**2)
return featuresFeature importance from domain knowledge:
| Feature | Physical Meaning | Fault Correlation |
|---|---|---|
| RMS | Overall vibration | General degradation (r=0.85) |
| Kurtosis | Impulsiveness | Bearing pitting (r=0.92) |
| Crest factor | Peak-to-RMS | Localized damage (r=0.78) |
| BPFO energy | Bearing fault frequency | Outer race wear (r=0.95) |
With only 12 failures vs 400,000+ hours of normal data, supervised classification would overfit. Use semi-supervised anomaly detection.
| Model | Approach | Strengths | Best For |
|---|---|---|---|
| Isolation Forest | Tree-based isolation | Fast, handles high-dim | General anomalies |
| One-Class SVM | Boundary around normal | Robust to outliers | Small datasets |
| LSTM-Autoencoder | Temporal reconstruction | Captures sequences | Time-series |
Selected: Ensemble of Isolation Forest + LSTM-Autoencoder
| Event | Cost |
|---|---|
| Unplanned failure | $15,000 (repair + downtime) |
| Planned maintenance (true positive) | $3,000 (scheduled repair) |
| False alarm inspection | $500 (technician visit) |
Threshold analysis:
| Threshold | Recall | FP/year | FN/year | Annual Cost |
|---|---|---|---|---|
| 0.70 | 100% | 80 | 0 | $76,000 |
| 0.80 | 95% | 35 | 0.6 | $60,700 |
| 0.85 | 92% | 18 | 1 | $57,000 |
| 0.90 | 83% | 8 | 2 | $64,000 |
Selected: 0.85 - Optimal cost with 92% recall
Edge gateway specs:
def monthly_model_update():
"""
1. Collect new data
2. Validate data quality
3. Retrain models
4. A/B test new model
5. Deploy if improved
"""
# Step 1: Data collection
new_normal_data = fetch_last_month_normal_data()
new_failure_data = fetch_confirmed_failures() # Added to holdout set
# Step 2: Data quality checks
assert len(new_normal_data) > 10000
assert check_feature_drift(new_normal_data) < 0.3
# Step 3: Retrain
combined_normal = np.vstack([historical_normal_data, new_normal_data])
new_iso_forest = IsolationForest(n_estimators=100)
new_iso_forest.fit(combined_normal)
# Step 4: A/B validation
old_recall = evaluate_model(current_model, holdout_failures)
new_recall = evaluate_model(new_iso_forest, holdout_failures)
# Step 5: Deploy if improved
if new_recall >= old_recall - 0.05:
deploy_to_edge_gateways(new_iso_forest)Drift detection thresholds:
| Metric | Threshold | Action |
|---|---|---|
| Feature distribution shift | KL > 0.3 | Investigate |
| False positive rate increase | > 50% | Retrain |
| Missed failure | Any | Immediately retrain |
| Metric | Value |
|---|---|
| Failure detection rate | 92% (11/12 failures caught) |
| Lead time | 2-4 weeks before failure |
| False positive rate | 18/year (< 2/pump/year) |
| Annual cost reduction | $123,000 (68% savings) |
| ROI | 820% (cost: $15,000 development) |
Deploying updated models to thousands of edge devices carries significant risk. A flawed model update can degrade performance across an entire fleet before teams detect the problem.
Unlike web services where a bad deployment can be rolled back in seconds, IoT devices may be offline, on cellular connections, or physically inaccessible. A full fleet push of a broken model to 10,000 devices could take weeks to remediate.
Canary deployment strategy for the water utility (200 pumps):
| Phase | Devices | Duration | Success Criteria | Rollback Time |
|---|---|---|---|---|
| Canary | 5 pumps (2.5%) | 7 days | FP rate < 25/year, no missed failures | 15 minutes (OTA) |
| Early adopter | 20 pumps (10%) | 14 days | FP rate within 20% of baseline | 2 hours |
| Broad rollout | 100 pumps (50%) | 14 days | All metrics within baseline tolerance | 4 hours |
| Full fleet | 200 pumps (100%) | Permanent | Continuous monitoring | 8 hours |
Selection criteria for canary devices: Choose pumps that represent the full operating range – different ages (2-year-old and 15-year-old), different loads (50% and 95% capacity), and different environments (indoor climate-controlled and outdoor exposed). A model that works on new indoor pumps may fail on weathered outdoor units.
Tesla deploys ML model updates to its vehicle fleet using a staged rollout approach. In 2021, a vision model update for Autopilot was pushed to approximately 2,000 vehicles in the “early access” program before fleet-wide release. Testers identified false braking events on specific road geometries (concrete overpass shadows interpreted as obstacles) that affected 0.3% of drives. Tesla refined the model and eliminated the issue before the broader rollout to 1.5 million vehicles.
The key insight: the 0.3% failure rate would have generated approximately 45,000 false braking events per day across the full fleet – potentially causing rear-end collisions. Canary testing on 2,000 vehicles caught it at approximately 60 events total, with no reported incidents.
The Mistake: Treating deployment as the final step without continuous monitoring for model degradation, data drift, or silent failures.
Why It Happens: Once accuracy looks good in testing, teams move on. ML monitoring is less established than application monitoring.
The Fix:
Warning sign: If you cannot answer “what was our model accuracy last week?”, you are operating blind.
## Concept Relationships
Production ML completes the IoT ML series by addressing deployment reality:
The critical insight is that deployment is not the end—it’s the beginning of a continuous adaptation cycle where models must evolve as IoT environments change.
Related Chapters:
External Resources:
Hands-On Challenge: Simulate and detect concept drift in a deployed model
Task: Build a simple drift detection system for temperature anomaly detection:
What to Observe:
Bonus: Add alarm fatigue simulation—too-sensitive thresholds generate false alarms, reducing operator trust.
Every model deployment must have a defined rollback procedure that can be executed in under 5 minutes if the new model degrades production metrics. Test the rollback procedure in a staging environment before production deployment.
ML model accuracy in IoT production environments degrades over time due to sensor aging, equipment changes, and seasonal variation. Implement continuous monitoring with automated alerts when key metrics cross SLO thresholds.
A test set that guides model selection for multiple deployment cycles gradually becomes part of the implicit training process. Hold back a final evaluation set that is used at most once, and generate new test sets from recent production data for ongoing evaluation.
Frequent model updates (daily retraining) require robust CI/CD pipelines, automated validation gates, and fleet-wide OTA update infrastructure. Design for the update cadence required before committing to a retraining schedule.
This chapter covered production ML for IoT:
Key Insight: Production ML requires as much engineering as model development. Monitoring, drift detection, and automated retraining are essential for long-term success.
Production ML for IoT requires continuous monitoring because model accuracy degrades silently over time as real-world data drifts away from training conditions. A fall detection system with 95% accuracy and 99.9% specificity still generates thousands of false alarms per user per year due to the base rate problem – solving this requires multi-stage filtering pipelines and cost-based threshold tuning, not just better models. If you cannot answer “what was our model accuracy last week?”, you are operating blind.
What happens AFTER you build a smart brain? The Sensor Squad learns about babysitting ML models!
The Sensor Squad has built an amazing brain that detects when grandma falls down. It is 95% accurate! Time to celebrate, right?
“Not so fast!” warns Max the Microcontroller. “We need to BABYSIT this brain forever!”
Problem 1: Too Many False Alarms! Sammy the Sensor checks the math: “Grandma does THOUSANDS of movements every day – standing up, sitting down, reaching for things. That is over TWO MILLION movement checks per year. Even with 99.9% accuracy, that is over 2,000 times the brain says ‘FALL!’ when grandma is actually just bending down to pet the cat!”
“2,000 false alarms?!” gasps Lila the LED. “Grandma would throw us out the window!”
Solution: They build a THREE-STAGE filter: 1. First check: Is the acceleration really high? (catches obvious non-falls) 2. Second check: Did grandma stay on the ground for 3 seconds? (pets don’t keep you down) 3. Third check: Did her heart rate change? (real falls cause stress)
Now they get only 1.5 false alarms per YEAR!
Problem 2: The Brain Gets Stale! After 6 months, the brain starts making more mistakes. Why? Because grandma got a new walking cane! The brain was never taught what “walking with a cane” looks like.
“This is called DRIFT,” explains Bella the Battery. “The real world changes, but our brain stays the same. We need to retrain it with new data!”
Problem 3: How Do We Know If Something Is Wrong? Max sets up a monitoring dashboard – like a report card for the brain: - How many predictions per day? - What percentage are confident? - Are there sudden changes?
“If Tuesday looks totally different from Monday, something is wrong!” says Max. “Maybe Sammy got dirty, or grandma started a new exercise routine.”
The Lesson: Building the brain is only HALF the work. Watching it, fixing it, and updating it is the OTHER half!
Write down a rule for predicting if you need a jacket: “If the temperature is below 15C, wear a jacket.” Follow this rule for a month. Did the rule ever get it wrong? Maybe it was 18C but super windy, and you wished you had a jacket! That is “drift” – your simple rule does not account for everything. Real ML systems face the same problem and need regular updates.
| Direction | Chapter | Focus |
|---|---|---|
| Explore | Multi-Sensor Data Fusion | Combining multiple sensor streams for robust estimates |
| Previous | Feature Engineering | Feature design for IoT ML models |
| Related | Stream Processing | Real-time data processing architectures |