Analytics & ML · Study deck
Production ML Monitoring
A leak detector can score well in a notebook and still miss events after a sensor replacement changes its readings.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: The labelled problems belong to different places: data drift changes inputs, concept drift changes the relation between inputs and outcomes, service faults interrupt inference, and feedback gaps hide whether predictions were right.
- Explain: The model service remains healthy and returns scores, yet its input distribution has moved by 0.7 mm/s, or 35% of the old median.
- Explain: A tiny fleet may not produce enough confirmed failures for an hourly precision estimate, while a service-health counter can still be checked each minute.
- Explain: A stable fleet-wide average can hide one small group collapsing.
Major section
Diagnose a Detector That Quietly Drifts
The labelled problems belong to different places: data drift changes inputs, concept drift changes the relation between inputs and outcomes, service faults interrupt inference, and feedback gaps hide whether predictions were right.
- A single uptime graph cannot cover all four.
- After maintenance, the new sensor reports a median of 2.7 mm/s.
Major section
Diagnose a Detector That Quietly Drifts (continued)
The model service remains healthy and returns scores, yet its input distribution has moved by 0.7 mm/s, or 35% of the old median.
- Monitoring should separate this feature shift from an increase in true pump faults.
- Retention and access should match the investigation need.
- An alert needs a response.
Major section
Diagnose a Detector That Quietly Drifts (continued)
If feature ranges move, compare device cohorts and recent firmware or sensor changes.
- Each route has a different owner and rollback choice.
- The old model flags 40 of 1,000 daily records, while the candidate flags 65.
- That difference is 25 records, but it is not automatically improvement or drift.
- Threshold changes alter production behavior.
Major section
Diagnose a Detector That Quietly Drifts (continued)
A percentage based on two labelled events should not look as certain as one based on two thousand.
- A shadow run keeps candidate decisions from controlling equipment while those cases are examined.
- Shadow a candidate threshold, count which records would change class, and inspect their confirmed outcomes.
- The monitoring system also requires monitoring.
Major section
Diagnose a Detector That Quietly Drifts (continued)
A tiny fleet may not produce enough confirmed failures for an hourly precision estimate, while a service-health counter can still be checked each minute.
- If alert volume becomes unmanageable, preserve the score distribution and reason instead of suppressing notices without evidence.
- Rollback can restore service while the team investigates; it does not erase the shifted data or explain its cause.
- A stable fleet-wide average can hide one small group collapsing.
Major section
Start With the Story
Monitor input ranges, warning rates, and missed events against a recorded baseline so a changed model has an owner and rollback path.
- Months later, it starts sending too many warnings.
- The pump may have changed.
- The sensor may have aged.
- The weather may be different.
Major section
Start With the Story (continued)
Under the Hood examines drift, feedback, thresholds, and rollback when the model and the world change together.
- A useful model can lose its fit after release.
- Production use begins when a model affects real work.
- The team must watch the whole path.
- The real world may shift.
Major section
Production ML as Monitored Release
Data drift calls for input and label review before retraining; stale model behavior calls for outcome comparison and a governed update; edge failure requires fallback and rollback; latency spikes may require profiling, pruning, or placement changes.
- This carries forward the chapter's running narrative: production monitoring protects a decision by connecting signals, owners, diagnostic evidence, and reversible action.
Major section
Summary
Production ML monitoring keeps an IoT model release tied to a bounded decision after deployment.
- A reviewable record includes input contract, feature meaning, model behavior, runtime limits, telemetry, feedback, rollback, ownership, and retest triggers.
- Drift can affect data, labels, decisions, runtime behavior, or release control, and each boundary needs different evidence.
- Canary and staged rollouts are useful only when the team records segment evidence and can stop, roll back, or narrow the release.
Deck summary
Key takeaways
The labelled problems belong to different places: data drift changes inputs, concept drift changes the relation between inputs and outcomes, service faults interrupt inference, and feedback gaps hide whether predictions were right.
- The model service remains healthy and returns scores, yet its input distribution has moved by 0.7 mm/s, or 35% of the old median.
- If feature ranges move, compare device cohorts and recent firmware or sensor changes.
- A percentage based on two labelled events should not look as certain as one based on two thousand.
- Under the Hood examines drift, feedback, thresholds, and rollback when the model and the world change together.
Retrieval practice
Recall check 1 of 3
Data Dora says: answer from memory, then check your reasoning.
Q1A trained IoT model passes offline validation and is ready for field rollout. What makes the production release reviewable?
Show answer
Answer: C Offline validation is one evidence source; production needs monitoring, ownership, rollback, and retest boundaries.
Retrieval practice
Recall check 2 of 3
Data Dora says: answer from memory, then check your reasoning.
Q2An edge predictive-maintenance model still meets latency targets, but vibration feature distributions shifted after a sensor remount. What should the production review do first?
Show answer
Answer: C The runtime path is healthy, but the input and feature evidence that justified the model has changed.
Retrieval practice
Recall check 3 of 3
Data Dora says: answer from memory, then check your reasoning.
Q3A canary model reduces false alerts for one device group but increases missed detections for a different operating mode. Which response is strongest?
Show answer
Answer: C The canary revealed a segmented decision boundary, so release control and evidence review should happen before expanding rollout.
Print reference
Answers
Answer key.
- C · Offline validation is one evidence source; production needs monitoring, ownership, rollback, and retest boundaries.
- C · The runtime path is healthy, but the input and feature evidence that justified the model has changed.
- C · The canary revealed a segmented decision boundary, so release control and evidence review should happen before expanding rollout.