Analytics & ML · Study deck
Production ML Monitoring
A leak detector can monitoring score well in a notebook and still miss events after a sensor replacement changes its readings.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: Data drift calls for input and label review before retraining; stale model behavior calls for outcome comparison and a governed update; edge failure requires fallback and rollback; latency spikes may require profiling, pruning, or placement changes.
- Explain: The labelled problems belong to different places: data drift changes inputs, concept drift changes the relation between inputs and outcomes, service faults interrupt inference, and feedback gaps hide whether predictions were right.
- Explain: The production model service remains healthy and returns scores, yet its input distribution has moved by 0.7 mm/s, or 35% of the old median.
Major section
Diagnose a Detector That Quietly Drifts
The labelled problems belong to different places: data drift changes inputs, concept drift changes the relation between inputs and outcomes, service faults interrupt inference, and feedback gaps hide whether predictions were right.
- A single uptime graph cannot cover all four.
- After maintenance, the new sensor reports a median of 2.7 mm/s.
Major section
Diagnose a Detector That Quietly Drifts (continued)
The production model service remains healthy and returns scores, yet its input distribution has moved by 0.7 mm/s, or 35% of the old median.
- Monitoring should separate this production feature shift from an increase in true pump faults.
- Retention and access should match the investigation need.
- An monitoring alert needs a response.
Major section
Diagnose a Detector That Quietly Drifts (continued)
If production feature ranges move, compare device cohorts and recent firmware or sensor changes.
- Each route has a different owner and rollback choice.
- The old production model flags 40 of 1,000 daily records, while the candidate flags 65.
- That difference is 25 records, but it is not automatically improvement or drift.
Major section
Diagnose a Detector That Quietly Drifts (continued)
A shadow run keeps candidate decisions from controlling equipment while those cases are examined.
- A tiny fleet may not produce enough confirmed failures for an hourly precision estimate, while a service-health counter can still be checked each minute.
- Threshold changes alter production behavior.
- A stable fleet-wide average can hide one small group collapsing.
Major section
Diagnose a Detector That Quietly Drifts (continued)
The monitoring system also requires monitoring.
- A percentage based on two labelled events should not look as certain as one based on two thousand.
- Shadow a candidate threshold, count which records would change class, and inspect their confirmed outcomes.
- Missing that canary means performance charts may be stale even when they still render.
Major section
Start With the Story
A device has produced messy evidence, an analytic step is about to change an alert or control decision, and someone has to explain why the result should be trusted.
- Months later, it starts sending too many warnings.
- The pump may have changed.
- The sensor may have aged.
- The weather may be different.
Major section
Start With the Story (continued)
Under the Hood examines drift, feedback, thresholds, and rollback when the model and the world change together.
- A useful model can lose its fit after release.
- Production use begins when a model affects real work.
- The team must watch the whole path.
- The real world may shift.
Major section
Production ML as Monitored Release
Data drift calls for input and label review before retraining; stale model behavior calls for outcome comparison and a governed update; edge failure requires fallback and rollback; latency spikes may require profiling, pruning, or placement changes.
- This carries forward the chapter's running narrative: production monitoring protects a decision by connecting signals, owners, diagnostic evidence, and reversible action.
Major section
Summary
Production ML monitoring keeps an IoT model release tied to a bounded decision after deployment.
- A reviewable record includes input contract, feature meaning, model behavior, runtime limits, telemetry, feedback, rollback, ownership, and retest triggers.
- Drift can affect data, labels, decisions, runtime behavior, or release control, and each boundary needs different evidence.
- Canary and staged rollouts are useful only when the team records segment evidence and can stop, roll back, or narrow the release.
Deck summary
Key takeaways
The labelled problems belong to different places: data drift changes inputs, concept drift changes the relation between inputs and outcomes, service faults interrupt inference, and feedback gaps hide whether predictions were right.
- The production model service remains healthy and returns scores, yet its input distribution has moved by 0.7 mm/s, or 35% of the old median.
- If production feature ranges move, compare device cohorts and recent firmware or sensor changes.
- A shadow run keeps candidate decisions from controlling equipment while those cases are examined.
- The monitoring system also requires monitoring.
Retrieval practice
Recall check 1 of 3
Data Dora says: answer from memory, then check your reasoning.
Q1A trained IoT model passes offline validation and is ready for field rollout. What makes the production release reviewable?
Show answer
Answer: C Offline validation is one evidence source; production needs monitoring, ownership, rollback, and retest boundaries.
Retrieval practice
Recall check 2 of 3
Data Dora says: answer from memory, then check your reasoning.
Q2An edge predictive-maintenance model still meets latency targets, but vibration feature distributions shifted after a sensor remount. What should the production review do first?
Show answer
Answer: A The runtime path is healthy, but the input and feature evidence that justified the model has changed.
Retrieval practice
Recall check 3 of 3
Data Dora says: answer from memory, then check your reasoning.
Q3A canary model reduces false alerts for one device group but increases missed detections for a different operating mode. Which response is strongest?
Show answer
Answer: C The canary revealed a segmented decision boundary, so release control and evidence review should happen before expanding rollout.
Print reference
Answers
Answer key.
- C · Offline validation is one evidence source; production needs monitoring, ownership, rollback, and retest boundaries.
- A · The runtime path is healthy, but the input and feature evidence that justified the model has changed.
- C · The canary revealed a segmented decision boundary, so release control and evidence review should happen before expanding rollout.