Analytics & ML · Study deck

Validation and Outlier Detection

Picture a greenhouse report that says humidity is 108 percent.

Data Dora is your guide for this deck.

dataqualityvalidation
Data Dora, the module guide, in a scene from this chapter.
iotclass.org

After studying this chapter

Learning objectives

You will be able to:

  • Explain: A mobile air-quality study might group 0.38-0.42 um particle counts by library, park walk, traffic street, car, and indoor context.
  • Explain: Hard failures can be rejected for automation, but the original payload, failed rule, quality state, and handling decision should remain available for audit.
  • Explain: Hard rules reject impossible data: malformed payloads, values beyond sensor limits, invalid timestamps, and impossible rates of change.
  • Explain: Treating every warning as a hard reject loses useful evidence; treating every warning as valid poisons downstream analytics.
iotclass.org

Major section

Start With the Story · Validation Stops Bad Data

The message arrived on time and looks tidy, yet using it could trigger the wrong control action.

  • JavaScript Object Notation, or JSON, is a text format for named data fields.
  • A payload means the useful data carried inside a message.
  • Valid structure does not prove that the values make sense.

Key terms

Validation
Validation is the entry gate that decides whether each reading is complete, plausible, timely, and safe enough to use.

Why it matters

It records why a reading passed, failed, or was quarantined so that data quality becomes observable instead of invisible.

Raw reading value + time Schema fields + types Physics range + rate Context peer checks Quality state + action Every failure writes evidence, not just a dropped row.
Start With the Story · Validation Stops Bad Data
iotclass.org

Major section

Rules Plus Outlier Bounds · Qualitative Evidence Has a Validation Workflow

Validation should run in layers.

  • Hard rules reject impossible data: malformed payloads, values beyond sensor limits, invalid timestamps, and impossible rates of change.
  • Treating every warning as a hard reject loses useful evidence; treating every warning as valid poisons downstream analytics.
  • Hard validation protects control decisions.
  • MADUses median absolute deviation.

Why it matters

For example, repeated “false alarm” comments may turn out to describe three different mechanisms: a drifting sensor, an alert without asset context, and an approval rule that prevents timely action.

iotclass.org

Major section

Compare Sensor Agreement · Policy for Failed Data

A Bland-Altman record makes method comparison explicit.

  • When a deployment compares places, people, or measurement methods, validation needs more than a pass/fail rule.
  • A mobile air-quality study might group 0.38-0.42 um particle counts by library, park walk, traffic street, car, and indoor context.
  • Each cell is an observed count.

Why it matters

This matters because a narrow interval is only a precise statement about the sampled design; it is not proof that the sensor is accurate or that the result generalizes to the fleet.

iotclass.org

Major section

Summary · Key Takeaway

Validation protects IoT analytics before data enters dashboards, models, alerts, and control loops.

  • Hard failures can be rejected for automation, but the original payload, failed rule, quality state, and handling decision should remain available for audit.
  • The practical goal is traceable quality state.
  • Each reading should become valid, flagged, quarantined, or rejected for a named reason.
iotclass.org

Deck summary

Key takeaways

The message arrived on time and looks tidy, yet using it could trigger the wrong control action.

  • Validation should run in layers.
  • A Bland-Altman record makes method comparison explicit.
  • Validation protects IoT analytics before data enters dashboards, models, alerts, and control loops.
iotclass.org

Retrieval practice

Recall check 1 of 5

Data Dora says: answer from memory, then check your reasoning.

Q1A reading is quarantined after a range rule fails. What record lets an auditor replay the decision?

AKeep raw value, timestamp, unit, rule version, reason, outcome, and action.
BKeep only a rejection count, losing the reading and rule needed for replay.
CReplace the raw value with a normal one, then call the reading valid.
DKeep only dashboard warning colour, losing the rule and rejected value.
Show answer

Answer: A The rule and evidence explain why this reading was held.

iotclass.org

Retrieval practice

Recall check 2 of 5

Data Dora says: answer from memory, then check your reasoning.

Q2A soil-moisture sensor reports percentage of saturation on a 0-100 percent scale. Which validation rule most directly catches a reading of 980 percent?

ASchema validation, because the number is encoded as a numeric field.
BRange validation, because soil moisture cannot exceed 100 percent.
CModel training, because the model can eventually learn the value is unusual.
DCompression, because high numbers take more bandwidth.
Show answer

Answer: B Physically impossible values are best caught by simple range validation.

iotclass.org

Retrieval practice

Recall check 3 of 5

Data Dora says: answer from memory, then check your reasoning.

Q3A reading is physically impossible for control, while another is unusual but plausible. How should validation treat them?

AReject impossible control input and flag unusual but plausible outliers.
BReplace both with last good values, hiding impossible and plausible cases alike.
CDelete both readings without a trace of the validation decision.
DSend both readings to control before checking physical plausibility.
Show answer

Answer: A Hard rejection protects action; a soft flag preserves uncertain evidence.

iotclass.org

Retrieval practice

Recall check 4 of 5

Data Dora says: answer from memory, then check your reasoning.

Q4With Q1 = 21 and Q3 = 24, what upper fence does the 1.5 * IQR rule use?

A24, using Q3 itself as the fence without the 1.5 IQR addition.
B25.5, adding half the IQR to Q3 rather than one and a half times.
C28.5, adding 1.5 times the IQR of 3 to the upper quartile of 24.
D85, using the largest observed sample rather than the quartile rule.
Show answer

Answer: C IQR = 24 - 21 = 3, so the upper outlier fence is 24 + 1.5 × 3 = 28.5.

iotclass.org

Retrieval practice

Recall check 5 of 5

Data Dora says: answer from memory, then check your reasoning.

Q5Why is silently deleting every reading that fails validation a poor policy?

ABecause a failed value should remain available for direct controller use.
BBecause silent deletion biases results and hides device or rule faults.
CBecause a database cannot delete one failed reading from its records.
DBecause schema checks should wait until after all outliers are scored.
Show answer

Answer: B Silent deletion hides quality evidence and can bias downstream analytics.

iotclass.org

Print reference

Answers

Answer key.

  1. A · The rule and evidence explain why this reading was held.
  2. B · Physically impossible values are best caught by simple range validation.
  3. A · Hard rejection protects action; a soft flag preserves uncertain evidence.
  4. C · IQR = 24 - 21 = 3, so the upper outlier fence is 24 + 1.5 × 3 = 28.5.
  5. B · Silent deletion hides quality evidence and can bias downstream analytics.
iotclass.org