Data Storage · Study deck

Data Quality Monitoring

Picture a cold-room chart that suddenly shows forty degrees instead of four.

Data Dora is your guide for this deck.

qualitymonitoring
iotclass.org

After studying this chapter

Learning objectives

You will be able to:

  • define the main IoT data-quality dimensions and connect each one to a detectable failure mode;
  • place cheap validation checks on the ingestion path and expensive checks in near-real-time or batch jobs;
  • design reject, quarantine, and flag routing for low-quality records;
  • explain why range checks do not catch slow sensor drift;
iotclass.org

Major section

Start With the Bad Reading

The store manager needs to know whether the room is unsafe, the unit changed, or the record was copied twice.

  • Saving the value first and asking later can turn a data fault into a costly action.
  • A gateway is a device that joins one network to another.
  • Firmware is the code stored on a device.

Key terms

Machine learning
Machine learning means software that finds patterns from examples.
One bad reading
One bad reading is enough to make a dashboard lie if the storage path accepts it silently.
iotclass.org

Major section

Start With the Bad Reading (continued)

Machine learning means software that finds patterns from examples.

  • None of these can rescue a record whose source, unit, time, and quality are missing.
  • These checks cannot prove that every accepted reading is accurate.
  • One bad reading is enough to make a dashboard lie if the storage path accepts it silently.
iotclass.org

Major section

Quality Is a Storage Contract

A storage system can be fast and still store untrustworthy data.

  • A sensor may send a value outside physical limits, a gateway may replay the same message, a firmware update may rename fields, or a device clock may be hours behind.
  • If those records silently enter the durable path, later systems inherit the error.
  • Suspect but usable records retain a visible quality state, quarantine preserves evidence for review, and rejection stops malformed input.

Why it matters

That routing contract prevents a bad sample from silently becoming trusted history.

Quality gates should route records before they contaminate durable analytics.
Quality gates should route records before they contaminate durable analytics.
iotclass.org

Major section

Monitoring the Quality Monitor

The quality system needs its own meta-metrics.

  • These signals tell you whether the quality gate is healthy before a monthly report, machine-learning feature, or billing export discovers the damage.
  • At scale, three mechanisms keep quality evidence honest.
  • Second, a lateness watermark separates live data from backfill, so old buffered readings recompute affected quality scores and rollups instead of silently changing dashboards.
Put fast checks on the write path and heavier evidence checks outside the critical insert path.
Put fast checks on the write path and heavier evidence checks outside the critical insert path.
iotclass.org

Major section

Concept Relationships

Data Storage and Databases explains how quality gates fit into broader storage contracts.

  • Database Selection Framework helps choose storage roles that support validation, quarantine, and analytical filtering.
  • Sharding Strategies covers how quality checks behave when writes are distributed across partitions.
  • Stream Processing Fundamentals covers validation before records reach durable storage.
iotclass.org

Deck summary

Key takeaways

The store manager needs to know whether the room is unsafe, the unit changed, or the record was copied twice.

  • Machine learning means software that finds patterns from examples.
  • A storage system can be fast and still store untrustworthy data.
  • The quality system needs its own meta-metrics.
  • Data Storage and Databases explains how quality gates fit into broader storage contracts.
iotclass.org

Retrieval practice

Recall check 1 of 6

Data Dora says: answer from memory, then check your reasoning.

Q1A soil sensor reports no readings between 02:00 and 03:00, then resumes normally. Which quality dimension should trigger first?

AAccuracy
BCompleteness
CUniqueness
DConsistency
Show answer

Answer: B The platform expected readings during that hour and did not receive them.

iotclass.org

Retrieval practice

Recall check 2 of 6

Data Dora says: answer from memory, then check your reasoning.

Q2A temperature reading says -273.5 C. The payload is otherwise valid and may indicate sensor failure. What is usually the best response for a predictive-maintenance system?

ASilently discard suspect data without an alert.
BReplace it with the last good value in the raw table
CTreat it as a valid -273.5 C temperature extreme.
DFlag or quarantine the record and alert operations.
Show answer

Answer: D The value should not drive analytics as a normal reading, but it is useful evidence of sensor or firmware failure.

iotclass.org

Retrieval practice

Recall check 3 of 6

Data Dora says: answer from memory, then check your reasoning.

Q3A sensor slowly drifts 2 C high over six months but stays within the allowed -10 C to 50 C range. Which check is most likely to catch it?

ARequired-field validation
BPeer or reference checks
CDuplicate-value detection
DObject-storage lifecycle policy
Show answer

Answer: B Range checks catch impossible values; peer or reference checks catch plausible but wrong values.

iotclass.org

Retrieval practice

Recall check 4 of 6

Data Dora says: answer from memory, then check your reasoning.

Q4A firmware rollout changes payloads from {'temp': 22.5} to {'temperature': 22.5, 'unit': 'C'}, but only half the fleet updates immediately. What should the ingestion layer do?

ADrop all old-format records until every device updates
BStore old and new formats in unrelated tables forever
CUse version-aware parsing and normalize both formats to one canonical schema
DAssume all devices updated because the firmware was released
Show answer

Answer: C Version-aware parsing lets old and new firmware coexist while storage receives a stable canonical record.

iotclass.org

Retrieval practice

Recall check 5 of 6

Data Dora says: answer from memory, then check your reasoning.

Q5Which evidence best proves that a quality gate handles device reconnects safely?

AA screenshot of a dashboard with green status
BA late-arrival test of event and receive time
CA list of database products used by the platform
DA cache warm-up script with no late-arrival test.
Show answer

Answer: B Reconnect behavior is about late data and replay, so the evidence must test event time, receive time, and downstream handling.

iotclass.org

Retrieval practice

Recall check 6 of 6

Data Dora says: answer from memory, then check your reasoning.

Q6A reading is well formed and remains useful for trend review, but its calibration warning means automation must exclude it. How should it be routed?

AFlagged data stays for trends, not automation.
BReject the entire device forever.
CSilently treat it as a normal reading.
DDrop every suspect reading without a trace.
Show answer

Answer: A Flagged storage preserves useful evidence while the quality state prevents it from silently driving automation.

iotclass.org

Print reference

Answers 1 of 2

Answer key.

  1. B · The platform expected readings during that hour and did not receive them.
  2. D · The value should not drive analytics as a normal reading, but it is useful evidence of sensor or firmware failure.
  3. B · Range checks catch impossible values; peer or reference checks catch plausible but wrong values.
  4. C · Version-aware parsing lets old and new firmware coexist while storage receives a stable canonical record.
iotclass.org

Print reference

Answers 2 of 2

Answer key.

  1. B · Reconnect behavior is about late data and replay, so the evidence must test event time, receive time, and downstream handling.
  2. A · Flagged storage preserves useful evidence while the quality state prevents it from silently driving automation.
iotclass.org