Data Storage · Study deck

Data Quality Monitoring

Picture a cold-room chart that suddenly shows forty degrees instead of four.

Data Dora is your guide for this deck.

qualitymonitoring
Data Dora, the module guide, in a scene from this chapter.
iotclass.org

After studying this chapter

Learning objectives

You will be able to:

  • define the main IoT data-quality dimensions and connect each one to a detectable failure mode;
  • place cheap validation checks on the ingestion path and expensive checks in near-real-time or batch jobs;
  • design reject, quarantine, and flag routing for low-quality records;
  • explain why range checks do not catch slow sensor drift;
iotclass.org

Major section

Start With the Bad Reading

The store manager needs to know whether the room is unsafe, the unit changed, or the record was copied twice.

  • Saving the value first and asking later can turn a data fault into a costly action.
  • A gateway is a device that joins one network to another.
  • Firmware is the code stored on a device.

Key terms

Machine learning
Machine learning means software that finds patterns from examples.
One bad reading
One bad reading is enough to make a dashboard lie if the storage path accepts it silently.
iotclass.org

Major section

Start With the Bad Reading (continued)

Machine learning means software that finds patterns from examples.

  • None of these can rescue a record whose source, unit, time, and quality are missing.
  • These checks cannot prove that every accepted reading is accurate.
  • One bad reading is enough to make a dashboard lie if the storage path accepts it silently.
iotclass.org

Major section

Quality Is a Storage Contract

A storage system can be fast and still store untrustworthy data.

  • A sensor may send a value outside physical limits, a gateway may replay the same message, a firmware update may rename fields, or a device clock may be hours behind.
  • If those records silently enter the durable path, later systems inherit the error.

Why it matters

That routing contract prevents a bad sample from silently becoming trusted history.

Quality gates should route records before they contaminate durable analytics.
Quality gates should route records before they contaminate durable analytics.
iotclass.org

Major section

Where Checks Belong

Not every quality check should run inside the write transaction.

  • Recent-history checks then detect gaps and late arrivals, while batch work can compare peers and long-term drift.
Put fast checks on the write path and heavier evidence checks outside the critical insert path.
Put fast checks on the write path and heavier evidence checks outside the critical insert path.
iotclass.org

Major section

Monitoring the Quality Monitor

The quality system needs its own meta-metrics.

  • These signals tell you whether the quality gate is healthy before a monthly report, machine-learning feature, or billing export discovers the damage.
  • At scale, three mechanisms keep quality evidence honest.
  • Second, a lateness watermark separates live data from backfill, so old buffered readings recompute affected quality scores and rollups instead of silently changing dashboards.
iotclass.org

Major section

Concept Relationships

Data Storage and Databases explains how quality gates fit into broader storage contracts.

  • Database Selection Framework helps choose storage roles that support validation, quarantine, and analytical filtering.
  • Sharding Strategies covers how quality checks behave when writes are distributed across partitions.
  • Stream Processing Fundamentals covers validation before records reach durable storage.
iotclass.org

Deck summary

Key takeaways

The store manager needs to know whether the room is unsafe, the unit changed, or the record was copied twice.

  • Machine learning means software that finds patterns from examples.
  • A storage system can be fast and still store untrustworthy data.
  • Not every quality check should run inside the write transaction.
  • The quality system needs its own meta-metrics.
iotclass.org

Retrieval practice

Recall check 1 of 4

Data Dora says: answer from memory, then check your reasoning.

Q1A soil sensor reports no readings between 02:00 and 03:00, then resumes normally. Which quality dimension should trigger first?

ACompleteness
BAccuracy
CUniqueness
DConsistency
Show answer

Answer: A The platform expected readings during that hour and did not receive them.

Q2Complete the routing function for a quality gate.

Aif 'schema_version' not in reading:
Bif reading.get('battery') is None:
Cif reading.get('device_id') == 'test':
Dif reading.get('value') == 0:
Show answer

Answer: A A missing schema version is not safely interpretable, so the record is rejected.

iotclass.org

Retrieval practice

Recall check 2 of 4

Data Dora says: answer from memory, then check your reasoning.

Q3Place each data-quality control where it lives so you can route trustworthy records to storage and isolate suspicious ones.

ASchema Check
BCold Archive
CDashboard Cache
DBackup Restore
Show answer

Answer: A The gate first proves that a record can be interpreted, then assigns quality evidence, and finally separates quarantine from accepted storage so you can preserve both safety and auditability.

Q4A sensor slowly drifts 2 C high over six months but stays within the allowed -10 C to 50 C range. Which check is most likely to catch it?

APeer or reference checks
BRequired-field validation
CDuplicate detection
DObject-storage lifecycle policy
Show answer

Answer: A Range checks catch impossible values; peer or reference checks catch plausible but wrong values.

iotclass.org

Retrieval practice

Recall check 3 of 4

Data Dora says: answer from memory, then check your reasoning.

Q5Why should a data-quality system emit meta-metrics such as points-per-second, quarantine rate, and ingest-lag distribution?

ABecause silent pipeline failures appear first in these meta-metrics.
BBecause meta-metrics replace range, completeness, and duplicate checks.
CBecause they let the team delete raw data without retention review.
DBecause a normal quarantine rate confirms that sensor calibration remains correct.
Show answer

Answer: A Quality gates need their own telemetry so silent pipeline failures are visible before downstream users trust the data.

iotclass.org

Retrieval practice

Recall check 4 of 4

Data Dora says: answer from memory, then check your reasoning.

Q6Which evidence best proves that a quality gate handles device reconnects safely?

AA screenshot of a dashboard with green status
BA list of database products used by the platform
CA cache warm-up script
DA late-arrival test of event and receive time
Show answer

Answer: D Reconnect behavior is about late data and replay, so the evidence must test event time, receive time, and downstream handling.

iotclass.org

Print reference

Answers 1 of 2

Answer key.

  1. A · The platform expected readings during that hour and did not receive them.
  2. A · A missing schema version is not safely interpretable, so the record is rejected.
  3. A · The gate first proves that a record can be interpreted, then assigns quality evidence, and finally separates quarantine from accepted storage so you can preserve both safety and auditability.
  4. A · Range checks catch impossible values; peer or reference checks catch plausible but wrong values.
iotclass.org

Print reference

Answers 2 of 2

Answer key.

  1. A · Quality gates need their own telemetry so silent pipeline failures are visible before downstream users trust the data.
  2. D · Reconnect behavior is about late data and replay, so the evidence must test event time, receive time, and downstream handling.
iotclass.org