Analytics & ML · Study deck
Cloud Data: Quality and Security
Picture a cold store that sends a high-temperature alert to a duty manager.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: A defensible policy names hot, warm, cold, aggregate, and deletion windows, plus who can place legal hold or privacy hold on a dataset.
- Explain: Security should bind device identity, transport encryption, encryption at rest, IAM, network segmentation, key management, and access logging to the data contract.
- Explain: Cloud IoT data is useful only when it is trustworthy enough to act on and protected enough to share responsibly.
- Explain: The third gate is statistical and physical validation: values are plausible, rate-of-change is reasonable, and related sensors agree.
Major section
Start With the Story
The number arrives in a shared data service, but its unit is missing and the device clock is wrong.
- The data team must decide whether to warn staff, hold the alert, or ask for a fresh reading.
- A neat chart cannot repair lost meaning.
- This opening does not choose storage or prove that every reading is true.
Major section
Trust Needs Quality and Security
Cloud IoT data is useful only when it is trustworthy enough to act on and protected enough to share responsibly.
- Data quality asks whether readings are accurate, complete, consistent, timely, and traceable.
- Data security asks whether the data is confidential, intact, available, and accessible only to identities with a legitimate need.
- Archiving retains protected evidence and 8.
Major section
Trust Needs Quality and Security (continued)
Without that evidence, a system can be fast and polished while still being wrong or unsafe.
- These concerns should be designed into the pipeline, not patched into dashboards after release.
- Storage pairs AES-256 with key management, while 5.
- Sharing add privacy and RBAC/ABAC controls.
- Controlled model and dashboard deployment with approval records.
Major section
Build Validation Release Gates
A cloud IoT pipeline should treat data quality as a release gate.
- The first gate is technical correctness: payload decodes, required fields exist, timestamps parse, and types match the schema.
- The second gate is format consistency: units, clocks, coordinate systems, and category names are canonical.
- Each failed gate should produce evidence rather than silent deletion.
Major section
Build Validation Release Gates (continued)
The third gate is statistical and physical validation: values are plausible, rate-of-change is reasonable, and related sensors agree.
- The fourth gate is completeness and freshness: expected devices and windows have arrived within the decision deadline.
- A rejected reading is often the first sign of a firmware problem, calibration drift, network corruption, or attempted data injection.
- statistical validation: range check: 29.4 C is above 26 C z-score: (29.4: 22) / 2 = 3.7.
Major section
Security as Data Control
Security controls are strongest when they are tied to the data contract.
- Stored data should use managed keys with rotation policy.
- Service accounts should have least privilege for specific topics, tables, and object prefixes.
- Sensitive fields should be classified, masked, tokenized, aggregated, or withheld according to purpose.
- Every human and service access should leave a reviewable log.
Major section
Security as Data Control (continued)
Retention is part of the same trust model.
- A defensible policy names hot, warm, cold, aggregate, and deletion windows, plus who can place legal hold or privacy hold on a dataset.
- customer portal view: customer-specific detail only for authenticated customer and approved support role.
- design reading: Minimization is not the same as deleting all data.
Major section
Security as Data Control (continued)
Encryption TLS in transit, managed encryption at rest, KMS/HSM key ownership, rotation policy, and key-use logging.
- Recovery Backups, replay offsets, restore drills, incident roles, RTO/RPO targets, and immutable audit records.
- Security and privacy reviews should therefore look at data products, not just infrastructure.
- A firewall rule cannot tell whether a dashboard is exposing raw location history to the wrong role.
- Those decisions belong in the data contract and must be verified before release.
Major section
Summary
Cloud IoT data needs both quality controls and security controls before it can safely support decisions.
- Validation should check schema, type, unit, timestamp, range, freshness, duplicates, calibration, and cross-sensor consistency.
- Provenance should record source, offsets, schema versions, transformation versions, quality outcomes, table versions, and release evidence.
- Security should bind device identity, transport encryption, encryption at rest, IAM, network segmentation, key management, and access logging to the data contract.
Deck summary
Key takeaways
The number arrives in a shared data service, but its unit is missing and the device clock is wrong.
- Cloud IoT data is useful only when it is trustworthy enough to act on and protected enough to share responsibly.
- Without that evidence, a system can be fast and polished while still being wrong or unsafe.
- A cloud IoT pipeline should treat data quality as a release gate.
- The third gate is statistical and physical validation: values are plausible, rate-of-change is reasonable, and related sensors agree.
Retrieval practice
Recall check 1 of 3
Data Dora says: answer from memory, then check your reasoning.
Q1Which evidence best supports a trusted cloud IoT analytics result?
Show answer
Answer: C Trusted analytics require source, validation, lineage, access, retention, and audit evidence.
Retrieval practice
Recall check 2 of 3
Data Dora says: answer from memory, then check your reasoning.
Q2An office HVAC reading converts from 85 F to 29.4 C, above the expected 18 C to 26 C range with z-score 3.7. What should the pipeline do?
Show answer
Answer: B Suspicious data should be flagged and traceable, not silently deleted or blindly accepted.
Retrieval practice
Recall check 3 of 3
Data Dora says: answer from memory, then check your reasoning.
Q3Why should smart-meter billing, grid-balancing, and customer-portal datasets be separated?
Show answer
Answer: A Data minimization creates separate data products with purpose-specific detail, access, retention, and audit rules.
Print reference
Answers
Answer key.
- C · Trusted analytics require source, validation, lineage, access, retention, and audit evidence.
- B · Suspicious data should be flagged and traceable, not silently deleted or blindly accepted.
- A · Data minimization creates separate data products with purpose-specific detail, access, retention, and audit rules.