Data Storage · Study deck
Storage Roles and Data Lifecycle
Picture a freezer alert whose chart changes after a late reading arrives.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
Follow a record from ingestion through its owned history and any rebuilt views.
- Each data shape needs an owned storage role.Device identity belongs in a registry, while timestamped measurements, alert events, current values, and camera artifacts have different query and lifecycle contracts.
- Authoritative history must remain behind a rebuildable cache.The greenhouse dashboard can use one current row per node, but its latest values need recovery from accepted durable telemetry.
- Quality checks must precede the durable write.Identity, schema, timestamps, units, bounds, and duplicate handling determine whether an incoming reading can enter trusted history.
- Retention needs evidence that archived history remains usable.A retained day must restore into the expected schema with checksums, permissions, metadata links, and a working sample query.
Major section
Start With the Storage Mess
A changing freezer chart needs a traceable history behind its latest display.
- A late reading can change the chart behind an earlier alert.The freezer investigation needs the original event and an explanation of which accepted copy now drives the decision.
- Every record needs identity, source time, arrival time, and ownership.A retention rule also belongs to that record before the team chooses a database product.
- Normal, late, duplicate, and corrected readings need traceable results.The chapter’s test follows each input through stored copies, query order, age, and restore behaviour across one complete record path.
- Stored copies and query results must support another reviewer’s replay.The evidence should make the same history recoverable without treating one successful test as proof against every possible data loss.
Major section
Overview: Storage Is A Set Of Roles
Start at validation, then follow accepted history into the derived views and tested archive path.
- Validation separates rejected readings from accepted telemetry history.The greenhouse reading must pass its quality contract; a rejected input goes to quarantine instead of becoming trusted history.
- Accepted history and the device registry have separate responsibilities.The measurement belongs in telemetry, while the registry supplies device identity, ownership, placement, firmware, credentials, and calibration state.
- Latest values and rollups are derived from authoritative history.The dashboard cache and summaries must be rebuildable from accepted readings rather than becoming independently edited sources of truth.
- The archive needs a tested return path into usable history.A stored copy is trustworthy only when restore, decoding, checksums, permissions, and the relevant query can be demonstrated.
Major section
Registry, telemetry, events, and artifacts
Different records need different owners and query contracts.
- The registry owns identity, placement, firmware, credentials, and calibration state.Those device facts are separate from measurements so ownership and firmware review do not depend on searching the telemetry stream.
- Timestamped measurements need telemetry history and a retention policy.Minute-by-minute greenhouse readings require append-friendly writes and time-window queries rather than only a latest-value row.
- An alert event needs the change, time, and producing rule.That context must explain why the alert occurred when a later customer-support investigation queries the original event.
- Camera images need object storage and searchable metadata.Checksums, access policy, and retention rules must accompany the binary artifact so it remains identifiable and usable later.
Major section
240 devices and 345,600 daily readings
Use the greenhouse example to separate registry size from telemetry arrival volume.
- The greenhouse has 240 nodes reporting once per minute.The telemetry calculation needs each node’s repeated measurements across the full day rather than a count of devices alone.
- The daily telemetry total is 345,600 readings.The chapter’s calculation is 240 × 60 × 24, showing the arrival volume that the historical write path must support.
- The registry has 240 device records plus identity and ownership information.Owner, zone, firmware, credentials, and calibration describe the devices; they are not additional minute-by-minute temperature measurements.
- Historical writes need partitions, late-arrival handling, and retention rules.The append-heavy telemetry workload must remain queryable by time window as the greenhouse fleet accumulates daily readings.
Activity 1 · Work it out
✎ Size the greenhouse history
I want you to calculate the history before choosing its storage tier.
On paper, use 240 nodes sending one reading per minute to calculate daily readings: 240 × 60 × 24. Contrast that result with the 240 registry records. Explain why a latest-value cache cannot replace history.
3 minutes · Pen and paper · Answer: Activity 1
Major section
Current values and the 30-day hot tier
Retention arithmetic must lead to a working recovery path.
- A latest-value cache can keep one current row per node.The greenhouse dashboard can read that smaller view without making it the only surviving copy of measurement state.
- The cache must rebuild from each device’s newest accepted telemetry point.A cleared or restarted cache needs durable history as its source rather than invented current readings.
- The greenhouse hot tier keeps raw readings for 30 days.The retention plan must say which detail later becomes a rollup or archive and which queries still depend on raw history.
- The hot window has about 10.4 million readings.The hot tier must hold the chapter’s 345,600 daily readings across 30 days before rollup or archive reduces the raw-history burden.
Major section
Practitioner: Build The Storage Evidence Record
Document the complete storage contract before comparing database products.
- Each fact needs a classification and an authoritative store.Registry, telemetry, events, latest values, artifacts, and archives must have clear responsibility before database products are compared.
- The write contract identifies how each record reaches storage.A record can be appended, transactional, derived, cached, imported, or archived, with different ownership and recovery implications.
- Representative queries must show how the service uses its records.Device lookups, time-window scans, latest-value reads, incident searches, and restore queries need captured evidence instead of a guessed access pattern.
- The full contract needs quality gates, lifecycle rules, and demonstrated recovery.Backup, restore, replay, checksums, and derived-store rebuilds must confirm the ownership and meaning established at ingestion.
Major section
Quality gates and conflicting copies
Quality evidence must explain both accepted records and rejected inputs.
- Accepted records need schema, timestamp, unit, identity, and bounds checks.Duplicate keys and quarantine behaviour are part of the quality contract before the reading becomes durable trusted history.
- Rejected messages need quarantine evidence that explains the failure.A malformed or late input must not be silently inserted and later treated as trustworthy history without its acceptance contract.
- Derived copies must not acquire independent ownership of the same fact.Editing registry, cache, analytics index, and export files separately can leave the system with conflicting versions of one device fact.
- Firmware, unit, timestamp, validation, or protocol changes require retesting.A changed ingestion contract can invalidate earlier acceptance and rejection evidence even when the database product remains the same.
Major section
Archive evidence before hot-copy removal
Moving old records is safe only when the retained data remains usable.
- The lifecycle plan must identify hot records, summaries, archives, and deletion.The greenhouse’s 30-day raw window needs explicit rules for what later queries can still recover in detail.
- Archive records need locations, checksums, retention rules, and schemas.Cold history also needs metadata links, restore permissions, and query instructions so the original meaning remains recoverable.
- Hot-copy removal needs successful restore, decoding, permissions, and query checks.A sample archived day must come back into the expected schema rather than merely existing as an uploaded object.
- A successful upload cannot establish usable recovery.The lifecycle evidence must show that retained history can be restored and joined to the relevant device metadata.
Major section
Queries that match dashboards and incidents
Query evidence should represent the work the service actually performs.
- Real dashboard, API, alert, and incident access patterns must guide assessment.Optimising a guessed query can leave production dashboards and customer-support investigations without the storage behaviour they need.
- Query proof needs index or partition evidence and latency samples.The record must include a load-test result alongside representative live and historical queries for the intended service.
- Migrations, retention changes, and partition changes need query retests.An earlier working time-window scan cannot automatically establish the same behaviour after the storage contract changes.
- Fleet, analytics, and incident-workflow changes require record updates.New access patterns or a larger fleet can change the query evidence needed even when the stored record shape looks familiar.
Major section
Under The Hood: Storage Handoffs And Failure Modes
Trusted history depends on the checks that happen before the database write.
- Device authentication and payload decoding must precede record acceptance.Schema validation then decides whether the incoming message belongs in accepted history or the rejected-input path.
- Durable writes need normalised times, units, metrics, IDs, and duplicate keys.These handoffs must preserve the same identity and meaning rather than leaving each derived store to interpret the payload differently.
- Event time and receive time have different explanatory roles.The first describes physical history, while the second helps explain network delay, offline buffering, clock drift, and late arrival.
- Durable timestamps need UTC, with local time applied for display.The storage contract must preserve the original timing evidence while allowing users to view a local-time presentation.
Major section
An 18-minute outage returns 1,440 readings
Delayed uploads need enough identity and timing information to preserve the original history.
- The gateway has 80 meters to serve during an 18-minute outage.The meters can keep sampling once per minute while the missing uplink delays their arrival at the platform.
- The recovery backlog has 1,440 delayed readings.The chapter’s calculation is 80 × 18, so the reconnecting gateway delivers a backlog rather than 1,440 newly sampled values.
- Each record needs both timestamps and a sequence or idempotency key.Stable device identity, schema version, and a quality flag also belong to the handoff before the durable write.
- A delayed 10:05 reading must remain distinct from a live 10:23 reading.Without the event-time and receive-time contract, dashboards can silently mix physical history with the order in which records arrived.
Activity 2 · Predict
✎ Recover a cleared cache
I want you to rebuild the dashboard without inventing fresh readings.
On paper, clear the latest-value cache for 600 devices. Predict how to rebuild it from durable history. The query finds 12 devices without a valid sample in the last hour; explain what their dashboard state should show.
3 minutes · Pen and paper · Answer: Activity 2
Major section
Cache rebuilds and explicit stale states
A derived view needs a named source and a tested rebuild procedure.
- Current views must recover from accepted durable records.Eviction, restart, or schema change requires the newest accepted point per device rather than treating the cleared cache as lost authoritative history.
- Every derived copy needs a source, rebuild query, exception route, and owner.A retest trigger completes the recovery contract so later changes do not silently inherit an outdated rebuild result.
- Missing recent samples require explicit stale or unknown states.The chapter’s 600-device rebuild finds 12 devices without a valid sample in the last hour, which is operational evidence to retain.
- Restored archives must decode and reconnect with device metadata.A cold-history copy needs schemas, metadata links, permissions, checksums, and query instructions to become usable evidence again.
Major section
Summary
Storage reliability comes from clear ownership and tested handoffs.
- Registry, telemetry, events, current views, and artifacts need separate contracts.A greenhouse device record describes ownership and firmware, while its repeated temperature readings create a much larger historical workload.
- Trusted durable history requires validated and normalised records.Identity, timestamps, units, schema, duplicate handling, and rejection evidence must be settled before a reading is accepted.
- Caches and summaries must derive from replayable authoritative data.The latest-value display needs a tested rebuild and explicit stale states when accepted history has no recent sample.
- Retention needs restore, checksum, schema, permission, and query evidence.A sample archived greenhouse day must remain usable after hot-tier removal rather than merely surviving as an uploaded object.
Deck summary
Key takeaways
Keep enough evidence to find a reading again and explain which copy is authoritative.
- Record meaning, queries, ownership, and lifecycle determine storage roles.Choosing a database name first cannot establish which store owns the accepted measurement or how a later incident query recovers its history.
- Late-arrival interpretation needs source and receive timestamps.The gateway’s delayed 10:05 reading must not be presented as a live 10:23 measurement simply because it arrives after reconnecting.
- Current dashboard values must remain rebuildable from accepted history.A cleared cache for 600 devices needs durable-record recovery, with stale or unknown states for the 12 devices lacking valid recent samples.
- Storage-contract changes require recovery and representative-query checks.Migration, retention, partitions, schema, or new service access patterns can invalidate earlier evidence without changing the basic record’s name.
Retrieval practice
Recall check 1 of 3
Data Dora says: answer from memory, then check your reasoning.
Q1A platform stores device owner, model, firmware version, deployment site, and active certificate state. Which storage role should own those facts?
Show answer
Answer: C Registry data is structured identity state, not high-volume telemetry.
Retrieval practice
Recall check 2 of 3
Data Dora says: answer from memory, then check your reasoning.
Q2A team wants to move raw telemetry older than 90 days to object storage. What evidence should be required before enabling the lifecycle job?
Show answer
Answer: D Archive policy is credible only when the team proves it can find, restore, decode, and use the retained data.
Retrieval practice
Recall check 3 of 3
Data Dora says: answer from memory, then check your reasoning.
Q3A dashboard is fast because it reads a latest-value cache, but after a cache outage the platform loses the only copy of current device state. Which storage handoff failed?
Show answer
Answer: B Latest-value views are useful, but they should be derived from durable records unless explicitly designed as authoritative state.
Print reference
Answers
Answer key.
- C · Registry data is structured identity state, not high-volume telemetry.
- D · Archive policy is credible only when the team proves it can find, restore, decode, and use the retained data.
- B · Latest-value views are useful, but they should be derived from durable records unless explicitly designed as authoritative state.
Print reference
Activity 1 answer
Model answer.
Work it out: 240 × 60 × 24 = 345,600 readings per day. The 240 registry records describe devices and ownership, while telemetry records measurements over time. A latest-value cache keeps current views and must be rebuildable from accepted durable telemetry.
Print reference
Activity 2 answer
Model answer.
Predict: Read the newest accepted durable record per device to rebuild the 600 cache entries. Mark the 12 devices with no valid recent sample as stale or unknown. Keep the exception evidence instead of inventing current values.