Data Storage Module Guide
Your guide: Data Dora
“Choose the questions, time horizon, and recovery evidence before you choose the database. Storage is a lifecycle, not a bucket.”
Follow Data Dora as two hours of offline readings arrive late without losing their event time.
A river sensor loses contact for two hours.
It saves readings while the link is down.
The saved readings arrive after the link returns.
Keep their event time separate from their arrival time.
Store the Question, Not Just the Reading
An IoT record begins as an event but quickly acquires several jobs. Operations wants the latest value now, engineers want a high-resolution window around a failure, finance wants a monthly summary, and an auditor wants to know which device, schema, unit, and quality rule produced the answer. One database product may hold several of those views, but no product name decides the data contract, query path, retention boundary, or recovery proof for you.
This module follows data from ingest to deletion. The first half teaches storage roles, database selection, partition behaviour, sharding, quality, encoding, and migration evidence. The second opens the time-series write and query path, compares platform fit, and treats rollups, archives, late arrivals, holds, and deletion as reviewed lifecycle transitions. The final lab connects a real sensor-to-dashboard pipeline so every abstract decision has somewhere to fail visibly.
Carry one scenario through the route: a refrigerated-delivery fleet reports temperature, door state, battery level, and location every few seconds. Dispatch needs a live dashboard, investigators need raw readings around a cold-chain alarm, service staff need the vehicle and sensor inventory, and each incident carries changing notes and attachments. Your job is to choose relational, time-series, and document storage by evidence, then set retention and a pipeline that can survive late, duplicated, and malformed events.
Learning Objectives
By the end of the route, you should be able to turn an IoT workload into data shapes, write rate and volume, bounded query families, consistency needs, quality states, retention tiers, ownership, and recovery criteria. You should be able to defend when relational tables fit governed relationships and transactions, when a time-series store fits timestamped append and window queries, when a document store fits variable incident records, and when using more than one store is clearer than forcing every question through one model.
You should also be able to explain partition behaviour without reciting CAP as a product label; choose and test a shard key; preserve timestamps, units, schema versions, and provenance through validation and encoding; trace a write through buffering and compaction; shape latest-value, window, rollup, and replay queries; and produce retention evidence for raw data, downsampled summaries, archives, restores, holds, late arrivals, and deletion. Finally, you should be able to operate and diagnose the sensor-to-MQTT-to-database-to-Grafana path.
Before You Start
You should be comfortable reading a JSON object, a table row, an MQTT topic, and a simple query. If publish-subscribe routing is new, start with MQTT Publish-Subscribe Basics. If you need the journey before the storage detail, Sensor-to-Network Pipeline supplies the wider acquire, process, package, and transmit context. This module does not assume prior administration of InfluxDB, TimescaleDB, MongoDB, or PostgreSQL.
Bring a workload question rather than a favourite database. For the fleet scenario, write five things before reading on: expected events per second, the timestamp authority, the latest-value and incident-window queries, how long raw evidence must remain useful, and what must happen when the cloud link is unavailable. Estimates are allowed; unlabeled assumptions are not. The chapters will refine those values and show which measurements must replace them before release.
A Worked Route Through the Module
First, split the workload by question and lifecycle
Start with Storage Roles and Data Lifecycle. Draw the fleet path from device validation and edge buffering through hot operational data, warm summaries, archive, and governed deletion. Keep event time, ingest time, device identity, unit, schema version, and quality state with every reading. Use the storage calculator now: sampling interval, bytes per event, fleet size, replication, compression, and retention turn “lots of telemetry” into a capacity range that a reviewer can challenge.
Continue to Database Selection for IoT Workloads and make three explicit choices. Put stable vehicle, sensor, customer, and work-order relationships in a relational store because keys, joins, constraints, and transactions are part of the question. Put timestamped temperature, door, battery, and location readings in a time-series store because bounded windows, latest values, aggregation, compression, and append rate dominate. Put variable incident narratives, external references, and evolving attachments in a document store, while keeping the governed incident identifier linked to the relational record. Use Database Detective and the selector to test that argument against query and operational clues, not to collect a winning product badge.
Read CAP Theorem and Database Categories by describing what the fleet should do during a network partition. A gateway may continue buffering locally and mark delayed telemetry rather than inventing global agreement; a maintenance assignment may require stronger consistency before two teams are dispatched. Then use Sharding Strategies to test candidate keys. Vehicle ID spreads ordinary fleet writes and supports per-vehicle investigations, but a single high-volume customer or region can still create a hot shard. Prove routing, rebalancing, tenant isolation, and cross-shard query cost with representative skew.
Next, protect meaning before optimizing bytes
Move to Data Quality Monitoring. Quarantine impossible temperatures, preserve missing and late readings as explicit states, detect unit or schema drift, and assign an owner to every rule. A dashboard line that silently drops bad events looks smooth while destroying incident evidence. In Data Encoding and Codecs, choose compact telemetry encoding only after recording schema evolution, metadata, fidelity, latency, replay, and consumer compatibility. Compression that saves storage but erases quality flags or makes an archived event undecodable is a lifecycle failure.
Use Storage Design Case Studies as the review gate for the three-store decision. Write the authoritative source for each entity, the key that links stores, the order and idempotency contract for duplicated events, the migration and rollback route, and the queries that prove nothing disappeared. Avoid dual writes with unclear ownership. If a pipeline fans one event into several stores, an event identifier and replayable log should let you reconcile outcomes without pretending the writes were one transaction.
Then, design the time-series path around real questions
Open Time-Series Database Fundamentals and trace a temperature event through validation, write-ahead logging, memory buffering, flush, chunks, compression, and compaction. Decide whether the device timestamp or gateway timestamp is authoritative and how clock uncertainty travels with a late event. The write-path animation belongs here: increase load and explain whether latency, memory pressure, or compaction debt becomes the first operational signal.
In Time-Series Platform Selection, compare platform roles against the fleet contract: ingestion pattern, schema and dimensions, query language, retention controls, high availability, backup and restore, observability, team skills, and migration risk. Then shape requests in Time-Series Query Optimization. “Show temperature” is not a query contract. “For vehicle 42, return five-minute minimum, maximum, average, count, and quality coverage for the last six hours” supplies time and entity predicates, aggregation, granularity, and correctness evidence. Use the query builder and analyzer to confirm pruning and plan cost, and send routine dashboards to reviewed rollups rather than scanning raw history.
Finally, set retention and build the pipeline
Take Data Retention and Downsampling before enabling deletion. For this design, propose a reviewable policy such as thirty days of raw telemetry for incident replay, thirteen months of hourly rollups for seasonal operations, and a separately governed archive for records under investigation. Those numbers are scenario decisions, not universal defaults. Prove rollup completeness, quality-field survival, archive manifests, a restored sample query, late-arrival and refresh windows, legal or incident holds, deletion candidates, owner approval, and rollback limits in a dry run.
Rehearse the evidence in MQTT to SQLite Time Series Lab: publish representative events, validate their schema, query the stored rows, refresh a late-data rollup, dry-run retention, and verify a restored copy. Break one field, timestamp, or database handoff in turn so you can distinguish transport success, validation, database writes, and query results instead of treating the pipeline as one opaque failure.
Labs and Interactives
Use the interactives as one design notebook. Record the storage calculator's ingest and retention range, the database selector's rejected alternatives, the time-series write path's pressure point, and the query analyzer's scanned rows and plan change. In the optimization game, change one dimension, index, rollup, or retention control at a time and explain the trade among cost, latency, fidelity, and operational complexity. A lower number is not a win if the intended incident query or recovery path no longer works.
For the pipeline lab, keep a trace identifier from a simulated sensor event through its MQTT topic and payload, bridge log, database measurement and tags, Flux query, and Grafana panel. Add an out-of-order event, a duplicate, an impossible temperature, a changed field name, and a short network outage. Record whether each event is accepted, rejected, quarantined, retried, deduplicated, or shown with a quality warning. Finish by replaying buffered data and confirming the live view, rollup, and raw incident window agree about counts and gaps.
Quick Route Check
The fleet needs transactional work orders, five-second temperature plots, and incident records whose fields vary by customer. Should all three go into the time-series database because telemetry is the largest volume?
Reveal the route
No. Choose by data shape and question: relational storage fits governed entities, constraints, and transactional work orders; time-series storage fits timestamped append, bounded windows, latest values, and rollups; document storage can fit variable incident narratives. Link them with governed identifiers and a replayable event contract, then test the joins the application actually needs. Volume affects capacity and sharding, but it does not erase transaction, query, lifecycle, and recovery requirements.