Data Storage · Study deck
Data Retention and Downsampling
Picture a cold store that keeps one temperature reading each minute.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Design a retention contract that separates raw telemetry, rollups, archives, deletion, and legal holds.
- Choose downsampling fields that preserve enough evidence for dashboards and investigations.
- Run a retention dry run before deleting raw data.
- Explain how late arrivals and rollup refresh windows affect retention safety.
Major section
Start With the Deletion Request
Telemetry means readings and status sent from equipment for remote use.
- The storage owner wants to remove old raw data, while the duty team may still need to explain a short heat spike from last month.
- This opening does not set one lifetime for every reading.
- Practitioner designs the raw, summary, archive, hold, and approval route.
Major section
Start With the Deletion Request (continued)
Under the Hood examines late arrivals, partition removal, refresh windows, platform commands, and the proof needed before detail is gone.
- Retention becomes serious the first time someone asks to delete raw readings.
- Downsampling is safe only when that evidence survives.
- “Every reading has a time and a cost — decide retention before you decide the database.”.
Major section
Lifecycle Map
Retention is a sequence of evidence-bearing state changes, not a single expiry number.
- Rollups preserve declared statistics and coverage, the archive preserves a recoverable copy, and a restore test proves that copy can answer a real query.
- Deletion is the final governed transition only after every replacement path has evidence.
Major section
Downsampling That Preserves Evidence
The mistake is treating the average as the whole story.
- For IoT telemetry, a useful rollup normally needs fields that preserve coverage, extremes, and quality.
- Cost: an average alone erases the spikes and sparse coverage that support and safety teams need.
Major section
Retention Dry Run
A dry run should run before the first automated deletion and before any policy change that shortens raw retention.
- It should report the raw partitions that would be affected, the rollups that replace them, archive status, restore status, query replay results, hold exceptions, and approval owner.
- That makes the dry run the operational proof of the lifecycle map.
Major section
Platform Patterns
Retention is implemented differently across systems.
- The review habit should stay stable even when the command changes.
- TimescaleDB retention commonly uses a background policy that drops chunks older than a chosen interval.
- Continuous aggregates can preserve summaries after raw chunks expire, but they need refresh windows and late-data rules.
Major section
Partition-Aligned Deletion Mechanics
Retention is cheap only when the storage layout gives the engine a whole unit to remove.
- In time-partitioned systems, old raw data should expire as a complete chunk, partition, shard, or block.
- One raw day contains 3,456,000,000 points before indexes, replicas, metadata, and compression.
- Boundary alignment is the trap.
Major section
Partition-Aligned Deletion Mechanics (continued)
A row-by-row delete for that day has to evaluate billions of row-level decisions.
- Dropping a one-day chunk removes the partition that already owns that time range.
- If chunks are one day long and the approved cutoff is midnight UTC, the engine can drop whole day chunks.
- Monitoring stores have the same idea with blocks: a Prometheus-style store removes old complete blocks, not arbitrary rows inside a block.
Major section
Common Pitfalls
Storage pressure may start the conversation, but the retention contract belongs to the product, operations, compliance, and support owners.
- The database job should implement an approved lifecycle decision.
- An average without sample count, min, max, quality status, and gap markers can hide missing samples and incident spikes.
- It may make a dashboard cheaper while making the system less explainable.
Major section
Common Pitfalls (continued)
Backup helps recover the platform.
- Retention design needs both concepts, but they are not the same artifact.
- Retention commands and scopes differ across TimescaleDB, InfluxDB versions, Prometheus, managed services, and self-hosted deployments.
- Teach the evidence model first, then confirm the exact implementation docs for the system being used.
Major section
Summary
Data retention is the lifecycle discipline behind time-series systems.
- It decides how long raw detail remains, what summaries replace it, what archive can be restored, which data is held, and when deletion is approved.
- The strongest retention design is evidence-first: preserve quality in rollups, dry-run every deletion policy, restore a sample before trusting an archive, make holds visible to automation, and record owner approval.
- That keeps storage manageable without letting quality drift or incident evidence disappear.
Deck summary
Key takeaways
Telemetry means readings and status sent from equipment for remote use.
- Under the Hood examines late arrivals, partition removal, refresh windows, platform commands, and the proof needed before detail is gone.
- Retention is a sequence of evidence-bearing state changes, not a single expiry number.
- The mistake is treating the average as the whole story.
- A dry run should run before the first automated deletion and before any policy change that shortens raw retention.
Retrieval practice
Recall check 1 of 4
Data Dora says: answer from memory, then check your reasoning.
Q1Which retention release criterion is strongest before deleting raw telemetry?
Show answer
Answer: B Retention is safe only when the replacement evidence, restore path, query behavior, hold status, and owner decision are visible.
Retrieval practice
Recall check 2 of 4
Data Dora says: answer from memory, then check your reasoning.
Q2Why does a time-partitioned store use chunk- or partition-drop deletion instead of a large DELETE WHERE time < cutoff job?
Show answer
Answer: B Retention is efficient when policy cutoffs align with chunks, partitions, or blocks; row-by-row deletes are the fallback to avoid.
Retrieval practice
Recall check 3 of 4
Data Dora says: answer from memory, then check your reasoning.
Q3Place each retention artifact where it lives so you can delete data without destroying required evidence.
Show answer
Answer: A Place source protection, substitute or restore proof, and authorization correctly so you can make retention release auditable.
Q4Complete the retention decision record with the evidence fields required before raw data can be deleted.
Show answer
Answer: A
Retrieval practice
Recall check 4 of 4
Data Dora says: answer from memory, then check your reasoning.
Q5A team wants to shorten raw telemetry retention because storage is growing. What should they do before changing the deletion policy?
Show answer
Answer: D Shortening raw retention changes what evidence remains available, so the release packet must prove the replacement path.
Print reference
Answers
Answer key.
- B · Retention is safe only when the replacement evidence, restore path, query behavior, hold status, and owner decision are visible.
- B · Retention is efficient when policy cutoffs align with chunks, partitions, or blocks; row-by-row deletes are the fallback to avoid.
- A · Place source protection, substitute or restore proof, and authorization correctly so you can make retention release auditable.
- A
- D · Shortening raw retention changes what evidence remains available, so the release packet must prove the replacement path.