Data Storage · Study deck
Data Retention and Downsampling
Picture a cold store that keeps one temperature reading each minute.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Design a retention contract that separates raw telemetry, rollups, archives, deletion, and legal holds.
- Choose downsampling fields that preserve enough evidence for dashboards and investigations.
- Run a retention dry run before deleting raw data.
- Explain how late arrivals and rollup refresh windows affect retention safety.
Major section
Start With the Deletion Request
Telemetry means readings and status sent from equipment for remote use.
- The storage owner wants to remove old raw data, while the duty team may still need to explain a short heat spike from last month.
- This opening does not set one lifetime for every reading.
- Practitioner designs the raw, summary, archive, hold, and approval route.
Major section
Start With the Deletion Request (continued)
Under the Hood examines late arrivals, partition removal, refresh windows, platform commands, and the proof needed before detail is gone.
- Retention becomes serious the first time someone asks to delete raw readings.
- Downsampling is safe only when that evidence survives.
- “Every reading has a time and a cost — decide retention before you decide the database.”.
Major section
Retention Dry Run
The hold gate then checks investigations, disputes, compliance rules, and warranty cases that still need raw detail.
- A dry run should run before the first automated deletion and before any policy change that shortens raw retention.
- It should report the raw partitions that would be affected, the rollups that replace them, archive status, restore status, query replay results, hold exceptions, and approval owner.
- Cost: a restored sample partition must match the expected query result before deletion is approved.
Major section
Platform Patterns
Retention is implemented differently across systems.
- The review habit should stay stable even when the command changes.
- TimescaleDB retention commonly uses a background policy that drops chunks older than a chosen interval.
- Continuous aggregates can preserve summaries after raw chunks expire, but they need refresh windows and late-data rules.
Major section
Partition-Aligned Deletion Mechanics
Retention is cheap only when the storage layout gives the engine a whole unit to remove.
- In time-partitioned systems, old raw data should expire as a complete chunk, partition, shard, or block.
- One raw day contains 3,456,000,000 points before indexes, replicas, metadata, and compression.
- Boundary alignment is the trap.
Major section
Partition-Aligned Deletion Mechanics (continued)
A row-by-row delete for that day has to evaluate billions of row-level decisions.
- Dropping a one-day chunk removes the partition that already owns that time range.
- If chunks are one day long and the approved cutoff is midnight UTC, the engine can drop whole day chunks.
- Monitoring stores have the same idea with blocks: a Prometheus-style store removes old complete blocks, not arbitrary rows inside a block.
Major section
Common Pitfalls
Storage pressure may start the conversation, but the retention contract belongs to the product, operations, compliance, and support owners.
- The database job should implement an approved lifecycle decision.
- An average without sample count, min, max, quality status, and gap markers can hide missing samples and incident spikes.
- It may make a dashboard cheaper while making the system less explainable.
Major section
Common Pitfalls (continued)
Backup helps recover the platform.
- Retention design needs both concepts, but they are not the same artifact.
- Retention commands and scopes differ across TimescaleDB, InfluxDB versions, Prometheus, managed services, and self-hosted deployments.
- Teach the evidence model first, then confirm the exact implementation docs for the system being used.
Major section
Summary
Data retention is the lifecycle discipline behind time-series systems.
- It decides how long raw detail remains, what summaries replace it, what archive can be restored, which data is held, and when deletion is approved.
- The strongest retention design is evidence-first: preserve quality in rollups, dry-run every deletion policy, restore a sample before trusting an archive, make holds visible to automation, and record owner approval.
- That keeps storage manageable without letting quality drift or incident evidence disappear.
Deck summary
Key takeaways
Telemetry means readings and status sent from equipment for remote use.
- Under the Hood examines late arrivals, partition removal, refresh windows, platform commands, and the proof needed before detail is gone.
- The hold gate then checks investigations, disputes, compliance rules, and warranty cases that still need raw detail.
- Retention is implemented differently across systems.
- Retention is cheap only when the storage layout gives the engine a whole unit to remove.
Retrieval practice
Recall check 1 of 5
Data Dora says: answer from memory, then check your reasoning.
Q1Which retention release criterion is strongest before deleting raw telemetry?
Show answer
Answer: B Retention is safe only when the replacement evidence, restore path, query behavior, hold status, and owner decision are visible.
Retrieval practice
Recall check 2 of 5
Data Dora says: answer from memory, then check your reasoning.
Q2How should the team handle retention for these two types of data?
Show answer
Answer: D Prometheus local retention serves monitoring; customer readings need their own archive and replay evidence.
Retrieval practice
Recall check 3 of 5
Data Dora says: answer from memory, then check your reasoning.
Q3Why does a time-partitioned store use chunk- or partition-drop deletion instead of a large DELETE WHERE time < cutoff job?
Show answer
Answer: B Retention is efficient when policy cutoffs align with chunks, partitions, or blocks; row-by-row deletes are the fallback to avoid.
Q4Complete the retention decision record with the evidence fields required before raw data can be deleted.
Show answer
Answer: A
Retrieval practice
Recall check 4 of 5
Data Dora says: answer from memory, then check your reasoning.
Q5A team wants to shorten raw telemetry retention because storage is growing. What should they do before changing the deletion policy?
Show answer
Answer: D Shortening raw retention changes what evidence remains available, so the release packet must prove the replacement path.
Retrieval practice
Recall check 5 of 5
Data Dora says: answer from memory, then check your reasoning.
Q6A reading arrives after its time bucket was summarized but before raw data expires. What must the retention process do?
Show answer
Answer: C A late reading must update the affected event-time summary while raw evidence remains.
Print reference
Answers 1 of 2
Answer key.
- B · Retention is safe only when the replacement evidence, restore path, query behavior, hold status, and owner decision are visible.
- D · Prometheus local retention serves monitoring; customer readings need their own archive and replay evidence.
- B · Retention is efficient when policy cutoffs align with chunks, partitions, or blocks; row-by-row deletes are the fallback to avoid.
- A · raw_window: observed_at_range, late_arrival_window, hold_scope
- D · Shortening raw retention changes what evidence remains available, so the release packet must prove the replacement path.
Print reference
Answers 2 of 2
Answer key.
- C · A late reading must update the affected event-time summary while raw evidence remains.