← Back to the chapter · Next detail →
Technical detail · Big Data Operations
Calculate the retention trade-off
Worked example: lifecycle and storage pressure raw daily volume: 69.12 GB/day current raw retention: 90 days proposed raw retention: 30 days hourly aggregate volume: 28.8 MB/day aggregate retention: 365 days current raw footprint: 69.12 * 90 = 6,220.8 GB = 6.22 TB proposed raw footprint: 69.12 * 30 = 2,073.6 GB = 2.07 TB raw storage reduction: 6.22 TB - 2.07 TB = 4.15 TB one year of hourly aggregates: 28.8 MB/day * 365 = 10,512 MB = 10.5 GB operational implication: Keeping raw data forever is rarely the right control. Keep raw long enough for replay, audit, and debugging, then keep compact aggregates or curated tables for longer-term trends. Retest replay before shortening raw retention.