← Back to the chapter · Next detail →

Technical detail · Big Data Operations

Calculate the retention trade-off

Worked example: lifecycle and storage pressure
raw daily volume: 69.12 GB/day
current raw retention: 90 days
proposed raw retention: 30 days
hourly aggregate volume: 28.8 MB/day
aggregate retention: 365 days

current raw footprint:
69.12 * 90 = 6,220.8 GB = 6.22 TB

proposed raw footprint:
69.12 * 30 = 2,073.6 GB = 2.07 TB

raw storage reduction:
6.22 TB - 2.07 TB = 4.15 TB

one year of hourly aggregates:
28.8 MB/day * 365 = 10,512 MB = 10.5 GB

operational implication:
Keeping raw data forever is rarely the right control. Keep raw long enough for
replay, audit, and debugging, then keep compact aggregates or curated tables
for longer-term trends. Retest replay before shortening raw retention.

← Back to the chapter · Next detail →