← Back to the chapter · Previous detail
Technical detail · Big Data Operations
Keep a lifecycle control record
| Control | What It Prevents | Retest Evidence | Failure Signal |
|---|---|---|---|
| Retention policy | Unbounded raw growth, slow scans, unclear replay windows, and accidental deletion. | Lifecycle rule, table owner, replay drill, and exception list. | Storage grows faster than forecast or old raw data is needed but missing. |
| Compaction | Small-file overload in object storage or lakehouse tables. | File-size distribution, compaction job logs, table statistics, and query plan. | Queries scan many tiny files and latency rises with file count. |
| Schema gate | Firmware changes silently changing units, fields, or enum meanings. | Compatibility test, rejected sample payloads, and schema registry version. | Spike in rejected events or wrong values after firmware deployment. |
| Replay drill | Discovering too late that raw data, code, schema, or checkpoint state cannot rebuild outputs. | Replay command, input offset range, table version, output diff, and owner approval. | A corrected job cannot reproduce a known aggregate from retained evidence. |