← Back to the chapter · Previous detail

Technical detail · Big Data Operations

Keep a lifecycle control record

ControlWhat It PreventsRetest EvidenceFailure Signal
Retention policyUnbounded raw growth, slow scans, unclear replay windows, and accidental deletion.Lifecycle rule, table owner, replay drill, and exception list.Storage grows faster than forecast or old raw data is needed but missing.
CompactionSmall-file overload in object storage or lakehouse tables.File-size distribution, compaction job logs, table statistics, and query plan.Queries scan many tiny files and latency rises with file count.
Schema gateFirmware changes silently changing units, fields, or enum meanings.Compatibility test, rejected sample payloads, and schema registry version.Spike in rejected events or wrong values after firmware deployment.
Replay drillDiscovering too late that raw data, code, schema, or checkpoint state cannot rebuild outputs.Replay command, input offset range, table version, output diff, and owner approval.A corrected job cannot reproduce a known aggregate from retained evidence.

← Back to the chapter · Previous detail