← Back to the chapter · Previous detail

Technical detail · Big Data Pipelines

Replay without double-counting

Replay also needs a contract. When a schema bug or enrichment error is fixed, the team should know which raw partitions, broker offsets, table versions, and transformation code can rebuild the affected outputs. Idempotent writes prevent replay from double-counting. Lineage links each corrected aggregate or model feature back to raw events, quality rules, enrichment versions, and processing time.

Replay. Rebuilds outputs from broker offsets, raw lake partitions, table versions, and deterministic transforms.

Idempotency. Repeating a write produces the same stored result. Event ids, stable keys, upserts, or transactional table writes keep retries from double-counting.

Lineage. Links every aggregate, alert, feature, or report back to source events and rule versions.

FailureDetectionControlEvidence
Malformed payloadSchema registry or validation rule rejects the event.Quarantine with field-level reason and firmware context.Raw payload pointer, schema id, rejected field, and producer version.
Duplicate eventEvent id, sequence number, or deterministic key repeats.Deduplicate before aggregation and use idempotent sink writes.Duplicate key, first-seen offset, retry count, and sink action.
Late beyond watermarkEvent time is older than the closed window threshold.Emit correction record, quarantine, or route to batch reconciliation.Event time, arrival time, watermark, window id, and correction policy.
Bad enrichmentReference table version or join key is wrong or missing.Version enrichment data and replay affected partitions after correction.Reference version, join key, affected output partitions, and replay command.

← Back to the chapter · Previous detail