← Back to the chapter · Next detail →

Technical detail · Big Data Pipelines

Quarantine and failed-event volume

A production pipeline is not complete until the failed-event path is designed. Some events are malformed, duplicated, late beyond the watermark, missing calibration context, or produced by devices with stale firmware. Dropping them silently makes metrics look clean while hiding operational problems. Data Quality Validation develops these schema checks alongside physical limits, rate rules, plausibility tests, outlier bounds, quarantine, and audit evidence. A dead-letter queue or quarantine table keeps failed records with reasons so the team can fix devices, schemas, or transformations.

Worked example: validation failures and dead-letter rate
ingest rate: 50,000 events/minute
schema validation failure rate: 0.4 percent
range validation failure rate: 0.2 percent
duplicate retry rate: 0.1 percent

failed or quarantined events per minute:
50,000 * (0.004 + 0.002 + 0.001) = 350 events/minute

failed or quarantined events per hour:
350 * 60 = 21,000 events/hour

assuming the three failure categories are mutually exclusive

if 80 percent of schema failures come from one firmware version:
schema failures/minute = 50,000 * 0.004 = 200
affected firmware contribution = 200 * 0.80 = 160 events/minute

design implication:
The dead-letter path is not a trash bin. It identifies fixable sources of bad
data and protects downstream aggregates from silently mixing incompatible
payloads, units, duplicates, or stale device behavior.

Dead Letter. Stores rejected events with reason, schema version, raw payload pointer, and owner for investigation.

← Back to the chapter · Next detail →