← Back to the chapter · Next detail →
Technical detail · Big Data Pipelines
Quarantine and failed-event volume
A production pipeline is not complete until the failed-event path is designed. Some events are malformed, duplicated, late beyond the watermark, missing calibration context, or produced by devices with stale firmware. Dropping them silently makes metrics look clean while hiding operational problems. Data Quality Validation develops these schema checks alongside physical limits, rate rules, plausibility tests, outlier bounds, quarantine, and audit evidence. A dead-letter queue or quarantine table keeps failed records with reasons so the team can fix devices, schemas, or transformations.
Worked example: validation failures and dead-letter rate ingest rate: 50,000 events/minute schema validation failure rate: 0.4 percent range validation failure rate: 0.2 percent duplicate retry rate: 0.1 percent failed or quarantined events per minute: 50,000 * (0.004 + 0.002 + 0.001) = 350 events/minute failed or quarantined events per hour: 350 * 60 = 21,000 events/hour assuming the three failure categories are mutually exclusive if 80 percent of schema failures come from one firmware version: schema failures/minute = 50,000 * 0.004 = 200 affected firmware contribution = 200 * 0.80 = 160 events/minute design implication: The dead-letter path is not a trash bin. It identifies fixable sources of bad data and protects downstream aggregates from silently mixing incompatible payloads, units, duplicates, or stale device behavior.
Dead Letter. Stores rejected events with reason, schema version, raw payload pointer, and owner for investigation.