54 Big Data Pipelines

54.2 Pipelines Turn Events to Evidence

An IoT big-data pipeline is the path from a physical reading to a trusted decision record. It should not be described only as "ETL" or "streaming." A useful pipeline states what each stage accepts, rejects, transforms, enriches, stores, and proves. Raw sensor evidence may flow through MQTT or a gateway, into Kafka or a managed stream, through validation and enrichment, then into raw, curated, aggregate, and serving zones.

The pipeline contract is different from the technology list. It defines ownership of timestamps, schemas, quality flags, retries, late data, units, enrichment joins, retention, and lineage. A temperature alert, maintenance dashboard, and model-training table may use the same raw event, but each needs different latency, completeness, and audit evidence.

Before selecting pipeline products, inspect Figure 54.2 to see which evidence each stage must accept, preserve, or produce.

Trace Figure 54.2 from the source contract through intake checks, processing rules, and stateful windows to outputs and review records. Intake establishes identity, time, and schema; processing transforms values under versioned rules; windows add event-time and late-data boundaries; outputs trigger storage or action; the review record makes the route reproducible. The final arrow is not an afterthought: it closes the chapter's running evidence chain by tying an operational decision back to the exact inputs, state, and code that produced it.

Overview Knowledge Check

The pipeline also needs a return path from decisions to evidence. Figure 54.3 makes that feedback explicit before latency and completeness are budgeted.

In Figure 54.3, PREPARE preserves quality flag Q2 rather than silently repairing the 7.4 °C event. REPORT carries both alert reason and caveat, while LEARN records outcomes and drift; this closes the loop back to the source instead of treating model release as the end.

54.3 Budget Latency and Completeness

Stream processing is useful only when its latency target matches the decision. A machine-protection alert may need a few seconds. A parking availability dashboard may need tens of seconds. A monthly compliance report may tolerate hours if it is more complete. The pipeline should state the latency budget from device sampling through gateway buffering, broker lag, stream processing, serving write, and user-visible refresh.

Completeness has a cost. Watermarks let Spark Structured Streaming, Flink, and similar systems decide how long to wait for late events before closing a window. A short watermark produces faster outputs but drops more late readings. A long watermark captures more delayed data but makes outputs arrive later. The right setting depends on the physical decision, not on a default value.

That latency-versus-completeness tension is easiest to hold as two physical paths fed by the same events. Figure 54.4 splits a cold-chain fleet’s traffic into a fast provisional lane and a complete historical lane, and pins each of the four V pressures to the spot in the pipeline where it actually bites.

Trace one reading through both lanes of Figure 54.4. On the top lane it meets the Stream window with its 12 s watermark and becomes an Alert in ≤ 30 s — provisional by construction, because the watermark closed the window before every straggler arrived. On the bottom lane the same reading lands in the Raw store at 69 GB/day and is recomputed overnight into the complete answer. The red VERACITY tie between the two outcomes is the operational obligation this section’s worked example quantifies: when the corrected nightly number disagrees with the 30-second alert, the pipeline must publish the correction rather than leave two truths standing.

Worked example: latency budget for a cold-chain alert
target user-visible alert latency: 30 seconds

budget by stage:
sensor sampling interval: 5 seconds
gateway batching: 4 seconds
network and broker delay: 3 seconds
stream window and watermark wait: 12 seconds
alert rule evaluation: 2 seconds
serving write and notification delivery: 4 seconds

total:
5 + 4 + 3 + 12 + 2 + 4 = 30 seconds

design implication:
If the watermark is raised from 12 seconds to 60 seconds, the same alert path
can no longer meet a 30-second promise. The team must either accept slower
alerts, use a separate fast-but-provisional alert path, or improve device and
network delay so the watermark does not carry the whole uncertainty.

Practitioner Knowledge Check

54.4 Pipeline Failure Paths

A production pipeline is not complete until the failed-event path is designed. Some events are malformed, duplicated, late beyond the watermark, missing calibration context, or produced by devices with stale firmware. Dropping them silently makes metrics look clean while hiding operational problems. A dead-letter queue or quarantine table keeps failed records with reasons so the team can fix devices, schemas, or transformations.

Replay also needs a contract. When a schema bug or enrichment error is fixed, the team should know which raw partitions, broker offsets, table versions, and transformation code can rebuild the affected outputs. Idempotent writes prevent replay from double-counting. Lineage links each corrected aggregate or model feature back to raw events, quality rules, enrichment versions, and processing time.

Worked example: validation failures and dead-letter rate
ingest rate: 50,000 events/minute
schema validation failure rate: 0.4 percent
range validation failure rate: 0.2 percent
duplicate retry rate: 0.1 percent

failed or quarantined events per minute:
50,000 * (0.004 + 0.002 + 0.001) = 350 events/minute

failed or quarantined events per hour:
350 * 60 = 21,000 events/hour

if 80 percent of schema failures come from one firmware version:
schema failures/minute = 50,000 * 0.004 = 200
affected firmware contribution = 200 * 0.80 = 160 events/minute

design implication:
The dead-letter path is not a trash bin. It identifies fixable sources of bad
data and protects downstream aggregates from silently mixing incompatible
payloads, units, duplicates, or stale device behavior.

Under the Hood Knowledge Check