Analytics & ML · Study deck
Big Data Pipelines
Imagine a food company receiving a temperature fact from every delivery box.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: The team must either accept slower alerts, use a separate fast-but-provisional alert path, or improve device and network delay so the watermark does not carry the whole uncertainty.
- Explain: Raw sensor evidence may flow through MQTT or a gateway, into Kafka or a managed stream, through validation and enrichment, then into raw, curated, aggregate, and serving zones.
- Explain: When a schema bug or enrichment error is fixed, the team should know which raw partitions, broker offsets, table versions, and transformation code can rebuild the affected outputs.
- Explain: Schema registry or validation rule rejects the event.
Major section
Start With the Story
One fact may pass through a vehicle, a message service, a cleaning step, storage, and a warning screen.
- If its time, unit, or box name changes on the way, a neat chart can support the wrong decision.
- A data pipeline is this ordered path of steps.
- This simple journey leaves out large-scale scheduling and storage choices.
Major section
Start With the Story (continued)
The system owner should begin with one known fact and its allowed delay.
- At every step, record what arrived, what changed, what was rejected, and what left.
- The path should recover without hiding a gap or counting the same event twice.
- Under the Hood explains event time, repeated work, back pressure, and recovery.
Major section
Pipelines Turn Events to Evidence
An IoT big-data pipeline is the path from a physical reading to a trusted decision record.
- It should not be described only as "ETL" or "streaming." A useful pipeline states what each stage accepts, rejects, transforms, enriches, stores, and proves.
- The pipeline contract is different from the technology list.
Major section
Pipelines Turn Events to Evidence (continued)
Every derived result should be traceable back to its source events, transformation version, quality checks, and timing assumptions.
- Raw sensor evidence may flow through MQTT or a gateway, into Kafka or a managed stream, through validation and enrichment, then into raw, curated, aggregate, and serving zones.
- It defines ownership of timestamps, schemas, quality flags, retries, late data, units, enrichment joins, retention, and lineage.
- Serve Publish alerts, dashboards, model features, reports, and replayable tables with lineage.
Major section
Pipelines Turn Events to Evidence (continued)
Ingest Receive device events, authenticate source identity, attach arrival metadata, and reject malformed payloads.
- A temperature alert, maintenance dashboard, and model-training table may use the same raw event, but each needs different latency, completeness, and audit evidence.
- Without that lineage, a dashboard number is hard to debug and a model feature is hard to trust.
- Watermark setting, checkpoint id, window bounds, and late-event policy.
Major section
Budget Latency and Completeness
Stream processing is useful only when its latency target matches the decision.
- A machine-protection alert may need a few seconds.
- A parking availability dashboard may need tens of seconds.
- A monthly compliance report may tolerate hours if it is more complete.
- Completeness has a cost.
Major section
Budget Latency and Completeness (continued)
Kappa-style pipelines keep one replayable stream path and recompute by replaying retained events.
- The pipeline should state the latency budget from device sampling through gateway buffering, broker lag, stream processing, serving write, and user-visible refresh.
- A long watermark captures more delayed data but makes outputs arrive later.
- Window length can exceed dashboard freshness requirements.
Major section
Budget Latency and Completeness (continued)
Moving averages or repeated trend checks need smoother updates.
- Watermarks let Spark Structured Streaming, Flink, and similar systems decide how long to wait for late events before closing a window.
- The right setting depends on the physical decision, not on a default value.
- More overlapping windows increase compute and state.
Major section
Budget Latency and Completeness (continued)
On the bottom lane the same reading lands in the: Raw store at 69 GB/day and is recomputed overnight into the complete answer.
- design implication: If the watermark is raised from 12 seconds to 60 seconds, the same alert path can no longer meet a 30-second promise.
- The team must either accept slower alerts, use a separate fast-but-provisional alert path, or improve device and network delay so the watermark does not carry the whole uncertainty.
- Lambda-style pipelines use separate speed and batch views when fast provisional answers and slower corrected answers are both required.
Major section
Pipeline Failure Paths
A production pipeline is not complete until the failed-event path is designed.
- Some events are malformed, duplicated, late beyond the watermark, missing calibration context, or produced by devices with stale firmware.
- Dropping them silently makes metrics look clean while hiding operational problems.
- Replay also needs a contract.
- design implication: The dead-letter path is not a trash bin.
Major section
Pipeline Failure Paths (continued)
Dead Letter Stores rejected events with reason, schema version, raw payload pointer, and owner for investigation.
- A dead-letter queue or quarantine table keeps failed records with reasons so the team can fix devices, schemas, or transformations.
- When a schema bug or enrichment error is fixed, the team should know which raw partitions, broker offsets, table versions, and transformation code can rebuild the affected outputs.
- Schema registry or validation rule rejects the event.
Major section
Pipeline Failure Paths (continued)
It identifies fixable sources of bad data and protects downstream aggregates from silently mixing incompatible payloads, units, duplicates, or stale device behavior.
- Idempotency Uses event ids, stable keys, upserts, or transactional table writes so retries do not double-count.
- Deduplicate before aggregation and use idempotent sink writes.
- Event time is older than the closed window threshold.
Deck summary
Key takeaways
One fact may pass through a vehicle, a message service, a cleaning step, storage, and a warning screen.
- The system owner should begin with one known fact and its allowed delay.
- An IoT big-data pipeline is the path from a physical reading to a trusted decision record.
- Every derived result should be traceable back to its source events, transformation version, quality checks, and timing assumptions.
- Ingest Receive device events, authenticate source identity, attach arrival metadata, and reject malformed payloads.
Retrieval practice
Recall check 1 of 3
Data Dora says: answer from memory, then check your reasoning.
Q1What makes an IoT data pipeline trustworthy rather than just functional?
Show answer
Answer: C A pipeline is trustworthy when its outputs can be traced through validation, timing, transformation, and source evidence.
Retrieval practice
Recall check 2 of 3
Data Dora says: answer from memory, then check your reasoning.
Q2A cold-chain alert has a 30-second user-visible latency target and its current stage budget totals exactly 30 seconds. What happens if the watermark wait is increased from 12 seconds to 60 seconds without changing anything else?
Show answer
Answer: B Watermark tolerance is part of the end-to-end latency budget for event-time stream outputs.
Retrieval practice
Recall check 3 of 3
Data Dora says: answer from memory, then check your reasoning.
Q3A pipeline ingests 50,000 events per minute. Schema failures are 0.4 percent, range failures are 0.2 percent, and duplicate retries are 0.1 percent. How many events per hour go to the failed or quarantined path?
Show answer
Answer: A Failed-event volume is the ingest rate multiplied by the combined validation failure rate, then multiplied by time.
Print reference
Answers
Answer key.
- C · A pipeline is trustworthy when its outputs can be traced through validation, timing, transformation, and source evidence.
- B · Watermark tolerance is part of the end-to-end latency budget for event-time stream outputs.
- A · Failed-event volume is the ingest rate multiplied by the combined validation failure rate, then multiplied by time.