Stream Processing · Study deck
Building IoT Streaming Pipelines
Picture a river sensor that sends a high reading into a chain of filters and rules.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: Source and intake establish what the event means, processing and state explain how it influenced a decision, and the output and retest record tell consumers whether that decision can still be trusted.
- Explain: The value crossed a threshold, a window produced a summary, or a dashboard changed color, but the evidence may hide a late event, duplicate input, overloaded stage, or stale state.
- Explain: Worked example: a dashboard shows a tank-fill average as final at 10:05, then two delayed readings with earlier event times arrive at 10:08 and change the average.
Major section
Start With the Handoff Between Two Stages · A Pipeline Is A Chain Of Decisions
The final screen shows a safe average, but the team cannot tell which stage removed the spike.
- A useful result without its handoff history is hard to trust.
- For every handoff, write the input shape, event time, unit, change made, output shape, and result kept for review.
Major section
Design The Pipeline Review Ledger · State, Replay, And Output Meaning
A practical pipeline review records stage responsibilities before implementation details dominate the discussion.
- The ledger should be short enough to keep current, but explicit enough to explain why an event was accepted, delayed, corrected, rejected, or turned into an output.
- The safest workflow is to separate the source contract, intake checks, processing rule, state ownership, output routing, and retest evidence.
- Firmware, protocol, topic, schema, unit, timestamp, or source-identity change.
Major section
Common Pitfalls and Worked Examples
The screen turns red, but the high reading arrived late and was counted in the wrong window.
- Delay a reading, send it twice, change the clock, fill a stage, restart with saved state, and correct an earlier value.
- These cases do not cover every stream or business rule.
- That same evidence-first habit prevents overcorrection.
Major section
Common Pitfalls and Worked Examples (continued)
The value crossed a threshold, a window produced a summary, or a dashboard changed color, but the evidence may hide a late event, duplicate input, overloaded stage, or stale state.
- This chapter shows how to trace that claim back through time basis, window rule, recovery behavior, and retest evidence before the system acts.
- The safe review path starts with evidence instead of a broad rewrite.
- Source timestamp, clock policy, window assignment, or consumer freshness requirement changes.
Major section
Common Pitfalls and Worked Examples (continued)
Worked example: a dashboard shows a tank-fill average as final at 10:05, then two delayed readings with earlier event times arrive at 10:08 and change the average.
- A single delayed alert may require a late-event policy for one path, not a new broker, a new database, or a rewrite of every stream window.
- If you only need the intuition, use this rule: classify a stream pitfall only after event time, receive time, identity, state, pressure, output, and consumer impact are visible.
- It separates the observed symptom from the evidence, category, correction, output impact, and retest trigger.
Major section
Common Pitfalls and Worked Examples (continued)
A pitfall review record should be short enough to use during an incident or design review, but specific enough to explain the decision later.
- The correction should stay bounded to the affected stream path until evidence supports a wider change.
- A delayed alert does not automatically prove the whole pipeline is wrong.
- It may come from event-time semantics, late-event policy, queue pressure, duplicate handling, state recovery, schema drift, or output routing.
Major section
Common Pitfalls and Worked Examples (continued)
Recompute or hold only the affected output using the agreed timestamp rule.
- Backpressure alarm, schema release, source mix change, consumer complaint, or alert-priority change.
- Event-time and lateness tests show whether ordering remains meaningful.
- Duplicate and restart tests expose identity, checkpoint, and idempotency gaps.
- Pressure and schema-change tests reveal stale output or interpretation failures.
Major section
Common Pitfalls and Worked Examples (continued)
Stream-processing pitfalls are evidence gaps around time basis, windows, late data, duplicates, state, pressure, schema drift, and output meaning.
- Outputs should state whether they are provisional, revised, stale, diverted, held, or final when stream evidence is incomplete.
- Recovery and replay need tests for duplicate handling, state restoration, late-event behavior, pressure, and consumer-facing explanation.
- Most stream-processing pitfalls are hidden assumptions about ordering, clocks, duplicates, late data, pressure, replay, and whether outputs still mean what consumers think they mean.
Major section
Summary
Output identity, freshness, and correction semantics keep a recovered or delayed result from masquerading as a new decision.
- A defensible pipeline preserves a continuous evidence thread even though responsibility changes at every stage.
- Stable keys and explicit window boundaries make replay predictable.
- Event identity, event time, schema version, source context, state evidence, and output boundary should survive the path.
Major section
Summary (continued)
Source and intake establish what the event means, processing and state explain how it influenced a decision, and the output and retest record tell consumers whether that decision can still be trusted.
- A streaming pipeline is a chain of reviewable decisions from source contract to intake, processing, state, output, and retest evidence.
- Intake checks should preserve reason codes for accepted, rejected, delayed, repaired, quarantined, and duplicate events.
- Processing boundaries should name the rule, key, state, window, late-event behavior, and output correction policy.
- Replay, recovery, and backpressure are part of the design because they affect whether outputs remain explainable after failure.
Deck summary
Key takeaways
The final screen shows a safe average, but the team cannot tell which stage removed the spike.
- A practical pipeline review records stage responsibilities before implementation details dominate the discussion.
- The screen turns red, but the high reading arrived late and was counted in the wrong window.
- The value crossed a threshold, a window produced a summary, or a dashboard changed color, but the evidence may hide a late event, duplicate input, overloaded stage, or stale state.
- Worked example: a dashboard shows a tank-fill average as final at 10:05, then two delayed readings with earlier event times arrive at 10:08 and change the average.
Retrieval practice
Recall check 1 of 6
Data Dora says: answer from memory, then check your reasoning.
Q1What makes an IoT streaming pipeline reviewable before downstream consumers depend on it?
Show answer
Answer: A Pipeline claims are reviewable when each stage preserves enough evidence to explain and retest the decision.
Retrieval practice
Recall check 2 of 6
Data Dora says: answer from memory, then check your reasoning.
Q2A sensor event has a valid source identity but an unknown schema version. What is the strongest first pipeline response?
Show answer
Answer: A Intake should protect the main stream without destroying the evidence needed to review or repair the event.
Retrieval practice
Recall check 3 of 6
Data Dora says: answer from memory, then check your reasoning.
Q3After a processor restart, replayed events produce duplicate alerts for the same equipment event. Which part of the pipeline review should be investigated first?
Show answer
Answer: B Duplicate alerts after replay point to idempotency and output-boundary evidence, not just transport speed.
Retrieval practice
Recall check 4 of 6
Data Dora says: answer from memory, then check your reasoning.
Q4A dashboard value is published as final, then changes after older event-time records arrive. What is the strongest first review response?
Show answer
Answer: C Pitfall review starts with the smallest evidence record that explains the mismatch between stream behavior and consumer decision.
Retrieval practice
Recall check 5 of 6
Data Dora says: answer from memory, then check your reasoning.
Q5A single equipment alert arrives late while other low-priority readings keep flowing through the same path. What should the review do before changing every window rule?
Show answer
Answer: D The practitioner record protects one affected path while preserving evidence for escalation if the pattern repeats.
Retrieval practice
Recall check 6 of 6
Data Dora says: answer from memory, then check your reasoning.
Q6After a processor restart, replayed events emit the same downstream action twice. Which evidence should be investigated first?
Show answer
Answer: D Recovery behavior must preserve output meaning after retries, restarts, duplicate delivery, and replay.
Print reference
Answers 1 of 2
Answer key.
- A · Pipeline claims are reviewable when each stage preserves enough evidence to explain and retest the decision.
- A · Intake should protect the main stream without destroying the evidence needed to review or repair the event.
- B · Duplicate alerts after replay point to idempotency and output-boundary evidence, not just transport speed.
- C · Pitfall review starts with the smallest evidence record that explains the mismatch between stream behavior and consumer decision.
Print reference
Answers 2 of 2
Answer key.
- D · The practitioner record protects one affected path while preserving evidence for escalation if the pattern repeats.
- D · Recovery behavior must preserve output meaning after retries, restarts, duplicate delivery, and replay.