Stream Processing · Study deck
Choosing a Stream Architecture
Picture a freezer sending one temperature reading each minute.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: That requires stable event identifiers, a checkpoint or log offset, an idempotency key for A9, and a sink contract that treats replayed output as a correction or confirmation rather than a new incident.
- Explain: Stream-processing architecture is the shape of the path that receives IoT events, preserves their meaning, processes them while they are still useful, and hands bounded results to downstream decisions.
- Explain: If you only need the intuition, use this rule: choose the smallest architecture that preserves event identity, time basis, ordering, state, failure behavior, and retest evidence for the downstream decision.
Major section
Start With the Path One Event Must Survive
A simple check can reject a bad row and store the rest.
- A different alert needs three hot readings from the same freezer within ten minutes, while an audit must later replay the exact inputs that caused the warning.
- A live result is not enough if the team cannot explain its time, state, and recovery behavior.
- This opening does not name one best tool or shape.
Major section
Start With the Path One Event Must Survive (continued)
The shared stream can compare fleets and retain review evidence, but it must not be the only route to a time-bound alarm.
- Practitioner selects direct, stateful, replayable, or mixed paths from the decision.
- Under the Hood examines windows, ordering, checkpoints, recovery, sink contracts, and the limits that justify added parts.
- An architecture is the answer to those handoff questions.
Major section
Choose Architecture From Evidence
Stream-processing architecture is the shape of the path that receives IoT events, preserves their meaning, processes them while they are still useful, and hands bounded results to downstream decisions.
- The strongest architecture is not the most elaborate one.
- A sensor stream that only validates independent events may need a direct path.
- A maintenance alert that depends on several related events may need keyed state and event-time windows.
- The architecture choice is therefore a review claim.
Major section
Choose Architecture From Evidence (continued)
A stream that must be replayed after rule changes may need a durable event log.
- The architecture choice follows from the event path, ordering need, state requirement, sink contract, and recovery evidence.
- Worked example: a cold-chain gateway may receive one temperature reading every minute.
- Common PatternsChoose among these patterns by asking what evidence must survive.
Major section
Choose Architecture From Evidence (continued)
If the only decision is "reject malformed readings and store valid readings," a direct validation path is enough.
- If the decision is "alert when three related readings exceed the limit inside a 10-minute event-time window," the path now needs keyed state and a window rule.
- If auditors may later ask which readings produced an alert after the rule changes, the path also needs a durable log or replayable evidence store.
- It should say why simpler routing is sufficient, or why extra state, replay, or hybrid batch-stream processing is justified by the decision rather than by tool preference.
Major section
Choose Architecture From Evidence (continued)
If you only need the intuition, use this rule: choose the smallest architecture that preserves event identity, time basis, ordering, state, failure behavior, and retest evidence for the downstream decision.
- Independent events may need only a direct path; correlations need retained state; replay and audit need a durable history; and a hybrid is justified only when live and retrospective consumers genuinely ask different questions.
- A direct stream path fits independent events that need validation, light enrichment, and a sink but little retained state.
- Neither pattern is inherently more mature; each preserves a different evidence obligation.
Major section
Write The Architecture Review Record
A practical architecture review record explains why the selected pattern is sufficient for one stream path.
- It should name the downstream decision, event contract, time basis, ordering rule, state record, sink contract, failure handling, and retest trigger.
- This prevents architecture diagrams from hiding the real claims.
- The review record should make those boundaries explicit.
Major section
Write The Architecture Review Record (continued)
The alert, dashboard, durable result, or automated action that depends on the stream output.
- A box labeled "stream processor" does not explain whether late events are accepted, whether output is provisional, whether replay can recreate decisions, or whether a sink can tolerate correction.
- That anchor makes ordering scope, retained state, failure behavior, and retest triggers requirements of a specific result rather than generic platform features.
- Consumers infer missing meaning differently and produce incompatible results.
Major section
Write The Architecture Review Record (continued)
This workflow leaves an architecture record that explains both why the pattern exists and when its justification becomes stale.
- The architecture optimizes transport speed but not the decision that consumes the output.
- A multi-event decision is implemented as independent filtering with no retained evidence.
- The sink receives outputs without knowing whether they are provisional, corrected, stale, or final.
Major section
Write The Architecture Review Record (continued)
That decision needs more than independent filtering because partial evidence must be retained while the second signal may still arrive.
- Worked Review: Equipment Alert StreamA factory equipment alert stream receives vibration and temperature events from edge gateways.
- The alert should fire only when related signals for the same asset appear inside a reviewed event-time window.
- A direct path may still be fine for an independent health-heartbeat stream where each event stands alone.
Major section
Architecture Is Failure Semantics
A durable log preserves an event history but does not automatically define event meaning.
- A state store can support windows and correlations but needs retention, checkpoint, and recovery rules.
- A sink can display fast results but still needs finality, freshness, and correction semantics.
- Worked example: a motor alert rule sees vibration event V17, then temperature event T22.
Major section
Architecture Is Failure Semantics (continued)
The expected output should be written before the test runs.
- The hard questions appear when the path is stressed: events arrive late, devices resend, processors restart, state expires, a sink is unavailable, schema changes, or a rule must be replayed.
- A practical review also names what operators will see.
- Those observable contracts are part of the architecture.
Major section
Architecture Is Failure Semantics (continued)
Replay the same events twice, delay one event beyond the window, send one duplicate, and restart the processor mid-window.
- A reviewable architecture can explain whether output is unchanged, corrected, withheld, provisional, duplicated, or routed for manual review.
- After a restart, replaying V17 and T22 should rebuild the same decision without creating a second work order.
- If the architecture cannot predict whether each output is final, corrected, withheld, or dead-lettered, it is not yet reviewable.
Major section
Architecture Is Failure Semantics (continued)
That requires stable event identifiers, a checkpoint or log offset, an idempotency key for A9, and a sink contract that treats replayed output as a correction or confirmation rather than a new incident.
- If replay confirms alert A9, the dashboard should show the same alert identity and an updated recovery note, not a second incident.
- If the late T22 event arrives after the cutoff, the sink should show either a correction event or a documented rejection reason.
- Mechanics To SurfaceEvent-Time Mechanics Window boundaries, late-event policy, watermark or cutoff behavior, and received-time exceptions decide what evidence can influence an output.
Major section
Architecture Is Failure Semantics (continued)
Replay MechanicsDurable logs, idempotency keys, duplicate policies, versioned rules, and side-effect boundaries determine whether replay is safe.
- Integration MechanicsSink contracts, freshness labels, correction events, backpressure policy, and dead-letter routing determine what downstream users can trust.
- Failure Modes To TestRun each failure against a written expected output.
- Pressure the path until processing lags and verify that the sink reports age instead of presenting delayed input as current.
Major section
Summary · Key Takeaway
Architecture is the allocation of evidence and failure responsibility across a stream path.
- The right pattern is the smallest one that preserves the decision's event meaning, ordering, state, recovery, and sink semantics under the failures the team has actually committed to handle.
- A direct path is sufficient when events are independent and no replayable history is required.
- Correlation or windows justify keyed state, while audit and recovery may justify a durable log.
Deck summary
Key takeaways
A simple check can reject a bad row and store the rest.
- The shared stream can compare fleets and retain review evidence, but it must not be the only route to a time-bound alarm.
- Stream-processing architecture is the shape of the path that receives IoT events, preserves their meaning, processes them while they are still useful, and hands bounded results to downstream decisions.
- A stream that must be replayed after rule changes may need a durable event log.
- If the only decision is "reject malformed readings and store valid readings," a direct validation path is enough.
Retrieval practice
Recall check 1 of 3
Data Dora says: answer from memory, then check your reasoning.
Q1What is the strongest basis for choosing an IoT stream-processing architecture?
Show answer
Answer: C Architecture choice is reviewable when it starts from the evidence the decision needs.
Retrieval practice
Recall check 2 of 3
Data Dora says: answer from memory, then check your reasoning.
Q2A stream path must alert when vibration and temperature events for the same motor occur within a reviewed event-time window. The proposed design independently filters each event and sends matches to a sink. What is the best architecture review finding?
Show answer
Answer: A A multi-event decision needs retained state and correlation evidence, not independent filtering alone.
Retrieval practice
Recall check 3 of 3
Data Dora says: answer from memory, then check your reasoning.
Q3A stream processor restarts, rebuilds state from a log, and emits duplicate maintenance alerts because the sink treats every replayed output as new. Which architecture evidence is missing?
Show answer
Answer: B Replayable architecture needs rules for rebuilding decisions without uncontrolled duplicate side effects.
Print reference
Answers
Answer key.
- C · Architecture choice is reviewable when it starts from the evidence the decision needs.
- A · A multi-event decision needs retained state and correlation evidence, not independent filtering alone.
- B · Replayable architecture needs rules for rebuilding decisions without uncontrolled duplicate side effects.