2 Stream Processing Fundamentals

2.2 Streams Are Data In Motion

This section: Streams Are Data In Motion — Stream processing handles events while new events are still arriving.

Stream processing handles events while new events are still arriving. In IoT, that matters when a reading should change a live status, alert, control review, anomaly review, or downstream record before a complete historical dataset exists.

The fundamentals are not tied to a single tool. A reviewable stream design states the event source, the timestamp that controls the calculation, the window that bounds the question, the rule for late or duplicated events, the state that must be remembered, and the evidence that should trigger retest.

Worked example: a freezer sensor sends one temperature reading per minute, but the device buffers data during a cellular outage. When connectivity returns, five older readings may arrive after newer readings. A stream that answers "was the package above 8 C for 15 minutes?" should use event time, not arrival time, because the question is about when the package was warm. The record also needs the allowed-late rule so reviewers know whether a delayed reading can revise the result.

The same stream can produce more than one output. A live dashboard may show the latest received value, while a compliance window uses event-time readings and correction markers. Both are valid if their contracts are explicit; the pitfall is letting the dashboard's freshness silently define the durable record.

Core Ideas

Beginner Examples

Sort these examples by the decision's useful delay, not by the number of events. The freezer alert benefits from current evidence, whereas a monthly summary can wait; the delayed reading and live dashboard examples then show why timing and presentation still need durable rules.

A live freezer alert usually belongs on the stream path because its action value falls if it waits for a daily report. A monthly energy summary can remain a batch result when no consumer acts on partial totals. That contrast makes the consumer's useful delay—not event volume—the first design test.

Timing semantics come next. Evaluate a delayed sensor reading with its event-time and late-event rule instead of discarding it solely because it arrived after newer records. A dashboard may show the latest received value quickly, but the durable stream record must still identify later corrections. These examples connect the decision for streaming to the evidence needed when arrival order and display order diverge.

Overview Knowledge Check

2.3 Build The Stream Review Record

This section: Build The Stream Review Record — A practical stream-processing review should produce a small record that another engineer can test against representative events.

A practical stream-processing review should produce a small record that another engineer can test against representative events. The record should explain why the stream is needed, what the output means, how events are grouped, what happens when events are late or duplicated, and which assumption reopens the decision.

This keeps the design from becoming a tool-first claim such as "we use a streaming platform, so the result is real time." The useful proof is narrower: given this source trace and this rule set, the processor emits this result, handles this degraded case, and can recover its state without changing the meaning of the output.

Review Sequence

Use one consumer decision to hold the sequence together. Each step narrows the same claim: why events must be processed now, what those events mean, how the question is bounded, and which state and correction evidence supports the output.

Name the consumer decision first: who acts, what changes, and how quickly the result loses value. Then classify the source events by recording their meaning, required fields, timestamp basis, and validation behavior. Those two steps stop a convenient transport or processor from defining the analytical question.

Turn the endless flow into a bounded question with a tumbling, sliding, session, or sequence rule that matches the consumer's need. Define whether late or corrected events revise the result, mark it provisional, enter review, or remain outside the decision. Finally, record the maintained state, checkpoint expectation, emitted output, correction behavior, and retest triggers. The resulting sequence links the original action need to evidence that can survive replay and review.

Evidence Ledger

Worked Review: Delayed Sensor Events

Suppose a condition-monitoring stream computes a five-minute vibration status for each motor. One reading arrives after newer readings from the same motor, but its event-time timestamp is trustworthy.

The review does not start by asking which product should process the stream. It starts by checking the event-time rule, window type, allowed lateness, duplicate policy, state key, and consumer tolerance for correction. If the rule allows updates, the affected window is corrected and the output contract tells consumers that the status changed. If the rule does not allow updates, the late reading is still recorded according to policy so the team can review whether the lateness assumption is still valid.

Practitioner Knowledge Check

2.4 Time Progress, State, And Recovery

This section: Time Progress, State, And Recovery — Under the hood, a stream processor turns an unbounded flow into state transitions.

Under the hood, a stream processor turns an unbounded flow into state transitions. Each accepted event is parsed, assigned to a key or grouping rule, placed against a timestamp basis, applied to maintained state, and then used to emit or revise an output when the policy says enough evidence exists.

Watermarks and checkpoints support that process, but they do not replace the stream contract. A watermark is a processor's estimate that event time has advanced far enough to close or publish a result. A checkpoint is recovery evidence for maintained state. Neither proves that the chosen window, timestamp, or late-event rule is semantically correct for the consumer.

Worked example: a motor stream keeps a five-minute rolling count of over-vibration events by asset id. The processor state includes the asset key, the active window, the current count, and whether the emitted status is provisional or final. A checkpoint can restore that count after restart, but the restored bytes are only meaningful if the same event-time rule, duplicate policy, and output contract are still in force.

To test recovery, replay a short trace through a restart: one on-time reading, one duplicate retry, one late reading, and one malformed payload. The expected result should say which events changed state, which were rejected, whether an output was revised, and which checkpoint or replay marker proves the restart did not duplicate the output.

Also record what the consumer sees after recovery. If the status is recalculated from the same events, the output should keep the same identity and show a recovery note, not create a second alert. If a late event changes the count, the output should identify the revised window and the event that caused the correction.

Internal Responsibilities

Failure Modes To Review

Exercise these failures with a short trace rather than discussing them abstractly. Include an offline-buffered reading, a duplicate retry, and a restart so the review can distinguish timestamp error, identity error, state recovery, and output ambiguity.

Start the trace with clock drift: an event can be syntactically valid while its event-time value is inconsistent. Add offline buffering so receive order differs from observation order after reconnection, then replay a duplicate with the same stable identity. These cases separate timestamp validation, ordering policy, and deduplication rather than hiding all three behind “late data.”

Continue long enough to expose state growth in sessions, joins, or wide windows, and inspect whether retention and cleanup match the contract. Then examine the consumer output. If a provisional result is displayed as final or a correction has no visible identity, the processor may be operational while the decision remains ambiguous. The review is complete only when each failure produces an expected state transition and an explainable output.

Retest Signals

Retest the stream when a source timestamp changes meaning, a firmware release changes event shape, offline buffering patterns shift, state retention changes, checkpoint or replay behavior changes, a consumer stops accepting corrections, or an output moves from advisory review to automatic action.

Under-The-Hood Knowledge Check