Stream Processing Module Guide

Decide Before the Event Arrives

Stream processing turns events into decisions while those events are still arriving. The central choice is not simply Kafka versus another product. It is which event-time facts must be preserved, which transformations may happen before storage, how long the system waits for disorder, and what it does when a rule, dependency, or consumer fails.

Carry one scenario through the route: vibration sensors on a production line publish one event per second. Maintenance needs an alert when three high-vibration events occur inside a two-minute event-time window, but gateways can reconnect late and replay buffered events. The alert must be deduplicated, explain which samples caused it, and remain correct during delay, restart, and backpressure.

Learning Objectives

By the end of the route, you should be able to distinguish event time, ingest time, and processing time; choose stateless or stateful operations, windows, watermarks, and late-event rules; and trace an event through ingestion, validation, enrichment, keying, aggregation, decision, sink, and replay. You should be able to explain when a broker, stream processor, database, and dashboard own different responsibilities.

You should also be able to compare single-stage, Lambda-like, Kappa-like, edge-cloud, and complex-event-processing architectures from latency, correctness, replay, state, and operations constraints; design idempotency and checkpoint behavior; detect backpressure and poison events; and define evidence that proves a recovered pipeline did not silently duplicate or lose decisions.

Before You Start

You should already recognise a publish-subscribe topic, message key, timestamp, schema, database write, and basic aggregation. Use MQTT Architecture if the broker path is unfamiliar, and Storage Roles and Data Lifecycle if retention and authoritative-store boundaries are unclear. Delivery QoS can change duplicate risk, but it does not define end-to-end processing correctness.

No distributed-systems course or cluster is required for the main route. The basic lab starts with a controlled or recorded event stream and a bounded window; the advanced lab assumes that you can already explain keys, event time, windows, late data, and state. This module teaches pipeline decisions and evidence, not vendor administration or arbitrary hardware wiring.

A Worked Route Through the Module

First, define the event and time contract

Begin with Stream Processing Fundamentals. Give each vibration event an event ID, sensor and machine ID, device timestamp with clock-quality evidence, sequence number, unit, schema version, and quality state. Decide which timestamp drives the two-minute rule. Processing time is easy to observe but would make a reconnect burst look like a new physical incident; event time better represents the machine, provided the system has an explicit rule for late and uncertain clocks.

Key by machine so related state meets the same logical operator. Define a two-minute sliding window, a watermark based on measured gateway delay, and a late-event path that preserves evidence rather than silently discarding it. State what happens when three events cross the threshold, when a duplicate repeats an event ID, and when a fourth late event would have changed a previously emitted result.

Next, choose the architecture from the failure contract

Continue to Choosing a Stream Architecture. Compare direct broker-to-consumer processing with a durable event log and stateful processor. The line needs replay after rule changes and restart, so retained ordered partitions, checkpointed state, and idempotent output matter more than the shortest demo path. Decide whether edge filtering may discard raw vibration evidence during an outage, or whether the gateway must buffer and forward it with original timestamps.

Use the Stream Processing game and Data Pipeline Builder here. First predict which architecture fits the event rate, latency target, replay need, team capacity, and state ownership. Then build the route and deliberately change one constraint. The tools should expose why a choice changes; they should not replace the written event, time, and failure contracts.

Then, make every stage observable and replayable

Use Building IoT Streaming Pipelines to draw ingress, schema validation, quality handling, enrichment, partitioning, window state, rule evaluation, alert output, storage, and dashboard delivery. Attach an owner, input/output contract, retry policy, dead-letter or quarantine route, metric, and trace field to every stage. Keep the event ID through the alert so an operator can recover the three samples behind the decision.

Choose acknowledgement and checkpoint boundaries together. If the system records input progress before durable state or output, a crash can lose a decision; if it retries after output without idempotency, it can duplicate one. “Exactly once” is a scoped implementation claim, not permission to ignore external sinks. Prove the boundary with a forced restart between state update and alert publication.

Finally, rehearse the failures before scaling

Read Common Pitfalls and Worked Examples with the scenario under pressure. Test a hot partition, schema change, poison event, slow sink, clock jump, reconnect burst, duplicated delivery, lost checkpoint, and alert-service outage. Observe lag, queue depth, watermark movement, state size, processing latency, rejected-event count, and replay progress. Scaling workers does not repair a bad key, unbounded state, or slow shared dependency.

Write a recovery decision for each failure: pause, buffer, shed non-critical work, quarantine, retry with a bound, fail closed, degrade, replay, or require operator approval. Preserve the reason and the point from which replay is safe. The route is finished only when a reviewer can distinguish delayed processing from missing source events and can reconcile every emitted alert with its inputs.

Labs and Interactives

Take Lab: Windowed Alerts immediately after the fundamentals and pipeline route. Feed a small timestamped fixture, predict each window, then introduce one late and one duplicated event. Capture window contents, watermark, emitted result, and reason. A simulator, recorded trace, classroom board, or instructor harness may supply events; the learning boundary is the event-time decision, not a particular device.

Use Lab: Advanced CEP only after you can explain keys, windows, state, and late data without the tool. Combine sequences or correlated signals, restart from a checkpoint, slow a sink to create backpressure, and replay the fixture. Keep one evidence ledger across both labs: input IDs, timestamps, partition/key, state transition, output ID, acknowledgement/checkpoint, failure injection, recovery action, and reconciled result.

Quick Route Check

Related Practice

Continue to Time-Series Database Fundamentals for durable write and query behavior, and to Real-Time Visualization for honest display of delay, gaps, and recovery. Use Anomaly Detection when the alert rule becomes a model, while preserving the same source, feature-time, quality, and replay evidence.

A strong final exercise is a one-page event-pipeline decision: event and timestamp contract, keys and partitions, window and watermark, stages and owners, state and checkpoints, output idempotency, backpressure response, quarantine, replay, negative fixtures, evidence, and retest triggers. If the diagram starts with a product logo, Data Dora would ask what happens to one late event first. Return to Browse Modules for the wider course route.