3 Choosing a Stream Architecture

3.2 Choose Architecture From Evidence

Stream-processing architecture is the shape of the path that receives IoT events, preserves their meaning, processes them while they are still useful, and hands bounded results to downstream decisions. The strongest architecture is not the most elaborate one. It is the simplest design that preserves the evidence the decision needs.

A sensor stream that only validates independent events may need a direct path. A maintenance alert that depends on several related events may need keyed state and event-time windows. A stream that must be replayed after rule changes may need a durable event log. The architecture choice follows from the event path, ordering need, state requirement, sink contract, and recovery evidence.

Worked example: a cold-chain gateway may receive one temperature reading every minute. If the only decision is "reject malformed readings and store valid readings," a direct validation path is enough. If the decision is "alert when three related readings exceed the limit inside a 10-minute event-time window," the path now needs keyed state and a window rule. If auditors may later ask which readings produced an alert after the rule changes, the path also needs a durable log or replayable evidence store.

The architecture choice is therefore a review claim. It should say why simpler routing is sufficient, or why extra state, replay, or hybrid batch-stream processing is justified by the decision rather than by tool preference.

Architecture Questions

Common Patterns

Choose among these patterns by asking what evidence must survive. Independent events may need only a direct path; correlations need retained state; replay and audit need a durable history; and a hybrid is justified only when live and retrospective consumers genuinely ask different questions.

A direct stream path fits independent events that need validation, light enrichment, and a sink but little retained state. Add a durable log before processors when recovery, replay, multiple consumers, or later audit requires the original event history to remain available. Neither pattern is inherently more mature; each preserves a different evidence obligation.

A stateful processor is warranted when keyed windows, aggregates, correlations, or partial pattern progress must survive while new events arrive. A hybrid separates live decisions from historical review, training, or retrospective analysis only when those consumers genuinely need different timing and evidence paths. Select the pattern by tracing what must be reconstructed after delay or failure, then keep the result bounded to that stream path.

Overview Knowledge Check

3.3 Write The Architecture Review Record

A practical architecture review record explains why the selected pattern is sufficient for one stream path. It does not need to approve the whole platform. It should name the downstream decision, event contract, time basis, ordering rule, state record, sink contract, failure handling, and retest trigger.

This prevents architecture diagrams from hiding the real claims. A box labeled "stream processor" does not explain whether late events are accepted, whether output is provisional, whether replay can recreate decisions, or whether a sink can tolerate correction. The review record should make those boundaries explicit.

Review Workflow

Keep the downstream decision fixed while working through this sequence. That anchor makes ordering scope, retained state, failure behavior, and retest triggers requirements of a specific result rather than generic platform features.

Name the downstream alert, aggregate, dashboard, durable record, automated action, or review queue before selecting a pattern. Record the event identifier, source identity, event and receive times, schema version, units, status, and quality flags that decision needs. Next, decide which key requires ordering, what state is retained, and when that state expires or is repaired.

Only then choose a direct path, durable log, stateful processor, hybrid path, or bounded combination. Test the choice with late, duplicate, malformed, missing, delayed, retried, and replayed events, stating the expected sink behavior for each. Attach retest triggers to source, schema, load shape, time basis, ordering, state, and sink-contract changes. This workflow leaves an architecture record that explains both why the pattern exists and when its justification becomes stale.

Architecture Record

Worked Review: Equipment Alert Stream

A factory equipment alert stream receives vibration and temperature events from edge gateways. The alert should fire only when related signals for the same asset appear inside a reviewed event-time window. That decision needs more than independent filtering because partial evidence must be retained while the second signal may still arrive.

A stateful stream processor is justified for the alert path because it can key state by asset, retain partial pattern evidence, apply an event-time window, and emit a bounded alert record. A durable log may also be justified if the team must replay the evidence after a rule change, audit a disputed alert, or recover after processor failure. A direct path may still be fine for an independent health-heartbeat stream where each event stands alone.

Practitioner Knowledge Check

3.4 Architecture Is Failure Semantics

Under the hood, architecture choices decide what evidence survives failure. A durable log preserves an event history but does not automatically define event meaning. A state store can support windows and correlations but needs retention, checkpoint, and recovery rules. A sink can display fast results but still needs finality, freshness, and correction semantics.

The hard questions appear when the path is stressed: events arrive late, devices resend, processors restart, state expires, a sink is unavailable, schema changes, or a rule must be replayed. A reviewable architecture can explain whether output is unchanged, corrected, withheld, provisional, duplicated, or routed for manual review.

Worked example: a motor alert rule sees vibration event V17, then temperature event T22. The processor emits alert A9 and checkpoints state. After a restart, replaying V17 and T22 should rebuild the same decision without creating a second work order. That requires stable event identifiers, a checkpoint or log offset, an idempotency key for A9, and a sink contract that treats replayed output as a correction or confirmation rather than a new incident.

Test the failure semantics with a small trace. Replay the same events twice, delay one event beyond the window, send one duplicate, and restart the processor mid-window. The expected output should be written before the test runs. If the architecture cannot predict whether each output is final, corrected, withheld, or dead-lettered, it is not yet reviewable.

A practical review also names what operators will see. If replay confirms alert A9, the dashboard should show the same alert identity and an updated recovery note, not a second incident. If the late T22 event arrives after the cutoff, the sink should show either a correction event or a documented rejection reason. Those observable contracts are part of the architecture.

Mechanics To Surface

Failure Modes To Test

Run each failure against a written expected output. A late event should expose correction semantics, a restart should expose state recovery, replay should expose idempotency, and pressure or schema drift should expose whether consumers can still interpret freshness and meaning.

Send a late event after the first output and check whether the architecture emits a correction or exposes a finality boundary. Restart mid-pattern and compare recovered state with the expected partial match. Then replay the same evidence: a second alert, command, or durable record reveals that output identity and idempotency were not carried through recovery.

Pressure the path until processing lags and verify that the sink reports age instead of presenting delayed input as current. Finally, introduce an unknown schema version at intake; it should be rejected or diverted with context rather than forcing every downstream consumer to guess. Together these tests cover semantic change, recovery, side effects, freshness, and interpretation—the failure responsibilities that distinguish a reviewable architecture from a collection of available components.

Under-the-Hood Knowledge Check