โ† Back to the chapter

Technical detail ยท Big Data Fundamentals

Match reliability symptoms to controls

The rate calculation exposes capacity trouble. The following mechanisms address correctness and uneven load; each needs evidence that its configuration matches the workload.

ProblemSymptomControlTechnology Tie-In
Late dataWindow totals change after a dashboard or alert has already used them.Use event-time windows, watermarks, allowed lateness, and correction records.Flink and Spark Structured Streaming both support event-time windows and watermarks.
Duplicate retriesA device resend makes one physical reading count twice.Use event ids, deterministic upserts, deduplication windows, and idempotent sinks.Kafka keys, transactional writes, Delta Lake, and Iceberg can support replay-safe pipelines when configured carefully.
Schema driftA firmware update changes a field name, unit, or enum value.Version schemas, validate at ingress, and reject or quarantine incompatible events.Schema Registry with Avro, Protobuf, or JSON Schema makes compatibility visible before deployment.
Hot partitionsOne broker partition or storage shard lags while the cluster looks underused.Choose partition keys that spread writes and preserve only the ordering that is actually needed.Kafka partitions, object-store folder layout, and time-series database shards all need balanced keys.

โ† Back to the chapter