Technical detail ยท Big Data Fundamentals
Match reliability symptoms to controls
The rate calculation exposes capacity trouble. The following mechanisms address correctness and uneven load; each needs evidence that its configuration matches the workload.
| Problem | Symptom | Control | Technology Tie-In |
|---|---|---|---|
| Late data | Window totals change after a dashboard or alert has already used them. | Use event-time windows, watermarks, allowed lateness, and correction records. | Flink and Spark Structured Streaming both support event-time windows and watermarks. |
| Duplicate retries | A device resend makes one physical reading count twice. | Use event ids, deterministic upserts, deduplication windows, and idempotent sinks. | Kafka keys, transactional writes, Delta Lake, and Iceberg can support replay-safe pipelines when configured carefully. |
| Schema drift | A firmware update changes a field name, unit, or enum value. | Version schemas, validate at ingress, and reject or quarantine incompatible events. | Schema Registry with Avro, Protobuf, or JSON Schema makes compatibility visible before deployment. |
| Hot partitions | One broker partition or storage shard lags while the cluster looks underused. | Choose partition keys that spread writes and preserve only the ordering that is actually needed. | Kafka partitions, object-store folder layout, and time-series database shards all need balanced keys. |