53 Big Data Technologies

53.2 Technologies Have Different Jobs

An IoT big-data stack is a chain of responsibilities, not a list of fashionable product names. Device protocols move readings from constrained devices to gateways or cloud ingress. A broker buffers events and lets several consumers read the same stream. Stream processors turn live events into windows, alerts, and enriched records. Storage systems keep raw, curated, and serving-ready data at different costs and query speeds. Governance tools make schemas, access, lineage, and replay visible.

The right technology depends on the job. MQTT is useful at the device edge because it is lightweight and supports publish/subscribe messaging. Apache Kafka, Redpanda, or Pulsar fit durable event streams with multiple downstream consumers. Spark fits batch analytics and micro-batch streaming, while Apache Flink fits low-latency stateful streaming and event-time windows. Parquet in object storage works well for large analytical scans, while InfluxDB, TimescaleDB, Cassandra, or ClickHouse may fit serving paths with specific write and query patterns.

Legacy Hadoop terminology is still useful when reviewing older IoT data-lake designs. HDFS separates storage metadata from storage blocks: the NameNode tracks the file-system namespace and block placement, while DataNodes store and serve the blocks. Hadoop MapReduce v1 separated coordination from worker execution in a similar way: the JobTracker accepted jobs, split them into map and reduce tasks, assigned work, and tracked progress, while TaskTrackers ran those tasks on worker nodes near the data. Modern stacks often replace those exact roles with YARN, Spark, Flink, or managed services, but the review question is the same: which component owns scheduling, which component owns data placement, and how does the system recover when a worker fails?

The selection diagram in Figure 53.1 applies the chapter's workload numbers before any product family is assigned a job.

Read Figure 53.1 from the 24,000-event-per-second workload through stream, process, storage, and serving roles. Check the eight-partition minimum and 12 MB/s ingest first, then the 36 MB/s fan-out created by three readers, the replay window against recovery time, and finally schema, lineage, and idempotent-sink controls. The order connects technology choice to the running narrative: each tool receives one primary job, and the composed path must prove throughput, recovery, and governance together.

Hadoop is the classic open-source example of this split-role design. It is not one database; it is a distributed data platform whose parts divide storage, processing, resource management, and shared utilities. Hadoop Common supplies shared libraries used by the other services. Hadoop Distributed File System (HDFS) stores large files as replicated blocks across worker machines. MapReduce runs batch jobs over those blocks, and YARN allocates CPU, memory, and scheduling capacity to applications running on the cluster.

In HDFS, the NameNode keeps filesystem metadata: file names, directories, block locations, permissions, and the map from each logical file to its physical blocks. DataNodes store the actual block replicas and periodically report health and block inventory back to the NameNode. That separation is the main review idea for IoT learners: metadata tells the cluster where evidence lives, while replicated data blocks make large historical telemetry, logs, images, or model-training files durable enough for batch analysis.

The legacy HDFS example matters because durable data and the metadata needed to find it fail in different ways. The replication diagram in Figure 53.2 makes that ownership split concrete before the chapter returns to governance and recovery contracts.

The replication diagram Figure 53.2 places NAMENODE - filesystem metadata above mappings from /users/sensors/part-0.log and part-1.log to numbered blocks and replica counts. Below, DataNode A through DataNode H hold the coloured block replicas themselves, while the heartbeat note shows DataNodes reporting inventory and health back to the NameNode. Replication therefore protects block bytes across storage workers; it does not make namespace and placement metadata irrelevant. The broader technology record must name both responsibilities, their recovery evidence, and the governance controls that make a replayable dataset findable.

Overview Knowledge Check

53.3 Size Brokers and Stores Together

Technology choice starts with a rate and retention model. Broker partitions must absorb producer bursts and feed consumers fast enough. Stream processors must keep state for the windows and joins they compute. Storage must handle raw replay, curated analytics, and serving queries without forcing one format to satisfy every access pattern. The same 20,000 events per second can be easy or hard depending on payload size, partition key, late data, fan-out, and retention.

Use a small number of explicit paths. A live path can read Kafka partitions into Flink for event-time alerts and write idempotent results to a serving store. A batch path can compact raw Parquet into Delta Lake or Iceberg tables for Spark and SQL analytics. A dashboard path can read hourly aggregates from TimescaleDB, InfluxDB, ClickHouse, or a warehouse. Each path should have its own freshness, replay, and cost target.

Worked example: broker partition sizing
ingest rate: 24,000 events/s
average event size: 500 bytes
target partition capacity for this workload: 3,000 events/s

minimum partitions by event rate:
24,000 / 3,000 = 8 partitions

raw ingest throughput:
24,000 events/s * 500 bytes = 12,000,000 bytes/s
12,000,000 bytes/s = 12 MB/s using decimal units

consumer fan-out:
alert processor reads the stream once = 12 MB/s
lake writer reads the stream once = 12 MB/s
dashboard aggregate job reads the stream once = 12 MB/s
total consumer read pressure = 36 MB/s

design implication:
Start above the minimum partition count if growth, reprocessing, or uneven
device keys are expected. Then verify partition lag and hot keys under load
instead of assuming the average rate is evenly distributed.

Practitioner Knowledge Check

53.4 Tools Need Operational Contracts

The technology stack is only correct if it can be operated during failures, upgrades, replay, and schema changes. Kafka retention must be long enough for consumers to recover. Stream checkpoints must be stored durably and tested during redeployments. Lakehouse compaction must keep file sizes queryable without deleting evidence needed for replay. Serving stores need retention and downsampling policies. Schema registry and catalog rules need compatibility checks before firmware changes reach production.

Delivery guarantees also depend on the whole path. Kafka can store events durably, Flink can checkpoint state, and a sink can support idempotent writes, but the final outcome is only replay-safe when the event id, checkpoint, sink write, and downstream table semantics agree. A system that says "exactly once" in one layer can still double-count if the serving store uses blind inserts or if enrichment creates nondeterministic keys.

Worked example: recovery window
broker retention: 72 hours
consumer outage: 9 hours
normal consumer catch-up rate: 30,000 events/s
ingest rate during recovery: 24,000 events/s

net catch-up rate:
30,000 - 24,000 = 6,000 events/s

events accumulated during outage:
24,000 events/s * 9 hours * 3,600 s/hour = 777,600,000 events

catch-up time:
777,600,000 / 6,000 events/s = 129,600 s = 36 hours

total time from outage start to full recovery:
9 hours outage + 36 hours catch-up = 45 hours

design implication:
72-hour broker retention is enough for this case, but only if the consumer
really can sustain 30,000 events/s while new data continues to arrive. A slower
catch-up rate or a second outage could exceed retention and force data loss or
manual backfill from raw object storage.

Under the Hood Knowledge Check