Analytics & ML · Study deck
Big Data Technologies
A broker is a service that accepts messages and routes or stores them for receivers.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: That separation is the main review idea for IoT learners: metadata tells the cluster where evidence lives, while replicated data blocks make large historical telemetry, logs, images, or model-training files durable enough for batch analysis.
- Explain: Kafka can store events durably, Flink can checkpoint state, and a sink can support idempotent writes, but the final outcome is only replay-safe when the event id, checkpoint, sink write, and downstream table semantics agree.
- Explain: A slower catch-up rate or a second outage could exceed retention and force data loss or manual backfill from raw object storage.
Major section
Start With the Story
A broker is a service that accepts messages and routes or stores them for receivers.
- A simple daily report has different needs from a live outage alarm.
- Copying a large-company design can add cost and hidden failure without improving either result.
- A tool can help with scale, but it cannot create trustworthy meaning or ownership.
Major section
Technologies Have Different Jobs
An IoT big-data stack is a chain of responsibilities, not a list of fashionable product names.
- A broker buffers events and lets several consumers read the same stream.
- Stream processors turn live events into windows, alerts, and enriched records.
- Storage systems keep raw, curated, and serving-ready data at different costs and query speeds.
Major section
Technologies Have Different Jobs (continued)
Governance tools make schemas, access, lineage, and replay visible.
- The right technology depends on the job.
- Apache Kafka, Redpanda, or Pulsar fit durable event streams with multiple downstream consumers.
- Spark fits batch analytics and micro-batch streaming, while Apache Flink fits low-latency stateful streaming and event-time windows.
Major section
Technologies Have Different Jobs (continued)
The order connects technology choice to the running narrative: each tool receives one primary job, and the composed path must prove throughput, recovery, and governance together.
- Parquet in object storage works well for large analytical scans, while InfluxDB, TimescaleDB, Cassandra, or ClickHouse may fit serving paths with specific write and query patterns.
- Legacy Hadoop terminology is still useful when reviewing older IoT data-lake designs.
- Partitions provide parallel read/write paths and let consumers replay from offsets.
Major section
Technologies Have Different Jobs (continued)
If one database is expected to buffer device bursts, run stream windows, serve dashboards, store cold history, and govern schemas, the architecture is hiding several different requirements in one box.
- Event Log Kafka, Redpanda, Pulsar, or managed stream service that buffers events and supports replay.
- Storage Object storage, Parquet, Delta Lake, Iceberg, time-series databases, warehouses, and serving stores.
- Event-time windows, watermarks, and stateful operators are first-class concepts.
Major section
Technologies Have Different Jobs (continued)
Exactly-once outcomes still require compatible checkpoints and idempotent sinks.
- Columnar files and table formats support large scans, compaction, and batch jobs.
- Small-file sprawl can make cheap object storage expensive to query.
- No serving store removes the need for quality flags and schema control.
- Hadoop is the classic open-source example of this split-role design.
Major section
Technologies Have Different Jobs (continued)
Hadoop Distributed File System (HDFS) stores large files as replicated blocks across worker machines.
- MapReduce runs batch jobs over those blocks, and YARN allocates CPU, memory, and scheduling capacity to applications running on the cluster.
- In HDFS, the NameNode keeps filesystem metadata: file names, directories, block locations, permissions, and the map from each logical file to its physical blocks.
- Replication therefore protects block bytes across storage workers; it does not make namespace and placement metadata irrelevant.
Major section
Technologies Have Different Jobs (continued)
That separation is the main review idea for IoT learners: metadata tells the cluster where evidence lives, while replicated data blocks make large historical telemetry, logs, images, or model-training files durable enough for batch analysis.
- The legacy HDFS example matters because durable data and the metadata needed to find it fail in different ways.
- The replication diagram in Figure: Metadata (which blocks belong to which file makes that ownership split concrete before the chapter returns to governance and recovery contracts.
- The broader technology record must name both responsibilities, their recovery evidence, and the governance controls that make a replayable dataset findable.
Major section
Size Brokers and Stores Together
Technology choice starts with a rate and retention model.
- Broker partitions must absorb producer bursts and feed consumers fast enough.
- Stream processors must keep state for the windows and joins they compute.
- Storage must handle raw replay, curated analytics, and serving queries without forcing one format to satisfy every access pattern.
Major section
Size Brokers and Stores Together (continued)
The same 20,000 events per second can be easy or hard depending on payload size, partition key, late data, fan-out, and retention.
- A live path can read Kafka partitions into Flink for event-time alerts and write idempotent results to a serving store.
- A batch path can compact raw Parquet into Delta Lake or Iceberg tables for Spark and SQL analytics.
- Each path should have its own freshness, replay, and cost target.
Major section
Size Brokers and Stores Together (continued)
A dashboard path can read hourly aggregates from TimescaleDB, InfluxDB, ClickHouse, or a warehouse.
- Partition count is not just a capacity number.
- It affects ordering, rebalance time, file sizes, stream-task parallelism, and operational noise.
- Changing it later can be possible, but it is not free.
- One busy key can overload a single partition.
Major section
Tools Need Operational Contracts
The technology stack is only correct if it can be operated during failures, upgrades, replay, and schema changes.
- Kafka retention must be long enough for consumers to recover.
- Stream checkpoints must be stored durably and tested during redeployments.
- Lakehouse compaction must keep file sizes queryable without deleting evidence needed for replay.
- Serving stores need retention and downsampling policies.
Major section
Tools Need Operational Contracts (continued)
A slower catch-up rate or a second outage could exceed retention and force data loss or manual backfill from raw object storage.
- Schema registry and catalog rules need compatibility checks before firmware changes reach production.
- Delivery guarantees also depend on the whole path.
- Checkpoint Stateful processors need durable checkpoints and restore drills before upgrades or failures.
Major section
Tools Need Operational Contracts (continued)
Retention The broker, lake, and serving stores need explicit retention windows tied to replay, audit, and cost.
- Kafka can store events durably, Flink can checkpoint state, and a sink can support idempotent writes, but the final outcome is only replay-safe when the event id, checkpoint, sink write, and downstream table semantics agree.
- Schema Avro, Protobuf, or JSON Schema with compatibility rules keeps firmware changes reviewable.
- Broker retention expires before a broken consumer catches up.
Major section
Tools Need Operational Contracts (continued)
Which field changes are compatible with existing producers and consumers.
- A system that says "exactly once" in one layer can still double-count if the serving store uses blind inserts or if enrichment creates nondeterministic keys.
- design implication: 72-hour broker retention is enough for this case, but only if the consumer really can sustain 30,000 events/s while new data continues to arrive.
- A unit or field rename silently changes downstream calculations.
Major section
Tools Need Operational Contracts (continued)
Idempotent Sink Replay-safe writes use stable event ids, deterministic keys, upserts, or transactional table semantics.
- A redeploy starts from the latest offset but loses the state needed for windows or joins.
- Compaction reports, retention policies, query costs, and aggregate definitions.
- Small files, duplicated streams, and unbounded raw retention make the stack costly to run.
Deck summary
Key takeaways
A broker is a service that accepts messages and routes or stores them for receivers.
- An IoT big-data stack is a chain of responsibilities, not a list of fashionable product names.
- Governance tools make schemas, access, lineage, and replay visible.
- The order connects technology choice to the running narrative: each tool receives one primary job, and the composed path must prove throughput, recovery, and governance together.
- If one database is expected to buffer device bursts, run stream windows, serve dashboards, store cold history, and govern schemas, the architecture is hiding several different requirements in one box.
Retrieval practice
Recall check 1 of 3
Data Dora says: answer from memory, then check your reasoning.
Q1Why should an IoT big-data architecture assign a primary job to each technology?
Show answer
Answer: A Technology selection should map each tool to the role it can operate and scale reliably.
Retrieval practice
Recall check 2 of 3
Data Dora says: answer from memory, then check your reasoning.
Q2A stream receives 24,000 events/s and a partition is budgeted for 3,000 events/s in this workload. What is the minimum partition count before adding growth headroom?
Show answer
Answer: A Minimum partitions from this budget equal ingest event rate divided by target events per second per partition.
Retrieval practice
Recall check 3 of 3
Data Dora says: answer from memory, then check your reasoning.
Q3A consumer is down for 9 hours while 24,000 events/s arrive. It later catches up at a net 6,000 events/s faster than the live ingest rate. How long after the outage begins until it is fully caught up?
Show answer
Answer: A Recovery time combines the outage duration and the time needed to process the accumulated backlog.
Print reference
Answers
Answer key.
- A · Technology selection should map each tool to the role it can operate and scale reliably.
- A · Minimum partitions from this budget equal ingest event rate divided by target events per second per partition.
- A · Recovery time combines the outage duration and the time needed to process the accumulated backlog.