Analytics & ML · Study deck

Big Data Technologies

A broker is a service that accepts messages and routes or stores them for receivers.

Data Dora is your guide for this deck.

datatechnologies
Data Dora, the module guide, in a scene from this chapter.
iotclass.org

After studying this chapter

Learning objectives

You will be able to:

  • Explain: If one database is expected to buffer device bursts, run stream windows, serve dashboards, store cold history, and govern schemas, the architecture is hiding several different requirements in one box.
  • Explain: The order connects technology choice to the running narrative: each tool receives one primary job, and the composed path must prove throughput, recovery, and governance together.
  • Explain: Parquet in object storage works well for large analytical scans, while InfluxDB, TimescaleDB, Cassandra, or ClickHouse may fit serving paths with specific write and query patterns.
  • Explain: A simple daily report has different needs from a live outage alarm.
iotclass.org

Major section

Start With the Story

A broker is a service that accepts messages and routes or stores them for receivers.

  • A simple daily report has different needs from a live outage alarm.
  • Copying a large-company design can add cost and hidden failure without improving either result.
  • A tool can help with scale, but it cannot create trustworthy meaning or ownership.
iotclass.org

Major section

Technologies Have Different Jobs

An IoT big-data stack is a chain of responsibilities, not a list of fashionable product names.

  • A broker buffers events and lets several consumers read the same stream.
  • Stream processors turn live events into windows, alerts, and enriched records.
  • Storage systems keep raw, curated, and serving-ready data at different costs and query speeds.

Key terms

MQTT
MQTT is useful at the device edge because it is lightweight and supports publish/subscribe messaging.
Legacy Hadoop terminology
Legacy Hadoop terminology is still useful when reviewing older IoT data-lake designs.
Hadoop
Hadoop is the classic open-source example of this split-role design.
Technology fit starts with the chapter's workload numbers, then proves the selected broker, processor, store, serving path, and governance controls can handle ingest rate, fan-out, replay, recovery, and audit evidence.
Technology fit starts with the chapter's workload numbers, then proves the selected broker, processor, store, serving path, and governance controls can handle ingest rate, fan-out, replay, recovery, and audit evidence.
iotclass.org

Major section

Technologies Have Different Jobs (continued)

Governance tools make schemas, access, lineage, and replay visible.

  • The right technology depends on the job.
  • Apache Kafka, Redpanda, or Pulsar fit durable event streams with multiple downstream consumers.
  • Spark fits batch analytics and micro-batch streaming, while Apache Flink fits low-latency stateful streaming and event-time windows.
iotclass.org

Major section

Technologies Have Different Jobs (continued)

The order connects technology choice to the running narrative: each tool receives one primary job, and the composed path must prove throughput, recovery, and governance together.

  • Parquet in object storage works well for large analytical scans, while InfluxDB, TimescaleDB, Cassandra, or ClickHouse may fit serving paths with specific write and query patterns.
  • Legacy Hadoop terminology is still useful when reviewing older IoT data-lake designs.
  • Partitions provide parallel read/write paths and let consumers replay from offsets.
iotclass.org

Major section

Technologies Have Different Jobs (continued)

If one database is expected to buffer device bursts, run stream windows, serve dashboards, store cold history, and govern schemas, the architecture is hiding several different requirements in one box.

  • Event LogKafka, Redpanda, Pulsar, or managed stream service that buffers events and supports replay.
  • StorageObject storage, Parquet, Delta Lake, Iceberg, time-series databases, warehouses, and serving stores.
  • Event-time windows, watermarks, and stateful operators are first-class concepts.
iotclass.org

Major section

Technologies Have Different Jobs (continued)

Exactly-once outcomes still require compatible checkpoints and idempotent sinks.

  • Columnar files and table formats support large scans, compaction, and batch jobs.
  • Small-file sprawl can make cheap object storage expensive to query.
  • No serving store removes the need for quality flags and schema control.
  • Hadoop is the classic open-source example of this split-role design.
iotclass.org

Major section

Technologies Have Different Jobs (continued)

Hadoop Distributed File System (HDFS) stores large files as replicated blocks across worker machines.

  • MapReduce runs batch jobs over those blocks, and YARN allocates CPU, memory, and scheduling capacity to applications running on the cluster.
  • In HDFS, the NameNode keeps filesystem metadata: file names, directories, block locations, permissions, and the map from each logical file to its physical blocks.
  • Replication therefore protects block bytes across storage workers; it does not make namespace and placement metadata irrelevant.
Metadata (which blocks belong to which file, and how many replicas each needs) lives on the NameNode; the replicated bytes themselves live on DataNodes.
Metadata (which blocks belong to which file, and how many replicas each needs) lives on the NameNode; the replicated bytes themselves live on DataNodes.
iotclass.org

Major section

Technologies Have Different Jobs (continued)

That separation is the main review idea for IoT learners: metadata tells the cluster where evidence lives, while replicated data blocks make large historical telemetry, logs, images, or model-training files durable enough for batch analysis.

  • The legacy HDFS example matters because durable data and the metadata needed to find it fail in different ways.
  • The replication diagram in Figure: Metadata (which blocks belong to which file makes that ownership split concrete before the chapter returns to governance and recovery contracts.
  • The broader technology record must name both responsibilities, their recovery evidence, and the governance controls that make a replayable dataset findable.
iotclass.org

Major section

Size Brokers and Stores Together · Tools Need Operational Contracts

Technology choice starts with a rate and retention model.

  • Broker partitions must absorb producer bursts and feed consumers fast enough.
  • Stream processors must keep state for the windows and joins they compute.
  • Storage must handle raw replay, curated analytics, and serving queries without forcing one format to satisfy every access pattern.

Key terms

Partition count
Partition count is not just a capacity number.
Which field changes
Which field changes are compatible with existing producers and consumers.
iotclass.org

Deck summary

Key takeaways

A broker is a service that accepts messages and routes or stores them for receivers.

  • An IoT big-data stack is a chain of responsibilities, not a list of fashionable product names.
  • Governance tools make schemas, access, lineage, and replay visible.
  • The order connects technology choice to the running narrative: each tool receives one primary job, and the composed path must prove throughput, recovery, and governance together.
  • If one database is expected to buffer device bursts, run stream windows, serve dashboards, store cold history, and govern schemas, the architecture is hiding several different requirements in one box.
iotclass.org

Retrieval practice

Recall check 1 of 5

Data Dora says: answer from memory, then check your reasoning.

Q1A team must replay device events after a schema fix and explain which result used which version. Which contract is needed?

AKeep only dashboard totals, discarding events and offsets for the replay.
BKeep broker throughput charts but discard the event history to replay.
CRetain raw events, offsets, versions, lineage, access rules, and replay owner.
DUse a new schema name while leaving old results without a version link.
Show answer

Answer: C Reprocessing needs both the source history and the rules that made each derived result.

iotclass.org

Retrieval practice

Recall check 2 of 5

Data Dora says: answer from memory, then check your reasoning.

Q2Why should an IoT big-data architecture assign a primary job to each technology?

ABecause separate jobs justify adding every named tool to the initial stack.
BBecause object storage can own ingestion and live stream processing alone.
CBecause tools have different jobs, failure modes, and scaling limits along the data chain.
DBecause a job assignment proves that newer products have more features.
Show answer

Answer: C Technology selection should map each tool to the role it can operate and scale reliably.

iotclass.org

Retrieval practice

Recall check 3 of 5

Data Dora says: answer from memory, then check your reasoning.

Q3A stream receives 24,000 events/s and a partition is budgeted for 3,000 events/s in this workload. What is the minimum partition count before adding growth headroom?

A8 partitions, from 24,000 divided by 3,000 events per second each.
B3 partitions, using the per-partition rate as the total count.
C12 partitions, using raw megabytes rather than event throughput.
D36 partitions, multiplying by three consumers without a rate reason.
Show answer

Answer: A Minimum partitions from this budget equal ingest event rate divided by target events per second per partition.

iotclass.org

Retrieval practice

Recall check 4 of 5

Data Dora says: answer from memory, then check your reasoning.

Q4A consumer is down for 9 hours while 24,000 events/s arrive. It later catches up at a net 6,000 events/s faster than the live ingest rate. How long after the outage begins until it is fully caught up?

A9 hours because the consumer only needs to restart.
B36 hours because the outage duration does not count.
C72 hours because broker retention is 72 hours.
D45 hours: 9 outage hours plus 36 hours of catch-up.
Show answer

Answer: D Recovery time combines the outage duration and the time needed to process the accumulated backlog.

iotclass.org

Retrieval practice

Recall check 5 of 5

Data Dora says: answer from memory, then check your reasoning.

Q5A platform needs raw replay, long-term analysis, and fast dashboard reads. What storage plan fits?

AKeep only dashboard aggregates, leaving no raw records to replay.
BPut raw history in the broker forever instead of a long-term tier.
CForce raw replay and dashboard queries through one serving table.
DUse storage tiers matched to raw replay, analysis, and fast reads.
Show answer

Answer: D Different query and replay needs can require different storage paths.

iotclass.org

Print reference

Answers

Answer key.

  1. C · Reprocessing needs both the source history and the rules that made each derived result.
  2. C · Technology selection should map each tool to the role it can operate and scale reliably.
  3. A · Minimum partitions from this budget equal ingest event rate divided by target events per second per partition.
  4. D · Recovery time combines the outage duration and the time needed to process the accumulated backlog.
  5. D · Different query and replay needs can require different storage paths.
iotclass.org