Analytics & ML · Study deck
Big Data Fundamentals
Telemetry means records sent from a system so its state can be watched.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: The trade-off flipped rather than vanished — separate networks wasted capacity, the converged network shares fate, and the IoT platform at the lower right now competes with video for the same pipe.
- Explain: Late events, clock skew, duplicate retries, and out-of-order batches can make a dashboard or alert incorrect unless the pipeline treats time as part of the data contract.
- Explain: Backpressure tells upstream components that the sink is slower than the source so the system can buffer, scale, sample, or shed noncritical work instead of failing unpredictably.
- translate the Five Vs into IoT requirements
Major section
Begin With One Day of Device Records
Telemetry means records sent from a system so its state can be watched.
- The first choice is not a brand of data tool.
- Keeping more can help later study, but it raises cost, privacy, and search load.
- Fast intake can protect fresh work, but it may pass poor data unless checks keep up.
Major section
Big Data as Engineering Contract
Big data in IoT is not just a synonym for a large database.
- A temperature history table, a vibration stream, a fleet of camera summaries, and a connected-vehicle telemetry feed can all be "big" for different reasons.
- Volume asks how many bytes and records must be retained.
Major section
Big Data as Engineering Contract (continued)
The THE TRADE-OFF card underneath states the economics that killed this model — every new service, IoT telemetry included, would have demanded yet another dedicated build.
- The trade-off flipped rather than vanished — separate networks wasted capacity, the converged network shares fate, and the IoT platform at the lower right now competes with video for the same pipe.
- Sensor fleets extend exactly this curve, which is why the rest of this chapter turns the vocabulary of scale into an engineering contract instead of a slogan.
- Velocity asks how quickly events arrive and how soon they must affect a decision.
Major section
When the Source Uses Seven Vs
Its two additional labels make useful review questions without replacing this chapter's five-part engineering contract.
- Variability asks whether meaning changes with context, as it can for language, hashtags, geospatial data, multimedia, or sensor events.
- Visualization asks whether a pictorial or graphical presentation helps a decision-maker see an analytic result or pattern.
- The important evidence is the requirement each label exposes, not whether a team says "five" or "seven.".
Major section
When the Source Uses Seven Vs (continued)
Volume How many records and bytes arrive, and how long raw, cleaned, and aggregated data must be retained.
- Velocity How quickly events arrive, queue, and need to be transformed into alerts, dashboards, or control actions.
- Veracity The trust controls for duplicates, gaps, calibration drift, schema drift, outliers, and late-arriving events.
- Motor vibration anomalies may need seconds, while monthly reports can wait.
- Validate at ingestion, keep quality flags, and design idempotent writes.
Major section
Size Pipeline Before Tools
The tool names matter only after the workload is sized.
- A small campus deployment may work with a time-series database and nightly Parquet export.
- A regional fleet may need Kafka partitions, schema registry, stream processors, and a raw/curated data lake.
- Partitioning is part of the data model.
Major section
Size Pipeline Before Tools (continued)
hourly aggregate retention: 10,000 sensors 24 hourly records/day 120 bytes = 28,800,000 bytes/day 28,800,000 bytes/day = 28.8 MB/day.
- Kafka partitions, Parquet folder layout, and database shard keys should spread load across time and devices.
- A poor key can create a hot partition even when the total cluster size looks adequate.
- Using processing time when event time is required for correct windows.
Major section
Time, State, and Backpressure
Idempotent writes keep retries from double-counting the same reading.
- Devices report event time, gateways add arrival time, brokers preserve ordering only within a partition, and stream processors must decide when a window is complete.
- Late events, clock skew, duplicate retries, and out-of-order batches can make a dashboard or alert incorrect unless the pipeline treats time as part of the data contract.
- design implication: A short sink slowdown can create a large queue.
Major section
Time, State, and Backpressure (continued)
The pipeline needs lag alarms, autoscaling rules, replay capacity, and a policy for dropping or downsampling noncritical streams before critical alerts are affected.
- Watermarks tell a stream processor how long to wait for late data before closing a window.
- Event Time The timestamp when the reading was measured.
- A device resend makes one physical reading count twice.
Major section
Time, State, and Backpressure (continued)
Backpressure tells upstream components that the sink is slower than the source so the system can buffer, scale, sample, or shed noncritical work instead of failing unpredictably.
- Watermark The processor's estimate that older late events are unlikely enough to close a window.
- Idempotency A write design where replaying the same event does not create duplicate facts or double-counted totals.
- Window totals change after a dashboard or alert has already used them.
Major section
Time, State, and Backpressure (continued)
Flink and Spark Structured Streaming both support event-time windows and watermarks.
- Kafka keys, transactional writes, Delta Lake, and Iceberg can support replay-safe pipelines when configured carefully.
- A firmware update changes a field name, unit, or enum value.
- Version schemas, validate at ingress, and reject or quarantine incompatible events.
Deck summary
Key takeaways
Telemetry means records sent from a system so its state can be watched.
- Big data in IoT is not just a synonym for a large database.
- The THE TRADE-OFF card underneath states the economics that killed this model — every new service, IoT telemetry included, would have demanded yet another dedicated build.
- Its two additional labels make useful review questions without replacing this chapter's five-part engineering contract.
- Volume How many records and bytes arrive, and how long raw, cleaned, and aggregated data must be retained.
Retrieval practice
Recall check 1 of 3
Data Dora says: answer from memory, then check your reasoning.
Q1Which statement is the strongest IoT big-data requirement?
Show answer
Answer: A A useful requirement gives concrete scale, rate, retention, and decision constraints.
Retrieval practice
Recall check 2 of 3
Data Dora says: answer from memory, then check your reasoning.
Q2A fleet sends 10,000 events/s and each parsed event is 80 bytes. About how much raw data arrives per day before compression?
Show answer
Answer: A Raw daily volume comes from events per second times payload bytes times seconds per day.
Retrieval practice
Recall check 3 of 3
Data Dora says: answer from memory, then check your reasoning.
Q3A stream ingests 12,000 events/s but the sink can write only 9,000 events/s for 10 minutes. What is the main engineering issue?
Show answer
Answer: A Backlog is the difference between source and sink rates multiplied by the slowdown duration.
Print reference
Answers
Answer key.
- A · A useful requirement gives concrete scale, rate, retention, and decision constraints.
- A · Raw daily volume comes from events per second times payload bytes times seconds per day.
- A · Backlog is the difference between source and sink rates multiplied by the slowdown duration.