Analytics & ML · Study deck
Big Data Case Studies
Picture a city team claiming that millions of road readings will reduce bus delays.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: A defensible pipeline calibrates and flags samples, channelizes or correlates signals, rejects known interference under recorded rules, forms scientifically useful products, and retains only the raw windows needed for calibration, anomaly review, or reproducibility.
- Explain: The first question is not "which big-data tool is fashionable?" It is "which decision is being made, which evidence proves it, and what must be retained if the decision is challenged later?".
- Explain: Under the Hood explains split work, event time, saved state, data rules, and the limits of transferring a successful case to a new place.
Major section
Start With the Story
The number sounds impressive, but the service planner needs to know which decision changes, how quickly it changes, and what evidence shows that the change helped.
- Large volume may justify work across several computers, but it does not prove value, truth, or fairness.
- Under the Hood explains split work, event time, saved state, data rules, and the limits of transferring a successful case to a new place.
- A device has produced messy evidence, an analytic step is about to change an alert or control decision, and someone has to explain why the result should be trusted.
Major section
Case Studies Start With Decisions
An IoT big-data case study is useful only when it connects a decision to the evidence path that supports it. "A city has millions of readings" is not enough.
- The architecture is credible when the retained evidence can be replayed, audited, and tied back to a decision.
Major section
Case Studies Start With Decisions (continued)
Otherwise it is only a volume story.
- The case study must state the operational decision, the sensors involved, the sampling rates, the data contracts, the processing path, the retention rule, and the measured outcome.
- The picture does not claim that distribution is automatically better.
- Comfort regressions, actuator limits, model bias, and uncontrolled manual overrides.
Major section
Case Studies Start With Decisions (continued)
Decision Name the action: reroute traffic, dispatch maintenance, change a cooling setpoint, or quarantine a bad sensor.
- A traffic platform may keep raw video locally or in an archive, but dashboards often need counts, speeds, incident flags, confidence scores, and short event clips.
- The first question is not "which big-data tool is fashionable?" It is "which decision is being made, which evidence proves it, and what must be retained if the decision is challenged later?".
- Architecture Map evidence through MQTT, Kafka, Flink, Spark, object storage, data lake tables, and serving stores.
Major section
Build the Case Ledger First
The numbers below are a representative smart-city workload, not a claim about one named city.
- The point is the calculation method: quantify the streams, separate metadata from heavy media, and make storage and processing choices from measured demand.
- Parking sensors and air-quality stations may fit batch summaries.
- Case-study math should change the design.
Major section
Build the Case Ledger First (continued)
Traffic-loop counts may need sub-minute stream processing.
- Camera-derived events may need edge inference and privacy controls.
- A single relational database or a single data lake table is unlikely to serve all access patterns cleanly.
- design reading: The platform is not dominated by parking or air-quality records.
Major section
Build the Case Ledger First (continued)
The same ledger also prevents architectural overreach.
- If one stream contributes 76 percent of daily volume, retention, privacy, compression, and query strategy should focus there first.
- A second case shows why big-data systems do not always need the same stack.
- In cold-chain logistics, the main decision is whether a shipment stayed within temperature limits.
Major section
Validate Economics and Evidence
Case studies become engineering assets when their numbers can be checked.
- Daily volume should follow from device count, sample rate, and payload size.
- Storage cost should separate hot, warm, and archive data.
- Outcomes should name the baseline and the measurement window: fewer truck rolls, shorter incident response, reduced energy use, lower spoilage, or faster analyst workflow.
Major section
Validate Economics and Evidence (continued)
The ten-times scenario is therefore.
- Cloud prices, storage classes, and managed-service charges change, so procurement should always refresh current pricing.
- design reading: The storage bill is not the only cost, but the calculation proves why lifecycle policy and archive sampling are architecture controls, not cleanup tasks.
- A dedicated 400 Gbit/s link carries at most.
Major section
Validate Economics and Evidence (continued)
The raw 12 ZB/year stream is roughly 7,600 times larger.
- It also shows which assumptions must be reviewed when actual access patterns arrive.
- The photograph therefore illustrates the collection footprint, while the worked example supplies the explicit rate and reduction assumptions that a photograph cannot establish.
- Adding one faster link does not close that gap.
Major section
Validate Economics and Evidence (continued)
Even ignoring replication, parity, metadata, failed media, cooling, and write bandwidth, storing the raw year would require the usable capacity of twelve billion 1 TB drives.
- The absurdity is the lesson: “save everything and decide later” is not an architecture at this scale.
- Equivalently, a 400 Gbit/s Perth backhaul could carry only about 0.013% of the hypothetical raw stream.
- where $r_i$ is the retained fraction after stage $i$.
Major section
Validate Economics and Evidence (continued)
A defensible pipeline calibrates and flags samples, channelizes or correlates signals, rejects known interference under recorded rules, forms scientifically useful products, and retains only the raw windows needed for calibration, anomaly review, or reproducibility.
- To fit the illustrative 400 Gbit/s link, the product of all retained fractions must be no more than about $1/7600$.
- Replay Evidence Keep topic offsets, schema ids, table versions, raw object paths, and job versions so a result can be rebuilt.
- The under-the-hood lesson is that a case study must be falsifiable.
- The system must therefore reduce data near the instruments.
Major section
Summary
IoT big-data case studies should start with a decision, not with a tool list or a headline sensor count.
- A case ledger connects sensor rates, payload sizes, evidence contracts, ingestion paths, storage tiers, query systems, operating metrics, and measured outcomes.
- Industrial, logistics, and building-energy cases may have lower total volume but higher evidence-integrity requirements.
- Strong case studies are falsifiable: their data rates, replay paths, cost assumptions, access controls, and outcome claims can be checked.
Deck summary
Key takeaways
The number sounds impressive, but the service planner needs to know which decision changes, how quickly it changes, and what evidence shows that the change helped.
- An IoT big-data case study is useful only when it connects a decision to the evidence path that supports it. "A city has millions of readings" is not enough.
- Otherwise it is only a volume story.
- Decision Name the action: reroute traffic, dispatch maintenance, change a cooling setpoint, or quarantine a bad sensor.
- The numbers below are a representative smart-city workload, not a claim about one named city.
Retrieval practice
Recall check 1 of 3
Data Dora says: answer from memory, then check your reasoning.
Q1What makes an IoT big-data case study credible?
Show answer
Answer: A A useful case study is an evidence-backed architecture ledger, not only a story about scale.
Retrieval practice
Recall check 2 of 3
Data Dora says: answer from memory, then check your reasoning.
Q2In the worked smart-city case, selected camera clips contribute 240 GB/day out of about 315 GB/day. What is the best first architecture response?
Show answer
Answer: C The largest and most sensitive stream should receive early attention for edge processing, lifecycle policy, access control, and evidence retention.
Retrieval practice
Recall check 3 of 3
Data Dora says: answer from memory, then check your reasoning.
Q3Which evidence best turns a big-data case study from a story into an auditable engineering result?
Show answer
Answer: C Auditable case studies preserve the calculations and runtime artifacts needed to check both architecture behavior and business outcomes.
Print reference
Answers
Answer key.
- A · A useful case study is an evidence-backed architecture ledger, not only a story about scale.
- C · The largest and most sensitive stream should receive early attention for edge processing, lifecycle policy, access control, and evidence retention.
- C · Auditable case studies preserve the calculations and runtime artifacts needed to check both architecture behavior and business outcomes.