57 Big Data Case Studies

57.2 Case Studies Start With Decisions

An IoT big-data case study is useful only when it connects a decision to the evidence path that supports it. "A city has millions of readings" is not enough. The case study must state the operational decision, the sensors involved, the sampling rates, the data contracts, the processing path, the retention rule, and the measured outcome. Otherwise it is only a volume story.

Good case studies also separate raw data from decision evidence. A traffic platform may keep raw video locally or in an archive, but dashboards often need counts, speeds, incident flags, confidence scores, and short event clips. A maintenance platform may keep raw vibration bursts for review, but streaming alerts often depend on features such as RMS, spectral energy, temperature trend, load state, and model version. The architecture is credible when the retained evidence can be replayed, audited, and tied back to a decision.

Before treating a case study as proof of scale, inspect the two system diagrams in Figure 57.1 and ask which constraint actually exceeds one machine.

Read Figure 57.1 from the single-laptop path to the partitioned path. One machine keeps coordination simple but concentrates CPU, memory, storage, and failure risk; the distributed system divides data and work across nodes, then pays for partitioning, transfer, coordination, and recovery. The picture does not claim that distribution is automatically better. It connects the chapter's case studies to the running decision rule: scale out only after measured volume, velocity, deadline, or resilience requirements justify the extra operating contract.

Overview Knowledge Check

57.3 Build the Case Ledger First

A practical case ledger converts a scenario into rates, payloads, storage tiers, and query paths. The numbers below are a representative smart-city workload, not a claim about one named city. The point is the calculation method: quantify the streams, separate metadata from heavy media, and make storage and processing choices from measured demand.

The same ledger also prevents architectural overreach. Parking sensors and air-quality stations may fit batch summaries. Traffic-loop counts may need sub-minute stream processing. Camera-derived events may need edge inference and privacy controls. A single relational database or a single data lake table is unlikely to serve all access patterns cleanly.

Worked case: representative smart-city platform
parking occupancy:
800 sensors * 2,880 readings/day * 80 bytes = 184,320,000 bytes/day = 0.184 GB/day

air quality:
200 stations * 1,440 readings/day * 220 bytes = 63,360,000 bytes/day = 0.063 GB/day

streetlight status:
15,000 lights * 1,440 readings/day * 90 bytes = 1,944,000,000 bytes/day = 1.944 GB/day

traffic-loop counts:
3,000 loops * 86,400 readings/day * 120 bytes = 31,104,000,000 bytes/day = 31.104 GB/day

camera event metadata:
600 cameras * 172,800 events/day * 400 bytes = 41,472,000,000 bytes/day = 41.472 GB/day

event clips selected by edge inference:
600 cameras * 200 clips/day * 2 MB = 240,000 MB/day = 240 GB/day

daily total:
0.184 + 0.063 + 1.944 + 31.104 + 41.472 + 240 = 314.767 GB/day

design reading:
The platform is not dominated by parking or air-quality records. It is dominated
by selected camera clips and traffic streams. That pushes the architecture toward
edge inference, hot summaries, lifecycle-managed object storage, and strict access
control for media.

A second case shows why big-data systems do not always need the same stack. In cold-chain logistics, the main decision is whether a shipment stayed within temperature limits. A gateway may buffer BLE or cellular sensor readings during connectivity gaps, then upload signed time-series records. TimescaleDB, InfluxDB, or a cloud time-series service can serve recent queries, while object storage preserves signed audit bundles. The critical evidence is not petabytes of volume; it is timestamp integrity, custody handoff, clock sync, tamper state, and rule evaluation.

Practitioner Knowledge Check

57.4 Validate Economics and Evidence

Case studies become engineering assets when their numbers can be checked. Daily volume should follow from device count, sample rate, and payload size. Query latency should be measured from real dashboards or replay tests. Storage cost should separate hot, warm, and archive data. Outcomes should name the baseline and the measurement window: fewer truck rolls, shorter incident response, reduced energy use, lower spoilage, or faster analyst workflow.

Cloud prices, storage classes, and managed-service charges change, so procurement should always refresh current pricing. The calculation pattern is still stable: estimate retained GB by tier, multiply by the contract price for that tier, add compute, network egress, observability, and operations labor, then compare the result to the operational value.

Worked example: tiered storage for the smart-city case
daily generated evidence: 314.767 GB/day
hot retention target: 7 days
warm retention target: 90 days total
archive policy after 90 days: keep only 20 percent of records and clips

example storage prices from a contract, not a universal quote:
hot object storage: \$0.023 per GB-month
warm object storage: \$0.012 per GB-month
archive object storage: \$0.004 per GB-month

all data hot for 90 days:
314.767 * 90 = 28,329 GB
28,329 * \$0.023 = \$651.57/month

tiered first 90 days:
hot: 314.767 * 7 = 2,203 GB; 2,203 * \$0.023 = \$50.67/month
warm: 314.767 * 83 = 26,126 GB; 26,126 * \$0.012 = \$313.51/month
first-90-day tiered total = \$364.18/month

archive after 90 days for a seven-year record:
archive days = (7 * 365) - 90 = 2,465 days
archive retained volume = 314.767 * 2,465 * 0.20 = 155,178 GB
archive monthly storage = 155,178 * \$0.004 = \$620.71/month

design reading:
The storage bill is not the only cost, but the calculation proves why lifecycle
policy and archive sampling are architecture controls, not cleanup tasks. It
also shows which assumptions must be reviewed when actual access patterns arrive.

57.4.1 SKA Capacity and Backhaul Envelope

The Square Kilometre Array is a useful scale exercise because it forces three units into the same ledger: annual volume, sustained ingest rate, and backhaul capacity. Treat the source figures as a historical order-of-magnitude scenario, not as a current SKA design guarantee. Start with a 2016 global-IP-traffic reference of approximately 1.2 zettabytes per year and ask what “ten times that” means arithmetically.

The antenna field in Figure 57.2 supplies the physical scale behind that ledger: many instruments operate together at a remote high-altitude site, while the arithmetic below asks what can be retained or transported after collection.

Photo: ESO/S. Seip, CC BY 4.0{target=“_blank” rel=“noopener”}

In Figure 57.2, the separated dishes are not independent archives: they are parts of a coordinated instrument whose observations must be combined. The photograph therefore illustrates the collection footprint, while the worked example supplies the explicit rate and reduction assumptions that a photograph cannot establish.

Using decimal storage units,

$$1\ \text{ZB}=10^{21}\ \text{bytes}=10^9\ \text{TB}=10^{12}\ \text{GB}.$$

The ten-times scenario is therefore

$$V_{year}=10(1.2\ \text{ZB})=12\ \text{ZB} =1.2\times10^{10}\ \text{TB} =12{,}000{,}000{,}000{,}000\ \text{GB per year}.$$

Annual capacity hides the ingest problem, so divide by the seconds in a 365-day year:

$$R_{raw}=\frac{12\times10^{21}\ \text{bytes}} {365\times24\times3600\ \text{s}} \approx3.81\times10^{14}\ \text{bytes/s}.$$

That is about 381 TB/s, or 3.05 Pbit/s, sustained. One day at that rate produces about 32.9 EB. Even ignoring replication, parity, metadata, failed media, cooling, and write bandwidth, storing the raw year would require the usable capacity of twelve billion 1 TB drives. The absurdity is the lesson: “save everything and decide later” is not an architecture at this scale.

Now test the transport boundary. A dedicated 400 Gbit/s link carries at most

$$V_{400G}=\frac{400\times10^9}{8}(365\times24\times3600) \approx1.58\times10^{18}\ \text{bytes/year}=1.58\ \text{EB/year}$$

before framing, protection, outages, or shared use. The raw 12 ZB/year stream is roughly 7,600 times larger. Equivalently, a 400 Gbit/s Perth backhaul could carry only about 0.013% of the hypothetical raw stream. Adding one faster link does not close that gap.

Boundary	Raw scenario	Design consequence
Annual volume	12 ZB/year	On-site storage cannot be a raw, indefinite archive
Average ingest	381 TB/s	Memory, buses, accelerators, and writers must be parallel and continuously provisioned
400 Gbit/s backhaul	0.05 TB/s ideal payload rate	Raw off-site transfer needs about 7,600:1 reduction before overhead
One day of raw data	32.9 EB	A short outage still creates an enormous local buffer obligation

The system must therefore reduce data near the instruments. A defensible pipeline calibrates and flags samples, channelizes or correlates signals, rejects known interference under recorded rules, forms scientifically useful products, and retains only the raw windows needed for calibration, anomaly review, or reproducibility. Each reduction stage needs a ratio and a loss statement:

$$R_{out}=R_{raw}\prod_{i=1}^{n} r_i, \qquad 0<r_i\leq1,$$

where $r_i$ is the retained fraction after stage $i$. To fit the illustrative 400 Gbit/s link, the product of all retained fractions must be no more than about $1/7600$. That target should be allocated across reversible formatting, aggregation, correlation, event selection, and retention rather than hidden behind the word “compression.” The case record must say which scientific questions each irreversible step makes impossible, which raw windows remain available for audit, and how a backhaul outage changes the on-site queue.

Replay Evidence

Keep topic offsets, schema ids, table versions, raw object paths, and job versions so a result can be rebuilt.

Privacy Evidence

Record access logs, redaction state, retention expiry, consent or policy basis, and review decisions for sensitive data.

Cost Evidence

Track retained GB by tier, query scan bytes, compute hours, network egress, small-file counts, and owner tags.

Outcome Evidence

Compare decisions against a baseline: downtime avoided, incidents detected, trips reduced, energy saved, or spoilage prevented.

The under-the-hood lesson is that a case study must be falsifiable. If the dashboard claims sub-second traffic visibility, logs should show event-time delay, partition lag, query latency, and missing-window rate. If the maintenance system claims avoided downtime, the work orders should connect model alerts to inspections, repairs, and avoided failures. If the architecture claims a lower cost, the cost report should show which lifecycle rule, partition layout, and materialized view produced the reduction.

Under-the-Hood Knowledge Check