52 Big Data Fundamentals

52.3 Big Data as Engineering Contract

This section: Big Data as Engineering Contract — Big data in IoT is not just a synonym for a large database.

Big data in IoT is not just a synonym for a large database. It describes a workload whose size, arrival rate, format mix, quality risk, and decision value shape the architecture. A temperature history table, a vibration stream, a fleet of camera summaries, and a connected-vehicle telemetry feed can all be "big" for different reasons.

Where the Flood Comes From

Big data did not start big. Figure 52.1 shows the world it replaced: three information services, each with its own dedicated delivery network and its own physical medium.

+

The lane worth pausing on in Figure 52.1 is the bottom one: the Broadcast network transmits one way only, which is why its receiver card simply shows the programme while the computer and telephone lanes exchange traffic in both directions. The THE TRADE-OFF card underneath states the economics that killed this model — every new service, IoT telemetry included, would have demanded yet another dedicated build.

Convergence answers that trade-off. In Figure 52.2 the same endpoints — now joined by IoT sensors — share a single IP network, and the centre of the diagram shows what that sharing physically means.

+

Look inside the Converged IP network block of Figure 52.2: the interleaved packet strip is the whole story, because congestion in the pipe touches all four services at once. The trade-off flipped rather than vanished — separate networks wasted capacity, the converged network shares fate, and the IoT platform at the lower right now competes with video for the same pipe. Prioritisation therefore appears in the figure as a design duty, not an optional extra.

Once every service travels as digital traffic, the stored record of the world turns digital too. Figure 52.3 puts measured numbers on how quickly that happened.

+

Hold the two end columns of Figure 52.3 against each other: 1986 stored 2.6 EB in total, 99 % of it analog, while 2007 stored digital 280 EB against analog 19 EB — roughly a hundredfold expansion in which the digital share crossed 50 % only in 2002. Sensor fleets extend exactly this curve, which is why the rest of this chapter turns the vocabulary of scale into an engineering contract instead of a slogan.

The common shorthand is the 5 Vs: volume, velocity, variety, veracity, and value. They become useful when each V turns into a design requirement. Volume asks how many bytes and records must be retained. Velocity asks how quickly events arrive and how soon they must affect a decision. Variety asks how many schemas, units, media types, and device models must be combined. Veracity asks how the system handles missing, duplicated, noisy, late, or drifting data. Value asks what decision justifies the cost.

Use the five-V diagram in Figure 52.4 to turn the familiar vocabulary into a review of the system's actual pressures.

+

Read Figure 52.4 around the centre, not as a ranking. Volume becomes bytes, record counts, and retention time. Velocity joins arrival rate to the decision deadline. Variety exposes schemas, units, and media that must agree before analysis. Veracity asks how missing, noisy, late, or drifting readings are handled. Value comes last because a decision or saved cost must justify the pipeline. The lower path turns those questions into ingest, quality, retention, and outcome requirements. A team can now test each pressure against the specific workload stated above instead of claiming that its data is merely “big.”

52.5 Size Pipeline Before Tools

This section: Size Pipeline Before Tools — Start with the event contract: timestamp, device id, schema version, units, quality flags, and payload size.

Start with the event contract: timestamp, device id, schema version, units, quality flags, and payload size. Then size the pipeline from the outside in. MQTT, HTTP, or gateway protocols deliver events to an ingestion service; Apache Kafka or a cloud equivalent buffers and partitions streams; Spark Structured Streaming, Apache Flink, or a managed stream processor computes windows; curated data lands in a time-series database, lakehouse table, or object store such as Parquet files in cloud storage.

The tool names matter only after the workload is sized. A small campus deployment may work with a time-series database and nightly Parquet export. A regional fleet may need Kafka partitions, schema registry, stream processors, and a raw/curated data lake. The design mistake is choosing a large stack because the phrase "big data" sounds impressive, or choosing a single server when the rate and retention math already shows a distributed workload.

Worked example: rate, volume, and retention
fleet size: 10,000 sensors
publish rate: 1 reading per second per sensor
payload after protocol parsing: 80 bytes per reading

events per second:
10,000 sensors * 1 reading/s = 10,000 events/s

ingest bytes per second:
10,000 events/s * 80 bytes = 800,000 bytes/s
800,000 bytes/s = 0.8 MB/s using decimal units

daily raw volume:
0.8 MB/s * 86,400 s/day = 69,120 MB/day = 69.12 GB/day

30-day raw retention:
69.12 GB/day * 30 = 2,073.6 GB, or about 2.07 TB

hourly aggregate retention:
10,000 sensors * 24 hourly records/day * 120 bytes = 28,800,000 bytes/day
28,800,000 bytes/day = 28.8 MB/day

design implication:
Keep raw records for replay and debugging while they are operationally useful.
Keep compact aggregates much longer for dashboards, reports, and trend models.

Practitioner Knowledge Check

52.6 Time, State, and Backpressure

This section: Time, State, and Backpressure — At scale, IoT big-data errors often come from timing and state rather than from the storage engine itself.

At scale, IoT big-data errors often come from timing and state rather than from the storage engine itself. Devices report event time, gateways add arrival time, brokers preserve ordering only within a partition, and stream processors must decide when a window is complete. Late events, clock skew, duplicate retries, and out-of-order batches can make a dashboard or alert incorrect unless the pipeline treats time as part of the data contract.

Watermarks tell a stream processor how long to wait for late data before closing a window. Idempotent writes keep retries from double-counting the same reading. Schema evolution rules prevent a new firmware version from silently breaking downstream jobs. Backpressure tells upstream components that the sink is slower than the source so the system can buffer, scale, sample, or shed noncritical work instead of failing unpredictably.

Worked example: velocity and backpressure
gateway ingest rate: 12,000 events/s
current sink write rate: 9,000 events/s

backlog growth:
12,000 - 9,000 = 3,000 events/s

after 10 minutes:
3,000 events/s * 600 s = 1,800,000 queued events

if each event averages 120 bytes after enrichment:
1,800,000 * 120 bytes = 216,000,000 bytes, or 216 MB

design implication:
A short sink slowdown can create a large queue. The pipeline needs lag alarms,
autoscaling rules, replay capacity, and a policy for dropping or downsampling
noncritical streams before critical alerts are affected.

Under the Hood Knowledge Check