← Back to the chapter · Previous detail · Next detail →

Technical detail · Big Data Case Studies

From annual volume to ingest and backhaul

Using decimal storage units,

1 ZB=1021 bytes=109 TB=1012 GB.1\ \text{ZB}=10^{21}\ \text{bytes}=10^9\ \text{TB}=10^{12}\ \text{GB}.

The ten-times scenario is therefore

Vyear=10(1.2 ZB)=12 ZB=1.2×1010 TB=12,000,000,000,000 GB per year.V_{year}=10(1.2\ \text{ZB})=12\ \text{ZB} =1.2\times10^{10}\ \text{TB} =12{,}000{,}000{,}000{,}000\ \text{GB per year}.

Annual capacity hides the ingest problem, so divide by the seconds in a 365-day year:

Rraw=12×1021 bytes365×24×3600 s≈3.81×1014 bytes/s.R_{raw}=\frac{12\times10^{21}\ \text{bytes}} {365\times24\times3600\ \text{s}} \approx3.81\times10^{14}\ \text{bytes/s}.

That is about 381 TB/s, or 3.05 Pbit/s, sustained. One day at that rate produces about 32.9 EB. Even ignoring replication, parity, metadata, failed media, cooling, and write bandwidth, storing the raw year would require the usable capacity of twelve billion 1 TB drives. The absurdity is the lesson: “save everything and decide later” is not an architecture at this scale.

Now test the transport boundary. A dedicated 400 Gbit/s link carries at most

V400G=400×1098(365×24×3600)≈1.58×1018 bytes/year=1.58 EB/yearV_{400G}=\frac{400\times10^9}{8}(365\times24\times3600) \approx1.58\times10^{18}\ \text{bytes/year}=1.58\ \text{EB/year}

before framing, protection, outages, or shared use. The raw 12 ZB/year stream is roughly 7,600 times larger. Equivalently, a 400 Gbit/s Perth backhaul could carry only about 0.013% of the hypothetical raw stream. Adding one faster link does not close that gap.

BoundaryRaw scenarioDesign consequence
Annual volume12 ZB/yearOn-site storage cannot be a raw, indefinite archive
Average ingest381 TB/sMemory, buses, accelerators, and writers must be parallel and continuously provisioned
400 Gbit/s backhaul0.05 TB/s ideal payload rateRaw off-site transfer needs about 7,600:1 reduction before overhead
One day of raw data32.9 EBA short outage still creates an enormous local buffer obligation

← Back to the chapter · Previous detail · Next detail →