← Back to the chapter · Previous detail · Next detail →
Technical detail · Big Data Case Studies
From annual volume to ingest and backhaul
Using decimal storage units,
The ten-times scenario is therefore
Annual capacity hides the ingest problem, so divide by the seconds in a 365-day year:
That is about 381 TB/s, or 3.05 Pbit/s, sustained. One day at that rate produces about 32.9 EB. Even ignoring replication, parity, metadata, failed media, cooling, and write bandwidth, storing the raw year would require the usable capacity of twelve billion 1 TB drives. The absurdity is the lesson: “save everything and decide later” is not an architecture at this scale.
Now test the transport boundary. A dedicated 400 Gbit/s link carries at most
before framing, protection, outages, or shared use. The raw 12 ZB/year stream is roughly 7,600 times larger. Equivalently, a 400 Gbit/s Perth backhaul could carry only about 0.013% of the hypothetical raw stream. Adding one faster link does not close that gap.
| Boundary | Raw scenario | Design consequence |
|---|---|---|
| Annual volume | 12 ZB/year | On-site storage cannot be a raw, indefinite archive |
| Average ingest | 381 TB/s | Memory, buses, accelerators, and writers must be parallel and continuously provisioned |
| 400 Gbit/s backhaul | 0.05 TB/s ideal payload rate | Raw off-site transfer needs about 7,600:1 reduction before overhead |
| One day of raw data | 32.9 EB | A short outage still creates an enormous local buffer obligation |