← Back to the chapter · Previous detail
Technical detail · Big Data Case Studies
Account for every reduction stage
In the historical SKA scenario, a hypothetical 12 ZB/year raw stream needs roughly 7,600:1 reduction to fit a 400 Gbit/s backhaul. The system must reduce data near the instruments. A defensible pipeline calibrates and flags samples, channelizes or correlates signals, rejects known interference under recorded rules, forms scientifically useful products, and retains only the raw windows needed for calibration, anomaly review, or reproducibility. Each reduction stage needs a ratio and a loss statement:
where is the retained fraction after stage . To fit the illustrative 400 Gbit/s link, the product of all retained fractions must be no more than about . That target should be allocated across reversible formatting, aggregation, correlation, event selection, and retention rather than hidden behind the word “compression.” The case record must say which scientific questions each irreversible step makes impossible, which raw windows remain available for audit, and how a backhaul outage changes the on-site queue.