Analytics & ML · Study deck
Edge Processing for Big Data
Picture a rail depot with cameras checking wheel damage.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: The key question is not "edge or cloud?" It is "which bytes must cross the network for this decision, and which bytes are only needed locally for validation, replay, or short-term debugging?".
- Explain: The edge decides which raw samples stay local, which features or aggregates move upstream, which anomaly evidence needs immediate cloud attention, and which raw window must be retained locally for replay.
- Explain: If the edge result is still too large, change the window length, feature set, anomaly policy, compression, or local retention before scaling the cloud path.
Major section
Start With the Story
A local computer must warn the crew within seconds, while engineers need enough saved evidence to review misses and improve the check.
- Sending every image forever is costly and may expose people or places without need.
- Less data is not automatically better.
- A summary can hide rare faults, bias, or the reason for an alarm.
Major section
Edge Chooses What Leaves Site
Edge processing is not just "doing analytics near the device." It is a data-reduction and decision contract.
- The edge decides which raw samples stay local, which features or aggregates move upstream, which anomaly evidence needs immediate cloud attention, and which raw window must be retained locally for replay.
Major section
Edge Chooses What Leaves Site (continued)
This matters when raw sensor, camera, audio, or vibration streams are too large, too private, too expensive, or too latency-sensitive to ship continuously.
- An edge design should name the local computation, the forwarded evidence, the retained raw window, and the cloud handoff.
- The cloud still matters, but it receives decisions and evidence rather than every raw byte.
- Aggregate Send min, max, count, mean, RMS, percentiles, or window features instead of every sample.
Major section
Edge Chooses What Leaves Site (continued)
Aggregate value, count, window bounds, quality flags, and anomaly id.
- The key question is not "edge or cloud?" It is "which bytes must cross the network for this decision, and which bytes are only needed locally for validation, replay, or short-term debugging?".
- Filtering removes records under an explicit rule, aggregation changes granularity, and compression changes representation while intending to preserve the agreed information.
- Encrypted short-term raw evidence with access controls.
Major section
Edge Chooses What Leaves Site (continued)
The sequence carries forward the chapter's narrative: edge reduction is valuable only when the cloud can still reconstruct, audit, or act on everything the data contract promises.
- Infer Run local classifiers, anomaly detectors, or rule engines when latency or privacy requires local action.
- Buffer Keep a bounded raw ring buffer so anomalies can include pre-event and post-event evidence.
- Counts, embeddings, labels, redacted records, or policy-approved extracts.
Major section
Calculate Data Reduction First
Edge processing should be justified with a workload calculation.
- If the edge result is still too large, change the window length, feature set, anomaly policy, compression, or local retention before scaling the cloud path.
- Different data types need different reductions.
- Temperature sensors may use change-threshold filtering and hourly aggregates.
Major section
Versioned Evidence for Edge
Edge processing changes the evidence that reaches the cloud, so correctness depends on versioning and synchronization.
- A feature value is meaningful only with its window bounds, units, sensor calibration, firmware version, model version, and quality state.
- Edge systems also have time and delivery risks.
- A gateway may batch data during an outage, then reconnect and backfill older summaries.
Major section
Versioned Evidence for Edge (continued)
Cloud pipelines must separate event time from arrival time so late backfill does not look like current behavior.
- If the edge performs local control, the cloud needs sequence numbers, acknowledgements, and conflict rules so replayed or delayed messages do not trigger incorrect actions.
- Evidence to Forward.
- Edge scores become less reliable after equipment, environment, or sensor changes.
Major section
Versioned Evidence for Edge (continued)
design implication: If a machine-protection action must occur inside 50 ms, the cloud path is too slow for the control decision.
- The edge can act locally, then send the decision, features, model version, and raw-window pointer upstream for audit and learning.
- Model Version Every edge score or class label should include model id, threshold, feature definition, and input window.
- Fallback Define what the gateway does when the model, network, clock, or cloud acknowledgement is unavailable.
Deck summary
Key takeaways
A local computer must warn the crew within seconds, while engineers need enough saved evidence to review misses and improve the check.
- Edge processing is not just "doing analytics near the device." It is a data-reduction and decision contract.
- This matters when raw sensor, camera, audio, or vibration streams are too large, too private, too expensive, or too latency-sensitive to ship continuously.
- Aggregate value, count, window bounds, quality flags, and anomaly id.
- The sequence carries forward the chapter's narrative: edge reduction is valuable only when the cloud can still reconstruct, audit, or act on everything the data contract promises.
Retrieval practice
Recall check 1 of 3
Data Dora says: answer from memory, then check your reasoning.
Q1What should an edge-processing contract define before the team chooses hardware or a cloud service?
Show answer
Answer: B Edge processing is a contract about local decisions, forwarded evidence, raw retention, and cloud handoff.
Retrieval practice
Recall check 2 of 3
Data Dora says: answer from memory, then check your reasoning.
Q2A fleet produces 1.728 TB/day of raw vibration data. Edge features reduce the forwarded stream to 6.912 GB/day. What is the approximate reduction factor?
Show answer
Answer: A Reduction factor is raw daily volume divided by forwarded daily feature volume.
Retrieval practice
Recall check 3 of 3
Data Dora says: answer from memory, then check your reasoning.
Q3A machine-protection decision must happen within 50 ms. The edge path takes 25 ms and the cloud round trip takes 235 ms. What is the correct architecture choice?
Show answer
Answer: A Latency-critical decisions should run where the deadline can be met, while cloud systems receive evidence and audit context.
Print reference
Answers
Answer key.
- B · Edge processing is a contract about local decisions, forwarded evidence, raw retention, and cloud handoff.
- A · Reduction factor is raw daily volume divided by forwarded daily feature volume.
- A · Latency-critical decisions should run where the deadline can be met, while cloud systems receive evidence and audit context.