Analytics & ML · Study deck

Edge Data: Semantic Compression and Pipelines

A single rule for cutting samples may still send data the app does not need.

Data Dora is your guide for this deck.

edgesamplingcompression

Data Dora, the module guide, in a scene from this chapter.

After studying this chapter

Learning objectives

You will be able to:

design semantic event extraction
select compression from evidence needs
validate a multi-sensor edge pipeline
Explain: Event-like data can often become sparse semantic records; smooth numeric streams may support delta or window summaries; spectral decisions may justify selected frequency features; audit-grade reconstruction keeps the route lossless.

Major section

Algorithm Selection Decision Tree

Event-like data can often become sparse semantic records; smooth numeric streams may support delta or window summaries; spectral decisions may justify selected frequency features; audit-grade reconstruction keeps the route lossless.

The final resource branch checks whether memory, latency, and energy fit the edge device.

Edge Data Compression Algorithm Selection Decision Tree

Major section

Over-Aggressive Lossy Compression

The Mistake: Applying high compression ratios uniformly across all sensor data without understanding which information is critical for downstream analytics, permanently destroying signals needed for root cause analysis.

Compression algorithms are chosen based on benchmark performance rather than domain-specific information preservation.
The "we can always collect more data later" assumption fails for non-reproducible events.
For predictive maintenance, preserve frequency-domain information (use FFT compression, not just statistics).

Major section

Compression Compute Cost

The difference in computational efficiency between an ESP32 and a laptop can be substantial for compression workloads.

Energy cost of computation versus transmission varies by network type (Wi-Fi is cheap to transmit, LoRa is expensive).
For an illustrative LoRaWAN device where a transmission costs 100+ mJ per packet, aggressive compression can save energy if its compute cost is lower than the avoided radio energy.
For an illustrative Wi-Fi device where transmission costs 1-5 mJ per packet, simple aggregation can beat complex compression if measured CPU energy exceeds the avoided radio energy.

Major section

Try It: Edge FFT Pipeline Data Reduction

Validate those features on representative traces; actual bearing-defect frequencies require bearing geometry.

These checks describe this workload and do not establish that selected FFT bins retain every fault feature.
The factory example turns the algorithm comparison into an operating choice: keep the frequency evidence, discard the raw waveform volume, and check latency against the maintenance requirement.
Adjust the vibration sensor parameters and FFT settings to see how edge FFT compression reduces bandwidth for factory monitoring.

A six-axis IMU produces 103.68 MB per day before overhead, while each reduction choice removes a different kind of motion evidence.

Major section

Illustrative Shaft-Order Sampling Exercise

These examples are possible indicators, not universal bearing-defect frequencies; actual BPFO/BPFI require bearing geometry.

Step 2: Calculate practical sampling rate with safety margin: For this example, use 2.5x to 3x the Nyquist minimum as an anti-aliasing filter margin.
Out-of-band input still requires adequate anti-aliasing filtering.
Sending selected FFT bins greatly reduces payload while retaining those selected spectral features.

Try it: Illustrative Shaft-Order Sampling Exercise in the chapter

Major section

Choose Edge Compression

The ratios in the comparison table are illustrative workload ranges, not guarantees for a new sensor stream.

If it does, keep the transformation lossless and prove that decompression reproduces the source bytes.
YES → Use event logging (transmit only state changes).
YES → Use lossless compression only (GZIP, DEFLATE).
YES (vibration, acoustics) → Use FFT + top-N bins.

Side-by-side motion reduction choices comparing raw and five-hertz waveform data with packed tilt and change-only state reports.

Major section

Choose Edge Compression (continued)

If a lossy representation is acceptable, inspect the signal: stable frequency components can support FFT features, slow trends can support window statistics, and bounded low-variance values can support delta encoding.

YES → Use statistical aggregation over time windows.
YES → Use delta encoding + lossless compression.
NO → Use adaptive sampling rate based on rate-of-change.

Estate-scale telemetry calculation from one sensor through one hundred five-sensor fog nodes, with downsampling and tier-specific retention windows.

Major section

Time-Series Compression Audit

The Keep raw windows around anomalies note makes the bargain explicit: compact events travel, while selected waveform evidence remains available for audit.

Six axes at 2 bytes per axis make a 12-byte frame.
At 100 frames per second, that is 12 B/frame x 100 frames/s = 1,200 B/s.

Edge processing path from a six-axis cow-collar stream through fog event extraction to herd-level cloud comparison, with raw anomaly windows retained.

Major section

Time-Series Compression Audit (continued)

Multiplying by 60 s/min gives 72,000 B/min; extending through 60 min/h and 24 h/day gives 103,680,000 B/day, or 103.68 MB/day before timestamps and identifiers.

Downsampling may miss short hoof-strike peaks, while event extraction keeps counts but discards waveform shape.
Those losses support different claims, so byte savings alone cannot choose the reducer.
Its retention panel separates a short replay buffer from ninety-day trends and an explicitly authorized long hold.
The shared Same ID and clock premise prevents a misleading comparison between differently described records.

Major section

Concept Relationships

Under-sampling can cause phantom patterns (an illustrative 150 Hz shaft-order component aliases when sampled below its Nyquist minimum).

Lossless (GZIP): 2-4x reduction, preserves all data (audit trails, compliance).
Lossy statistical (aggregation): 10-100x reduction, preserves trends (environmental monitoring).
In the illustrative workload comparison, FFT-based: 50-500x reduction, preserves selected frequency components (vibration analysis).

Major section

Concept Relationships (continued)

Semantic (event extraction): 100-1000x reduction, preserves state changes (threshold monitoring).

Edge Data Acquisition: Power and Gateways: Aggregation reduces transmission frequency and can extend battery life (illustrative factory case: 14,400x record-count reduction makes LoRa feasible versus the assumed cellular schedule).
Edge Data Acquisition: Architecture: Device category determines compression need (cameras need heavy compression; temperature sensors need aggregation).
Edge Compute Patterns: Edge ML requires compressed features (FFT bins, not raw waveforms).

Major section

Concept Relationships (continued)

Vibration monitoring can use FFT features selected from representative traces; validate their fault-detection performance because discarded bins and time detail cannot be recovered.

data Quality and Preprocessing: Validation must occur before compression (compressing invalid data wastes resources).
Key Insight: Compression algorithm selection depends on analytics requirements, not just compression ratio.
Choosing a compression method without checking the required evidence can permanently discard signal details needed for analysis.

Deck summary

Key takeaways

Event-like data can often become sparse semantic records; smooth numeric streams may support delta or window summaries; spectral decisions may justify selected frequency features; audit-grade reconstruction keeps the route lossless.

The Mistake: Applying high compression ratios uniformly across all sensor data without understanding which information is critical for downstream analytics, permanently destroying signals needed for root cause analysis.
The difference in computational efficiency between an ESP32 and a laptop can be substantial for compression workloads.
Validate those features on representative traces; actual bearing-defect frequencies require bearing geometry.
These examples are possible indicators, not universal bearing-defect frequencies; actual BPFO/BPFI require bearing geometry.

Retrieval practice

Recall check 1 of 3

Data Dora says: answer from memory, then check your reasoning.

Q1A vibration gateway uploads only selected FFT bins and a later fault review needs harmonics that were discarded. What should the team conclude?

AThe omitted spectrum and time detail cannot be reconstructed; validate the bin selection against representative traces before relying on it.

BThe shaft fundamental is enough to reconstruct every missing harmonic.

CIncreasing cloud storage can recreate bins the gateway never transmitted.

DTreat the quiet selected-bin stream as proof that no bearing fault occurred.

Show answer

Answer: A Selected-bin upload is lossy: representative traces must validate which harmonics and time details can be discarded.

Retrieval practice

Recall check 2 of 3

Data Dora says: answer from memory, then check your reasoning.

Q2An edge device has 64 KiB of RAM and must retain frequency-domain shaft-order features from a 1 kHz vibration stream. Its FFT uses an input and a scratch array of single-precision complex samples; the available semantic extractor reports only threshold events. Which edge reduction approach fits best?

AGZIP Level 9 (best compression ratio)

BFFT with 4096-point window (best frequency resolution)

CSemantic event extraction (lowest memory)

D1024-point FFT window (balanced memory/resolution)

Show answer

Answer: D see answers page

Retrieval practice

Recall check 3 of 3

Data Dora says: answer from memory, then check your reasoning.

Q3What carries the compressed sample stream together with timing, scale, codec version and loss bound?

ASignal Bandwidth and Error Budget

BAnti-Alias Filter

CADC Sample Stream

DCompressed Payload and Metadata

Show answer

Answer: D Compression removes stated redundancy while retaining timing, scale, codec version, and an explicit loss bound.

Q4Complete the edge computing data aggregation function:

Aself.buffer = deque(maxlen=window_size)

Bself.buffer = list()

Cself.buffer = np.zeros(window_size)

Dself.buffer = deque()

Show answer

Answer: A Edge aggregation uses a fixed-size deque buffer to maintain a sliding window of readings.

Print reference

Answers 1 of 2

Answer key.

A · Selected-bin upload is lossy: representative traces must validate which harmonics and time details can be discarded.
D · With two single-precision complex arrays, a 1024-point FFT needs 16 KiB and gives about 0.977 Hz bin spacing at 1 kHz. A 4096-point FFT needs the full 64 KiB before application memory. GZIP and the available threshold-event extractor do not produce the required frequency features.
D · Compression removes stated redundancy while retaining timing, scale, codec version, and an explicit loss bound.

Print reference

Answers 2 of 2

Answer key.

A · Edge aggregation uses a fixed-size deque buffer to maintain a sliding window of readings.