Analytics & ML · Study deck
Edge Data: Semantic Compression and Pipelines
Reducing every sample by the same rule still sends data the application may not need.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- design semantic event extraction
- select compression from evidence needs
- validate a multi-sensor edge pipeline
- Explain: Event-like data can often become sparse semantic records; smooth numeric streams may support delta or window summaries; spectral decisions may justify selected frequency features; audit-grade reconstruction keeps the route lossless.
Major section
Algorithm Selection Decision Tree
Event-like data can often become sparse semantic records; smooth numeric streams may support delta or window summaries; spectral decisions may justify selected frequency features; audit-grade reconstruction keeps the route lossless.
- The final resource branch checks whether memory, latency, and energy fit the edge device.
Major section
Over-Aggressive Lossy Compression
The Mistake: Applying high compression ratios uniformly across all sensor data without understanding which information is critical for downstream analytics, permanently destroying signals needed for root cause analysis.
- Compression algorithms are chosen based on benchmark performance rather than domain-specific information preservation.
- The "we can always collect more data later" assumption fails for non-reproducible events.
- For predictive maintenance, preserve frequency-domain information (use FFT compression, not just statistics).
Major section
Compression Compute Cost
The 1000x difference in computational efficiency between an ESP32 and a laptop is underestimated.
- Energy cost of computation versus transmission varies by network type (Wi-Fi is cheap to transmit, LoRa is expensive).
- For LoRaWAN devices where transmission costs 100+ mJ per packet, aggressive compression (even expensive algorithms) saves energy.
- For Wi-Fi devices where transmission costs 1-5 mJ per packet, simple aggregation beats complex compression.
Major section
Choose Edge Compression
The ratios in the comparison table are illustrative workload ranges, not guarantees for a new sensor stream.
- If it does, keep the transformation lossless and prove that decompression reproduces the source bytes.
- YES → Use event logging (transmit only state changes).
- YES → Use lossless compression only (GZIP, DEFLATE).
- YES (vibration, acoustics) → Use FFT + top-N bins.
Major section
Choose Edge Compression (continued)
If a lossy representation is acceptable, inspect the signal: stable frequency components can support FFT features, slow trends can support window statistics, and bounded low-variance values can support delta encoding.
- YES → Use statistical aggregation over time windows.
- YES → Use delta encoding + lossless compression.
- NO → Use adaptive sampling rate based on rate-of-change.
Major section
Time-Series Compression Audit
The Keep raw windows around anomalies note makes the bargain explicit: compact events travel, while selected waveform evidence remains available for audit.
- The closing Reduction changes the claim panel distinguishes losing hoof-strike peaks from losing the waveform itself.
- Its retention panel separates a short replay buffer from ninety-day trends and an explicitly authorized long hold.
Major section
Concept Relationships
Under-sampling causes phantom patterns (150 Hz bearing fault aliased to 30 Hz when sampled at 60 Hz).
- Lossless (GZIP): 2-4x reduction, preserves all data (audit trails, compliance).
- Lossy statistical (aggregation): 10-100x reduction, preserves trends (environmental monitoring).
- FFT-based: 50-500x reduction, preserves frequency spectrum (vibration analysis).
- Semantic (event extraction): 100-1000x reduction, preserves state changes (threshold monitoring).
Major section
Concept Relationships (continued)
Edge Data Acquisition: Architecture: Device category determines compression need (cameras need heavy compression; temperature sensors need aggregation).
- Edge Compute Patterns: Edge ML requires compressed features (FFT bins, not raw waveforms).
- data Quality and Preprocessing: Validation must occur before compression (compressing invalid data wastes resources).
- Key Insight:: Compression algorithm selection depends on analytics requirements, not just compression ratio.
Deck summary
Key takeaways
Event-like data can often become sparse semantic records; smooth numeric streams may support delta or window summaries; spectral decisions may justify selected frequency features; audit-grade reconstruction keeps the route lossless.
- The Mistake: Applying high compression ratios uniformly across all sensor data without understanding which information is critical for downstream analytics, permanently destroying signals needed for root cause analysis.
- The 1000x difference in computational efficiency between an ESP32 and a laptop is underestimated.
- The ratios in the comparison table are illustrative workload ranges, not guarantees for a new sensor stream.
- Edge Data Acquisition: Architecture: Device category determines compression need (cameras need heavy compression; temperature sensors need aggregation).
Retrieval practice
Recall check 1 of 3
Data Dora says: answer from memory, then check your reasoning.
Q1A motor operates at 3600 RPM (60 Hz). Bearing defects typically manifest as harmonics at 2x, 3x, and 4x the rotation frequency. What is the minimum sampling rate to capture all relevant fault signatures without aliasing?
Show answer
Answer: B The highest frequency of interest is 4x the 60 Hz fundamental = 240 Hz.
Retrieval practice
Recall check 2 of 3
Data Dora says: answer from memory, then check your reasoning.
Q2An edge device has 64KB of RAM. Which compression algorithm is MOST suitable for continuous vibration monitoring at 1 kHz sampling?
Show answer
Answer: C see answers page
Q3Place each sampling and compression responsibility where it lives so you can shrink payloads without aliasing events or losing the evidence needed to reconstruct them.
Show answer
Answer: A Place each sampling and compression responsibility where it lives so you can shrink payloads without aliasing events or losing the evidence needed to reconstruct them.
Retrieval practice
Recall check 3 of 3
Data Dora says: answer from memory, then check your reasoning.
Q4Complete the edge computing data aggregation function:
Show answer
Answer: A Edge aggregation uses a fixed-size deque buffer to maintain a sliding window of readings.
Print reference
Answers 1 of 2
Answer key.
- B · The highest frequency of interest is 4x the 60 Hz fundamental = 240 Hz.
- C · - GZIP requires 32-64KB just for compression state, leaving little for application - 4096-point FFT needs ~64KB for the complex buffer alone - won't fit - 1024-point FFT needs ~16KB, leaving room for application code and buffers - Semantic compression loses frequency information needed for vibration analysis For vibration monitoring specifically, FFT is essential to detect bearing harmonics.
Print reference
Answers 2 of 2
Answer key.
- A · Place each sampling and compression responsibility where it lives so you can shrink payloads without aliasing events or losing the evidence needed to reconstruct them.
- A · Edge aggregation uses a fixed-size deque buffer to maintain a sliding window of readings.