Analytics & ML · Study deck
Time-Series Compression Audit Limits
Picture a rail depot storing wheel vibration every second.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain why regular timestamps and slowly changing values make sensor streams highly compressible.
- Separate delta, delta-of-delta, and Gorilla-style XOR encoding roles in time-series blocks.
- Decide when lossless compression is required and when lossy reductions are acceptable.
- Record compression metadata so cloud analytics can judge audit-grade, trend-grade, or exploratory evidence.
Major section
Start With the Story
Recent raw detail helps explain a new fault, while year-long trends can use smaller records.
- The data owner must decide which facts each storage tier must preserve and which later questions a reduction will no longer answer.
- Encode, decode where possible, and compare counts, times, extremes, events, and summary results.
- A distant store may hold compact history, but every tier must label its evidence honestly.
- This opening does not pick one compression method for all streams.
Major section
Why Compression Follows Sampling
Edge Sampling and Compression teaches the main sampling, aggregation, FFT, semantic extraction, and edge algorithm-selection workflow.
- This page narrows in on the deeper storage boundary: how a stream can shrink without changing facts, and how to prove which facts were preserved.
- The labelled compression ratio is a storage outcome, not proof of fidelity.
Major section
Compress Similar Samples
If the value moves from 3.01 C to 3.02 C to 3.02 C, value deltas or XORs are also small.
- A sensor emits a relentless stream of timestamped values, and in most streams each value is very close to the one before it, arriving at a very regular interval.
- These power production time-series databases and cut the amount of data an edge node must ship.
- Worked example: a refrigerated-truck sensor reports temperature every second.
Major section
Compress Similar Samples (continued)
Two ideas carry most of the gain: encode differences rather than absolute values (they are small when the signal changes slowly), and exploit the bit-level similarity of consecutive floating-point numbers.
- The changes are tiny, so they need far fewer bits than the full numbers.
- If the timestamp interval is almost always one second, delta-of-delta mostly records zero.
- The boundary is fidelity: compression is acceptable only when the receiver can still answer the same question.
Major section
Practitioner: Delta Encoding and Gorilla
Splitting the work by column is deliberate: timestamps compress best with delta-of-delta (they are almost perfectly regular), while values compress best with XOR (they are numerically similar but not linearly spaced).
- Worked example continuation: if a gateway receives 10,000 points from one sensor, a row layout repeats timestamp and value metadata for each record.
- A columnar block instead stores a timestamp column, a value column, and a small header that says which compression method each column used.
- On readback, the decompressor expands timestamp intervals, applies value XOR or deltas in order, and reconstructs the original point sequence.
Major section
Lossless, Lossy, and Edge Payoff
Lossless methods must return the exact original sequence, so they are appropriate for billing, compliance, incident review, and any record where a later investigator may ask what the sensor actually reported.
- Delta, delta-of-delta, run-length, and Gorilla-style XOR encoders are in this family.
- They exploit regularity, but they do not change the facts.
- Choosing "best ratio" without naming the decision can erase the signal that matters.
Major section
Lossless, Lossy, and Edge Payoff (continued)
The edge payoff is mostly radio time.
- Window aggregation, downsampling, piecewise-linear approximation, FFT peak extraction, and event extraction can reduce a stream far more than lossless compression, but the discarded detail is gone.
- Sending that raw stream continuously is unrealistic for a battery node.
- Lossless vs lossy Delta and Gorilla are lossless -- exact reconstruction.
Major section
Lossless, Lossy, and Edge Payoff (continued)
A trend dashboard may need hourly min, max, and mean; a vibration diagnosis may need spectral peaks; an alarm stream may need threshold crossings and durations.
- If an FFT window keeps ten frequency bins with frequency, magnitude, and phase, the payload may be roughly 120 to 160 bytes plus metadata.
- The catch is that the waveform cannot be reconstructed exactly from those peaks, so the raw window must be kept locally or sampled on demand when full evidence is required.
- Those fields let cloud analytics decide whether a block can support an audit-grade answer, a trend answer, or only an exploratory visualization.
Major section
Lossless, Lossy, and Edge Payoff (continued)
For more compression, lossy methods (downsampling, piecewise-linear approximation like the swinging-door algorithm) trade a bounded error for smaller size -- fine for trends, not for audit-grade records.
- Compression saves radio energy On the edge, transmission dominates the energy budget, so sending 1.4 bytes instead of 16 is a direct battery win as well as a storage win.
- The CPU-vs-transmit trade Compression costs cycles on a small MCU.
- On the edge the payoff is double -- less flash used and, more importantly, far less energy spent shipping data over the radio.
Deck summary
Key takeaways
Recent raw detail helps explain a new fault, while year-long trends can use smaller records.
- Edge Sampling and Compression teaches the main sampling, aggregation, FFT, semantic extraction, and edge algorithm-selection workflow.
- If the value moves from 3.01 C to 3.02 C to 3.02 C, value deltas or XORs are also small.
- Two ideas carry most of the gain: encode differences rather than absolute values (they are small when the signal changes slowly), and exploit the bit-level similarity of consecutive floating-point numbers.
- The edge payoff is mostly radio time.
Retrieval practice
Recall check 1 of 2
Data Dora says: answer from memory, then check your reasoning.
Q1What redundancy does time-series compression primarily exploit?
Show answer
Answer: A Small changes and regular intervals make delta-based encoding highly effective.
Q2Why is 'delta-of-delta' encoding so effective for time-series timestamps?
Show answer
Answer: A Regular sampling makes the second difference nearly constant (zero), which is extremely compressible.
Retrieval practice
Recall check 2 of 2
Data Dora says: answer from memory, then check your reasoning.
Q3For an audit-grade sensor log that must reconstruct every value exactly, which compression choice is appropriate?
Show answer
Answer: A Audit-grade records require lossless compression for exact reconstruction.
Print reference
Answers
Answer key.
- A · Small changes and regular intervals make delta-based encoding highly effective.
- A · Regular sampling makes the second difference nearly constant (zero), which is extremely compressible.
- A · Audit-grade records require lossless compression for exact reconstruction.