← Back to the chapter · Next detail →
Technical detail · Data Encoding and Codecs
Exact compression for numeric time series
Time-series storage often gets large savings without losing exact raw values. The trick is exploiting regularity column by column. Timestamps are near-periodic, so delta-of-delta encoding stores the change in the interval rather than every full timestamp. A steady 10-second cadence becomes a long run of zeros after the first values. Numeric values can use Gorilla-style XOR compression: each floating-point value is compared with its predecessor, and because adjacent sensor readings often differ only slightly, the XOR has long runs of leading and trailing zero bits.
This only works well when the storage layout groups similar values together. A column of timestamps, a column of temperatures, and a column of quality flags each give the encoder a predictable pattern. A row that mixes timestamps, JSON blobs, identifiers, units, and arbitrary strings gives the codec much less to exploit.
For a 10-second sensor, timestamp deltas are usually 10, 10, 10, 10. Delta-of-delta stores the change between those deltas: 0, 0, 0 after the first value. If one packet arrives late and the interval becomes 12 seconds, the encoder records a small exception and then returns to zeros once the interval is stable again. That is different from storing every full timestamp, and it is still lossless.
Separate three decisions in the storage plan:
- use lossless column compression for the raw audit window so incidents can be replayed exactly;
- use rollups or downsampling only after the team defines which questions no longer need raw samples;
- keep schema and unit metadata beside compressed values so a dashboard can distinguish a physical change from an encoding, unit, or calibration change.