← Back to the chapter · Next detail →

Technical detail · Data Encoding and Codecs

Exact compression for numeric time series

Time-series storage often gets large savings without losing exact raw values. The trick is exploiting regularity column by column. Timestamps are near-periodic, so delta-of-delta encoding stores the change in the interval rather than every full timestamp. A steady 10-second cadence becomes a long run of zeros after the first values. Numeric values can use Gorilla-style XOR compression: each floating-point value is compared with its predecessor, and because adjacent sensor readings often differ only slightly, the XOR has long runs of leading and trailing zero bits.

This only works well when the storage layout groups similar values together. A column of timestamps, a column of temperatures, and a column of quality flags each give the encoder a predictable pattern. A row that mixes timestamps, JSON blobs, identifiers, units, and arbitrary strings gives the codec much less to exploit.

For a 10-second sensor, timestamp deltas are usually 10, 10, 10, 10. Delta-of-delta stores the change between those deltas: 0, 0, 0 after the first value. If one packet arrives late and the interval becomes 12 seconds, the encoder records a small exception and then returns to zeros once the interval is stable again. That is different from storing every full timestamp, and it is still lossless.

Separate three decisions in the storage plan:

← Back to the chapter · Next detail →