← Back to the chapter

Technical detail · Time-Series Database Fundamentals

How log-structured writes move work into the background

8.5 Log-Structured Write Mechanics

A normal B-tree index can store timestamps, but high-rate append telemetry stresses the in-place update model. A B-tree keeps keys sorted in pages, so a stream of inserts must update the main index and every secondary index while the write path is still waiting. At 50,000 readings per second, a table with a primary time index plus three secondary indexes asks the engine to maintain four sorted structures for every row.

Log-structured write paths change the timing of that work. The hot path appends to a write-ahead log (WAL), a log used to recover accepted writes, for durability, inserts into an in-memory buffer, then flushes larger immutable files, blocks, segments, or chunks when the buffer fills. Background compaction later merges smaller units, rewrites colder data, and updates summary metadata. That pattern is visible in systems such as InfluxDB’s TSM lineage, RocksDB-backed stores, Cassandra-style storage, and Prometheus’s WAL, head block, immutable block, and compaction flow, even though each product names the pieces differently.

Log-structured time-series write path: append to a write-ahead log, batch readings in an in-memory buffer, flush immutable files or blocks, and later compact smaller units while updating summary metadata.
Log-structured write paths make ingestion cheap first, then pay merge and compression cost later through controlled compaction.

The tradeoff is read amplification. A query for the last hour of one device may need to check several immutable units because recent data can live in multiple levels or blocks. A good time-series design keeps time ranges, tag or label summaries, min/max values, row counts, and bloom-like metadata beside those units so a range query can skip files whose interval or dimensions cannot match.

← Back to the chapter