Managing TSDB Storage

1. Understanding TSDB Directory Structure

Directory	Description	Contents
chunks	Stores the actual time series data in compressed format	`chunks` files with `.tsdb` extension
index	Index files for fast querying and metadata lookup	`index` files with `.tbi` extension
wal	Write-ahead log to ensure data durability during writes	`wal` files with `.wal` extension
meta.json	Metadata about the storage state and configuration	JSON file containing storage metadata
lock	Lock file to prevent concurrent access to the storage	`lock` file to ensure exclusive access

Example: TSDB Directory Layout

prometheus/
├── chunks/
│   ├── 000001
│   └── 000002
├── index/
│   ├── 000001
│   └── 000002
├── wal/
│   ├── 000001.wal
│   └── 000002.wal
├── meta.json
└── lock

2. Configuring Data Retention

Option	Values	Description
storage.tsdb.retention.time	`2d`	How long to retain samples in storage. Default is `15d`.
storage.tsdb.retention.size	`50GB`	Maximum size of the data directory. When exceeded, oldest data is deleted.

Example: Configuring Data Retention

storage:
  tsdb:
    retention:
      time: 30d
      size: 100GB

Note: Setting storage.tsdb.retention.size is useful for enforcing disk space limits in environments with strict quotas.

3. Using Write-Ahead Log

Configuration Option	Default Value	Description
storage.tsdb.wal-compression	`true`	Enables compression of the WAL using Snappy.
storage.tsdb.wal-dir	`<data-dir>/wal`	Directory to which WAL files are written.
storage.tsdb.min-block-duration	`2h`	Minimum duration of a block in the TSDB.
storage.tsdb.max-block-duration	`10d`	Maximum duration of a block in the TSDB.

Example: WAL Compression Configuration

storage:
  tsdb:
    wal-compression: true

Note: Enabling WAL compression reduces disk I/O and storage usage at the cost of CPU overhead.

4. Implementing Block Compaction

Configuration Option	Description	Default Value	Notes
--storage.tsdb.block-duration	Duration of compaction blocks	`2h`	Controls how long data is kept in a block before compaction
--storage.tsdb.max-block-size	Maximum size of a block before compaction	`512MB`	Blocks are compacted once they reach this size
--storage.tsdb.retention.time	How long to keep data in the database	`15d`	Data older than this is deleted after compaction
--storage.tsdb.compaction.strategy	Strategy used for compaction	`default`	Options: `default`, `fast`

Example: Compaction Configuration

storage:
  tsdb:
    block-duration: 2h
    max-block-size: 512MB
    retention.time: 15d
    compaction.strategy: default

Note: Block compaction is a background process that merges smaller blocks into larger ones to optimize storage and query performance.

Compaction Process Overview

Blocks are created every --storage.tsdb.block-duration
When blocks reach --storage.tsdb.max-block-size, they are compacted
Compaction reduces storage space and improves query performance
Older blocks are eventually deleted based on --storage.tsdb.retention.time

5. Configuring Chunk Encoding

Option	Values	Description
tsdb.chunk-encoding-version	`2`	Version of chunk encoding to use. Defaults to `2` in Prometheus 2.30+.
tsdb.chunk-compression	`snappy`	Compression algorithm for chunks. Supported values: `none`, `snappy`, `gzip`, `zstd`.
tsdb.chunk-size	`1024`	Size of chunks in bytes. Default is `1024` (1KB) per chunk.

Example: Setting Chunk Compression

tsdb:
  chunk-compression: gzip

Note: Chunk encoding version 2 introduces improvements in compression and query performance over version 1.

6. Setting Up Memory Mapping

Configuration Option	Description	Default Value	Notes
storage.tsdb.max-block-duration	Maximum duration a block can last before being closed and written to disk.	`2h`	Blocks are written to disk when they reach this duration or size limit.
storage.tsdb.min-block-duration	Minimum duration a block must last before being considered for compaction.	`2h`	Blocks smaller than this duration are not compacted.
storage.tsdb.head-chunk-time	Duration of a chunk in the head block.	`2m`	Chunks are filled with samples until this time is reached.
storage.tsdb.head-chunk-size	Maximum size of a chunk in the head block.	`1MB`	Chunks are flushed when they reach this size.

Example: Memory Mapping Configuration

storage:
  tsdb:
    max-block-duration: 3h
    min-block-duration: 1h
    head-chunk-time: 3m
    head-chunk-size: 2MB

Note: Adjusting these values can impact query performance and memory usage. Larger chunks reduce overhead but increase memory consumption.

7. Managing Disk Space

Example: Configuring retention and storage limits

storage:
  tsdb:
    retention.time: 30d
    retention.size: 50GB
    dir: /prometheus/data

Configuration	Values	Description
storage.tsdb.retention.time	`30d`	Maximum time to keep samples in storage (default: 15d)
storage.tsdb.retention.size	`50GB`	Maximum size of storage to keep samples (default: unlimited)
storage.tsdb.dir	`/prometheus/data`	Directory to store the TSDB data files

Note: When both retention.time and retention.size are set, Prometheus will delete samples based on whichever limit is reached first.

Example: Monitoring disk usage via Prometheus metrics

# HELP prometheus_storage_block_duration_seconds Duration of a block in seconds
# TYPE prometheus_storage_block_duration_seconds gauge
prometheus_storage_block_duration_seconds 3600

# HELP prometheus_storage_series_created_total Total number of series created
# TYPE prometheus_storage_series_created_total counter
prometheus_storage_series_created_total 123456

# HELP prometheus_storage_series_removed_total Total number of series removed
# TYPE prometheus_storage_series_removed_total counter
prometheus_storage_series_removed_total 12345

Workflow: Managing TSDB disk space

Monitor current disk usage with prometheus_storage_series_created_total and prometheus_storage_block_duration_seconds
Adjust storage.tsdb.retention.time or storage.tsdb.retention.size in configuration
Restart Prometheus to apply new settings
Verify disk space is freed by checking prometheus_storage_series_removed_total

Key Points

Prometheus retains data for a configurable time or size limit
When limits are reached, older blocks are deleted to free space
Disk usage can be monitored via Prometheus metrics
Configuration changes require a Prometheus restart to take effect

8. Implementing Snapshot API

Example: Creating a Snapshot via HTTP API

curl -X POST http://localhost:9090/api/v1/admin/tsdb/snapshot

Endpoint	Method	Description	Notes
/api/v1/admin/tsdb/snapshot	POST	Creates a snapshot of the TSDB	Snapshot is stored in the data directory under `snaphots/`

Note: The snapshot includes all data up to the point of the request. It is not a real-time backup.

Warning: Snapshots are not automatically deleted. Manual cleanup is required to prevent disk space exhaustion.

Example: Listing Snapshots

curl -X GET http://localhost:9090/api/v1/admin/tsdb/snapshots

Endpoint	Method	Description	Response Format
/api/v1/admin/tsdb/snapshots	GET	Lists all existing snapshots	JSON array of snapshot metadata

Example: Deleting a Snapshot

curl -X POST http://localhost:9090/api/v1/admin/tsdb/snapshot/delete -d 'snapshot_id=12345'

Endpoint	Method	Description	Parameters
/api/v1/admin/tsdb/snapshot/delete	POST	Deletes a specific snapshot by ID	`snapshot_id` (required)

9. Using Clean Tombstones

When Prometheus deletes time series, it marks them as "tombstoned" rather than immediately removing them from disk. This allows for safe handling of data in case of system failures or rollbacks.

Option	Values	Description
--storage.tsdb.tombstone-retention	`2d`	Duration for which tombstones are retained before being cleaned up.

Example: Configuring Tombstone Retention

storage:
  tsdb:
    tombstone-retention: 2d

Note: The tombstone retention period should be long enough to ensure that any potential rollbacks or data recovery operations can still access the deleted series.

Warning: If tombstone retention is too short, data recovery becomes impossible once the tombstones are cleaned up.

10. Monitoring Storage Metrics

Metric Name	Description	Type	Notes
prometheus_tsdb_head_chunks	Number of chunks in the head block	Gauge	Represents the current number of chunks in the head block, which is the most recently written data
prometheus_tsdb_head_series	Number of series in the head block	Gauge	Tracks the number of time series currently in the head block
prometheus_tsdb_head_chunks_created_total	Total number of chunks created in the head block	Counter	Cumulative count of chunks created in the head block since Prometheus started
prometheus_tsdb_head_chunks_removed_total	Total number of chunks removed from the head block	Counter	Cumulative count of chunks removed from the head block due to compaction or retention
prometheus_tsdb_wal_truncations_total	Total number of WAL truncations	Counter	Counts how many times the Write-Ahead Log (WAL) has been truncated
prometheus_tsdb_wal_fsync_duration_seconds	Duration of fsync operations on the WAL	Histogram	Measures how long fsync operations take on the WAL, useful for identifying I/O bottlenecks
prometheus_tsdb_compaction_duration_seconds	Duration of compaction operations	Histogram	Measures how long compaction of chunks into blocks takes
prometheus_tsdb_compaction_chunks_processed_total	Total number of chunks processed during compaction	Counter	Cumulative count of chunks that have been processed during compaction
prometheus_tsdb_compaction_chunks_skipped_total	Total number of chunks skipped during compaction	Counter	Cumulative count of chunks that were skipped during compaction (e.g., due to being too small)
prometheus_tsdb_storage_blocks	Number of storage blocks	Gauge	Represents the number of block directories in the storage directory

Example: Querying Storage Metrics

prometheus_tsdb_head_chunks
prometheus_tsdb_wal_truncations_total
prometheus_tsdb_compaction_duration_seconds

Note: These metrics are essential for monitoring the health and performance of Prometheus' storage engine. Monitoring these can help detect issues like high I/O, memory pressure, or retention policy misconfigurations.

11. Configuring Out-of-Order Samples

Configuration Option	Description	Default Value	Notes
out_of_order_time_window	Time window in seconds during which out-of-order samples are accepted and stored.	`0`	Set to `0` to disable out-of-order sample acceptance. Values greater than 0 enable storage of samples that are older than the latest sample for a given metric.
out_of_order_time_window.duration	Duration of the out-of-order window in seconds.	`0s`	Used to configure how far back in time out-of-order samples can be accepted. Setting this to `0s` disables the feature.

Example: Enabling Out-of-Order Sample Storage

storage:
  tsdb:
    out_of_order_time_window: 300

Note: Enabling out-of-order sample storage can increase memory usage and affect query performance. It is recommended to set a reasonable time window based on your system's clock drift and data ingestion patterns.

12. Using WAL Compression

Configuration Option	Values	Description
storage.tsdb.wal-compression	`true` \| `false`	Enables or disables WAL compression. Defaults to `true`.

Example: Enabling WAL Compression

storage:
  tsdb:
    wal-compression: true

Note: WAL compression is available starting in Prometheus v2.30.0 and reduces disk I/O and storage usage by compressing the Write-Ahead Log (WAL) files.