Configuring Storage Retention

1. Setting Retention Time

Configuration Option	Values	Description
storage.tsdb.retention.time	`1d`, `7d`, `30d`, etc.	Duration for which samples are stored before being deleted. Default is `15d`.
storage.tsdb.retention.size	`512MB`, `1GB`, etc.	Maximum size of the data directory before old data is deleted based on retention time.

Example: Configuring Retention Time

storage:
  tsdb:
    retention:
      time: 30d
      size: 50GB

Note: Setting storage.tsdb.retention.time to 0s disables automatic deletion of samples.

Warning: Increasing retention time significantly increases disk usage. Ensure sufficient storage is available.

2. Configuring Retention Size

Option	Values	Description
storage.tsdb.retention.size	`2GB`	Maximum size of the data directory before old data is deleted. Defaults to `2GB` if not set.
storage.tsdb.retention.time	`15d`	Maximum time to keep data. Defaults to `15d` if not set.

Example: Setting retention size in Prometheus configuration

storage:
  tsdb:
    retention:
      size: 5GB

Note: Setting storage.tsdb.retention.size overrides the default time-based retention. If both size and time are set, Prometheus will use whichever limit is reached first.

3. Setting Max Block Duration

Configuration Option	Default Value	Description	Notes
max_block_duration	`2h`	Maximum duration of a block in the storage	Used to control how long data is kept in memory before being flushed to disk

Example: Configuring max_block_duration

storage:
  tsdb:
    max_block_duration: 3h

4. Understanding Retention Precedence

Configuration Source	Scope	Override Behavior	Notes
Command-line flags	`-storage.tsdb.retention.time`	Global override	Applies to all data in the Prometheus instance
Configuration file	`storage.tsdb.retention.time`	Global override	Same as CLI flag, but defined in YAML config
Remote write	`remote_write` block	Per remote write config	Can define retention settings for each remote storage target
Rule files	`rule_files`	Per recording rule	Retention can be set via `keep_fresh_samples` in recording rules
Alerting rules	`alerting.rules`	Per alert rule	Alerts may have different retention based on `keep_fresh_samples`

Example: Setting retention via CLI flag

prometheus --storage.tsdb.retention.time=30d

Example: Setting retention in config file

storage:
  tsdb:
    retention:
      time: 30d

Note: The storage.tsdb.retention.time setting in the config file or CLI flag takes precedence over any remote write retention settings.

Retention Precedence Summary

Global settings (CLI or config) apply to all data unless overridden
Remote write retention is specific to each remote target
Recording and alerting rules can override global settings with keep_fresh_samples
Higher precedence configurations override lower ones in the hierarchy

5. Calculating Storage Requirements

Component	Storage Type	Retention Period	Estimated Size
Samples	Time Series Data	30 days	~2GB per 1M samples
Chunks	Compressed Blocks	2 hours	~50MB per 1M samples
Index	Metadata Index	30 days	~50MB per 1M samples

Example: Estimating Storage for 10M Samples

# Calculate total storage needed for 10M samples
# Assuming 2GB per 1M samples for raw data
# And 50MB per 1M samples for chunks and index

TOTAL_RAW_DATA="20GB"
TOTAL_CHUNKS="500MB"
TOTAL_INDEX="500MB"

echo "Total estimated storage: $((20 + 0.5 + 0.5)) GB"

Note: Actual storage usage may vary based on sample frequency, label cardinality, and compression settings.

Storage Calculation Factors

Sample Frequency: Higher frequency = more data
Label Cardinality: More unique labels = larger index
Retention Period: Longer periods = more data retained
Compression: Prometheus uses Snappy compression to reduce size

6. Monitoring Disk Usage

Metric	Description	Unit	Notes
prometheus_local_storage_memory_chunks	Number of chunks in memory	chunks	Increases as data is ingested
prometheus_local_storage_series	Number of series in storage	series	Represents unique time series
prometheus_local_storage_disk_bytes	Bytes used on disk for storage	bytes	Directly impacts retention settings
prometheus_local_storage_checkpoint_duration_seconds	Duration of checkpoint creation in seconds	seconds	Can indicate disk I/O issues

Example: Disk Usage Alert Configuration

groups:
- name: disk-usage-alerts
  rules:
  - alert: HighDiskUsage
    expr: prometheus_local_storage_disk_bytes > (10 * 1024 * 1024 * 1024)
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High disk usage detected"
      description: "Prometheus is using more than 10GB of disk space."

Note: Disk usage can be monitored via Prometheus's built-in metrics or external tools like df on Unix systems.

Warning: If disk space is exhausted, Prometheus will stop ingesting new samples and may cause data loss.

7. Implementing Retention Policies

Policy	Description	Configuration	Notes
storage.tsdb.retention.time	Duration to keep samples in storage	`2d`	Default is `15d`. Can be set to `0` to disable automatic cleanup
storage.tsdb.retention.size	Maximum size of storage before purging oldest data	`50GB`	Applies when `retention.time` is exceeded or if set to `0`
storage.tsdb.min-block-size	Minimum size of a block in bytes	`512MB`	Smaller blocks may cause performance issues
storage.tsdb.max-block-size	Maximum size of a block in bytes	`2GB`	Blocks larger than this are split into smaller ones

Example: Setting retention policy via config file

storage:
  tsdb:
    retention:
      time: 30d
      size: 100GB

Note: Setting storage.tsdb.retention.time to 0 disables automatic cleanup and keeps all data indefinitely.

Warning: Setting storage.tsdb.retention.size too low can lead to frequent block compactions and increased I/O.

8. Using Remote Storage for Long-Term

Storage Type	Description	Configuration	Notes
remote_write	Writes metrics to a remote endpoint	`{"url": "http://remote-prometheus:9090/api/v1/write"}`	Used for long-term storage and backup
remote_read	Reads metrics from a remote endpoint	`{"url": "http://remote-prometheus:9090/api/v1/read"}`	Allows querying historical data from remote storage

Example: Configuring remote_write

remote_write:
  - url: http://prometheus-remote:9090/api/v1/write
    queue_config:
      capacity: 1000
      max_shards: 10

Example: Configuring remote_read

remote_read:
  - url: http://prometheus-remote:9090/api/v1/read
    name: "remote"

Note: Remote storage is ideal for long-term retention and compliance requirements.

Warning: Ensure network connectivity to remote endpoints to avoid data loss.

9. Configuring Block Compaction

Option	Values	Description
compaction.retention	`30d`	How long to retain samples in the storage before they are deleted.
compaction.block-size	`256MB`	Size of each block in the storage.
compaction.max-block-size	`512MB`	Maximum size of a block before it is compacted.
compaction.min-block-duration	`2h`	Minimum duration of a block before it is eligible for compaction.
compaction.max-block-duration	`1d`	Maximum duration of a block before it is compacted.

Example: Compaction Configuration

storage:
  retention: 30d
  compaction:
    block-size: 256MB
    max-block-size: 512MB
    min-block-duration: 2h
    max-block-duration: 1d

Note: Compaction reduces storage space by merging blocks and removing stale samples. Ensure sufficient disk space for temporary files during compaction.

10. Managing Head Block Size

Configuration Option	Description	Default Value	Notes
storage.tsdb.head_chunks	Maximum number of chunks to keep in memory for each series	`1000`	Controls memory usage for active series
storage.tsdb.min_block_duration	Minimum duration of a block in the storage	`2h`	Smaller blocks can increase storage overhead
storage.tsdb.max_block_duration	Maximum duration of a block in the storage	`2d`	Blocks larger than this will be split

Example: Adjusting Head Block Size

storage:
  tsdb:
    head_chunks: 2000
    min_block_duration: 1h
    max_block_duration: 3d

Note: Increasing head_chunks can improve query performance for long-running series but will increase memory consumption.

11. Setting Minimum Block Duration

Configuration Option	Default Value	Description
min-block-duration	`2h`	Minimum duration of a block in the storage. Blocks are stored in chunks of this duration.

Example: Configuring min-block-duration

storage:
  tsdb:
    min-block-duration: 3h

Note: Setting this value too low can lead to increased storage overhead and reduced query performance.

12. Understanding Tombstone Cleanup

Property	Description	Default Value	Notes
storage.tsdb.retention.duration	Duration for which samples are retained in storage	15d	Used to configure how long data is kept before being deleted or marked for deletion
storage.tsdb.tombstone.file	Path to the tombstone file used for tracking deleted samples	$data-dir/tombstones.json	Contains metadata about deleted samples and their time ranges
storage.tsdb.tombstone.cleanup.interval	Interval at which tombstones are cleaned up from disk	1h	Controls how often the system checks for and removes old tombstone entries

Example: Configuring Tombstone Cleanup

storage:
  tsdb:
    retention:
      duration: 30d
    tombstone:
      file: /prometheus/data/tombstones.json
      cleanup:
        interval: 2h

Note: Tombstone cleanup does not immediately delete data from disk. Instead, it marks samples as deleted and removes them during compaction or when the retention period expires.