Integrating with Thanos

1. Understanding Thanos Architecture

Thanos Components Overview

Query: Serves queries across multiple Prometheus instances and Thanos components
Receive: Receives metrics from Prometheus instances via remote write
Store: Stores and indexes time-series data from Prometheus instances
Rule: Evaluates recording and alerting rules across multiple Prometheus instances
Compactor: Compacts and downsamples long-term stored data

Component	Description	Key Features	Version
Query	Aggregates and serves queries from multiple sources	Global query view, federation support	0.32.0+
Receive	Accepts metrics via Prometheus remote write protocol	High availability, deduplication	0.32.0+
Store	Stores and indexes time-series data	Long-term storage, block format	0.32.0+
Rule	Evaluates recording and alerting rules	Global rule evaluation, alert routing	0.32.0+
Compactor	Compacts and downsamples data blocks	Data lifecycle management, retention policies	0.32.0+

Thanos vs Prometheus

Thanos: Distributed, long-term storage, global query view
Prometheus: Single instance, in-memory storage, local query scope

                ┌─────────────┐
                │   Prometheus│
                │   (Remote)  │
                └──────┬──────┘
                       │
            ┌────────────┴────────────┐
            │                         │
    ┌───────▼───────┐       ┌────────▼────────┐
    │   Thanos      │       │   Thanos        │
    │   Receive     │       │   Store         │
    └───────────────┘       └─────────────────┘
            │                         │
            └────────┬────────────────┘
                     │
         ┌──────────▼──────────┐
         │   Thanos Query      │
         └─────────────────────┘

2. Using Thanos Sidecar

The Thanos sidecar is a component that runs alongside a Prometheus instance to expose its data in a format compatible with Thanos.

Configuration Option	Description	Default Value	Notes
--prometheus.url	URL of the Prometheus instance to connect to.	`http://localhost:9090`	Must be accessible from the sidecar.
--grpc-address	Address to listen for gRPC connections.	`0.0.0.0:10901`	Used for communication with the Thanos Query component.
--http-address	Address to listen for HTTP connections.	`0.0.0.0:10902`	Used for metrics and health checks.
--objstore.config	Configuration file for object storage (e.g., S3, GCS).	`""`	Required for long-term storage.

Example: Sidecar Configuration

prometheus:
  url: http://localhost:9090
grpc-address: 0.0.0.0:10901
http-address: 0.0.0.0:10902
objstore:
  config:
    type: S3
    config:
      bucket: thanos-bucket
      region: us-east-1

Note: The sidecar must be configured to connect to the same Prometheus instance it is monitoring.

Warning: Ensure that the sidecar has access to the Prometheus instance's data directory for efficient operation.

3. Configuring Thanos Query

Option	Values	Notes
--query.config	`string`	Path to the query configuration file
--query.timeout	`duration`	Timeout for query requests (default: 2m0s)
--query.max-concurrency	`int`	Maximum number of concurrent query requests (default: 10)
--query.lookback-delta	`duration`	Lookback delta for query (default: 5m0s)
--query.max-stats-memory	`bytes`	Maximum memory to use for query stats (default: 1GB)
--query.log-level	`string`	Log level for query service (default: info)
--query.log-format	`string`	Log format for query service (default: logfmt)

Example: Basic Thanos Query Configuration

query:
  config: /etc/thanos/query.yaml
  timeout: 3m0s
  max-concurrency: 15
  lookback-delta: 10m0s
  max-stats-memory: 2GB
  log-level: debug
  log-format: json

Note: The --query.config option allows loading a YAML configuration file for more complex setups.

Warning: Setting --query.max-concurrency too high can lead to resource exhaustion.

Configuration Field	Description	Default Value	Units
query.timeout	Maximum time to wait for a query response	2m0s	Duration
query.max-concurrency	Maximum concurrent queries allowed	10	Integer
query.lookback-delta	Time to look back for data during queries	5m0s	Duration
query.max-stats-memory	Maximum memory used for query statistics	1GB	Bytes

4. Implementing Thanos Store Gateway

Example: Store Gateway Configuration

prometheus:
  config_file: "prometheus.yml"

thanos:
  store:
    grpc_address: "0.0.0.0:10901"
    http_address: "0.0.0.0:10902"
    data_dir: "/var/thanos/store"
    objstore.config: "objstore.yml"
    retention_period: "30d"
    block_rate_limit: 100
    max_block_size: "256MB"

Configuration Option	Description	Default Value	Notes
grpc_address	gRPC address to listen on for store gRPC API	`0.0.0.0:10901`
http_address	HTTP address to listen on for web UI and metrics	`0.0.0.0:10902`
data_dir	Directory to store local cache and temporary files	`/var/thanos/store`	Must be writable
objstore.config	Path to the object storage configuration file	`objstore.yml`	Required for remote storage
retention_period	How long to keep blocks before deletion	`30d`	Use `0s` to disable retention
block_rate_limit	Maximum blocks to process per second	`100`	Prevents overloading the system
max_block_size	Maximum size of a block to be stored	`256MB`	Adjust based on available memory

Note: The Store Gateway is responsible for serving blocks from object storage and handling queries that span multiple blocks.

Warning: Ensure that the objstore.config file is correctly configured to allow access to your object storage backend (e.g., S3, GCS, Azure Blob).

Key Features

Serves blocks from object storage (S3, GCS, etc.)
Handles queries across multiple blocks
Supports query sharding and replication
Can be scaled horizontally for better performance

5. Setting Up Thanos Compactor

Example: Basic Thanos Compactor Configuration

compactor:
  compaction:
    # Compaction interval
    interval: 1h
    # Number of concurrent compactions
    concurrency: 1
  # Data retention policy (optional)
  retention:
    # Retention period for raw data
    raw: 30d
    # Retention period for downsampled data
    downsample: 90d

Configuration Option	Description	Default Value	Notes
compactor.compaction.interval	How often compaction runs	`1h`	Time duration string (e.g., `30m`, `1h`)
compactor.compaction.concurrency	Number of concurrent compactions	`1`	Higher values may improve performance but increase resource usage
compactor.retention.raw	Retention period for raw data	`30d`	Must be a valid duration string (e.g., `1d`, `30d`)
compactor.retention.downsample	Retention period for downsampled data	`90d`	Must be a valid duration string (e.g., `1d`, `30d`)

Note: Thanos Compactor requires a storage backend (e.g., S3, GCS) to store compacted data and metadata. Ensure that the storage is properly configured before starting the compactor.

Warning: Compaction can be resource-intensive. Monitor CPU and memory usage during compaction to avoid performance degradation.

6. Using Thanos Ruler

The Thanos Ruler is a component of the Thanos ecosystem that evaluates recording and alerting rules against Prometheus data. It allows you to run rule evaluation on a distributed system, providing high availability and scalability for alerting and recording rules.

Configuration Option	Description	Default Value
--rule.file	Path to the rule files to be loaded. Can be specified multiple times.	None
--alertmanagers.url	URL of the Alertmanager to send alerts to. Can be specified multiple times.	None
--prometheus.url	URL of the Prometheus server to query. Defaults to http://localhost:9090.	http://localhost:9090
--ruler.config	Path to the ruler configuration file.	None
--ruler.storage.path	Path to the local storage directory for rule evaluation.	/data/ruler

Example: Basic Thanos Ruler Configuration

rule_files:
  - rules.yml
alertmanagers:
  - url: http://alertmanager:9093
prometheus:
  url: http://prometheus:9090
ruler:
  storage:
    path: /data/ruler

Note: Thanos Ruler requires a Prometheus instance to query and rule files to evaluate. Ensure that the Prometheus URL is accessible from the Thanos Ruler instance.

Warning: Rule evaluation can be resource-intensive. Monitor CPU and memory usage to prevent performance degradation.

Key Features of Thanos Ruler

Distributed Rule Evaluation: Thanos Ruler can distribute rule evaluation across multiple instances for high availability.
Alerting: It evaluates alerting rules and forwards alerts to Alertmanager.
Recording Rules: Thanos Ruler can evaluate recording rules to precompute and store results for faster queries.
Storage: It uses local storage to cache rule evaluation results.

7. Configuring Thanos Receive

Option	Values	Description
--receive.enabled	`true`	Enables the Thanos Receive component
--receive.replica-count	`1`	Number of replicas for the receive component
--receive.grpc-address	`0.0.0.0:10901`	gRPC address to listen on for incoming requests
--receive.http-address	`0.0.0.0:10902`	HTTP address to listen on for metrics and web UI
--receive.data-dir	`/var/thanos/receive`	Directory to store received data
--receive.remote-write.url	`http://prometheus:9090/api/v1/write`	URL to remote write endpoint of Prometheus instance
--receive.tenant-header	`X-Tenant`	Header name to use for tenant identification
--receive.default-tenant-id	`default`	Default tenant ID if no header is provided
--receive.hashmod	`true`	Use hashmod for consistent hashing of samples

Example: Basic Thanos Receive Configuration

receive:
  enabled: true
  grpc-address: 0.0.0.0:10901
  http-address: 0.0.0.0:10902
  data-dir: /var/thanos/receive
  remote-write:
    url: http://prometheus:9090/api/v1/write
  tenant-header: X-Tenant
  default-tenant-id: default

Note: Ensure that the --receive.data-dir directory is writable by the Thanos Receive process and has sufficient disk space for incoming metrics.

8. Implementing Object Storage

Storage Type	Configuration	Notes
S3 Compatible	`s3:` `bucket: "prometheus-backup"` `endpoint: "s3.amazonaws.com"` `access_key: "AKIAIOSFODNN7EXAMPLE"` `secret_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"`	Requires AWS S3 compatible API
GCS	`gcs:` `bucket: "prometheus-backup"` `service_account: "/path/to/service-account.json"`	Google Cloud Storage backend
Azure Blob	`azure:` `storage_account: "prometheusbackup"` `container: "prometheus"` `sas_token: "?sv=2019-02-02&ss=bq"`	Microsoft Azure Blob Storage
Swift	`swift:` `container: "prometheus"` `username: "user"` `password: "password"` `auth_url: "https://storage.myprovider.com/v2.0"`	OpenStack Swift object storage

Example: S3 Configuration

prometheus:
  storage:
    tsdb:
      retention: 30d
    remote_write:
      - url: http://prometheus-remote-write:9090/api/v1/write
    remote_read:
      - url: http://prometheus-remote-read:9090/api/v1/read
    object_storage:
      s3:
        bucket: "prometheus-backup"
        endpoint: "s3.amazonaws.com"
        access_key: "AKIAIOSFODNN7EXAMPLE"
        secret_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
        region: "us-east-1"

Note: Ensure that the IAM role or user has appropriate permissions to read and write to the specified bucket.

Warning: Using secret_key in configuration files is insecure. Use environment variables or IAM roles for production environments.

Key Considerations

Use NEW S3-compatible object storage for cost-effective long-term retention
Ensure proper region setting for optimal performance
Enable versioning and lifecycle policies on the storage bucket
Secure credentials using env or secret files

9. Using Thanos Query Frontend

The Thanos Query Frontend is a component of the Thanos ecosystem that provides a query interface for Prometheus-compatible systems. It allows users to query metrics from multiple Prometheus instances and Thanos components through a unified API.

Configuration Option	Description	Default Value	Notes
--query.timeout	Timeout for query requests	`2m`	Used to prevent long-running queries from consuming resources
--query.max-concurrent	Maximum number of concurrent query requests	`10`	Controls the level of parallelism in query execution
--query.auto-downsample	Enable automatic downsampling of data	`false`	Improves query performance for long-term data
--web.external-prefix	External path prefix for the frontend		Used when the frontend is served under a subpath

Example: Configuring Thanos Query Frontend

query:
  timeout: 2m
  max-concurrent: 10
  auto-downsample: true
web:
  external-prefix: /thanos

Note: The Thanos Query Frontend is typically used in conjunction with the query component to provide a unified query interface across multiple Prometheus instances.

Key Features

Unified query interface for multiple Prometheus instances
Supports downsampling of long-term data
Configurable timeout and concurrency settings
External path prefix support for reverse proxy setups

10. Configuring Deduplication

Option	Values	Description
--deduplication.enabled	`true` \| `false`	Enables or disables deduplication of samples from multiple Prometheus instances.
--deduplication.replica-label	`string`	Label name used to identify replicas (e.g., `job` or `instance`).
--deduplication.timeout	`duration`	Timeout for deduplication operations, defaults to `2m`.

Example: Enabling Deduplication

prometheus --deduplication.enabled=true \
          --deduplication.replica-label=job \
          --deduplication.timeout=3m

Note: Deduplication is critical when running multiple Prometheus instances for high availability or sharding.

11. Implementing Multi-Cluster Monitoring

Example: Configuring Thanos for Multi-Cluster Setup

prometheus:
  remote_write:
    - url: "http://thanos-remote-write:19291/api/v1/receive"
      queue_config:
        capacity: 1000
        max_samples_per_send: 1000

thanos:
  store:
    grpc_address: "0.0.0.0:10901"
    http_address: "0.0.0.0:10902"
    data_dir: "/var/thanos/store"
    objstore.config:
      type: S3
      config:
        bucket: "thanos-bucket"
        endpoint: "s3.amazonaws.com"
        access_key: "YOUR_ACCESS_KEY"
        secret_key: "YOUR_SECRET_KEY"

Component	Description	Configuration
Prometheus Remote Write	Writes metrics to Thanos receiver	`remote_write.url`
Thanos Store Gateway	Serves metrics from object storage	`store.grpc_address`, `objstore.config`
Query Frontend	Aggregates queries across clusters	`query.frontend`

Note: Ensure all Prometheus instances in clusters write to the same Thanos receiver endpoint.

Warning: Misconfigured remote write can lead to data loss or duplication.

Key Steps for Multi-Cluster Setup

Configure Prometheus instances to remote write to a shared Thanos receiver
Set up Thanos Store Gateway with object storage backend (e.g., S3)
Deploy Query Frontend to aggregate queries across clusters
Ensure consistent labeling and metric names across clusters

12. Monitoring Thanos Components

Component	Metrics Endpoint	Description	Notes
Thanos Query	`/metrics`	Exposes Prometheus-compatible metrics for query performance and internal state	Includes query latency, cache hits, and rule evaluation stats
Thanos Receive	`/metrics`	Exposes metrics for receiving and storing incoming metrics from Prometheus instances	Tracks receive rate, dropped samples, and storage size
Thanos Ruler	`/metrics`	Exposes metrics related to rule evaluation and alerting	Provides rule execution time, alert firing counts, and evaluation failures
Thanos Store	`/metrics`	Exposes metrics for store performance and data retrieval	Tracks block loading, query latency, and memory usage
Thanos Sidecar	`/metrics`	Exposes metrics for sidecar operation and data forwarding	Shows sync status, data send rate, and connection health

Example: Querying Thanos Component Metrics

curl http://thanos-query:10902/metrics | grep &#34;thanos_query_&#34;

Note: Thanos components expose metrics in Prometheus format, which can be scraped by a dedicated monitoring instance or integrated into existing setups.

Integrating with Thanos

1. Understanding Thanos Architecture

Thanos Components Overview

Thanos vs Prometheus

2. Using Thanos Sidecar

Example: Sidecar Configuration

3. Configuring Thanos Query

Example: Basic Thanos Query Configuration

4. Implementing Thanos Store Gateway

Example: Store Gateway Configuration

Key Features

5. Setting Up Thanos Compactor

Example: Basic Thanos Compactor Configuration

6. Using Thanos Ruler

Example: Basic Thanos Ruler Configuration

Key Features of Thanos Ruler

7. Configuring Thanos Receive

Example: Basic Thanos Receive Configuration

8. Implementing Object Storage

Example: S3 Configuration

Key Considerations

9. Using Thanos Query Frontend

Example: Configuring Thanos Query Frontend

Key Features

10. Configuring Deduplication

Example: Enabling Deduplication

11. Implementing Multi-Cluster Monitoring

Example: Configuring Thanos for Multi-Cluster Setup

Key Steps for Multi-Cluster Setup

12. Monitoring Thanos Components

Example: Querying Thanos Component Metrics

Was this article helpful?

Related Cheat Sheets

Using Role-Based Access Control

Implementing Monitoring and Logging

Understanding Linux Capabilities

Implementing Security Best Practices