Monitoring Prometheus Itself

1. Tracking Scrape Duration

Metric Name	Description	Labels	Unit
prometheus_target_scrape_duration_seconds	Duration of the last scrape in seconds	`instance`, `job`	seconds
prometheus_target_scrape_duration_seconds_summary	Summary of scrape durations with quantiles	`instance`, `job`	seconds

Example: Querying Scrape Duration

prometheus_target_scrape_duration_seconds{job="node"} * 1000

Note: The metric prometheus_target_scrape_duration_seconds is updated every time a scrape completes, providing real-time insight into scraping performance.

Warning: If scrape durations are consistently high, it may indicate issues with the target or Prometheus configuration.

2. Monitoring Sample Ingestion

Property	Description	Default Value	Notes
prometheus.remote.storage.sample.ingestion.duration	Duration for sample ingestion in seconds	30s	Configurable via `--storage.remote.sample.ingestion.duration`
prometheus.remote.storage.sample.ingestion.batch.size	Batch size for sample ingestion	1000	Adjusts batch processing for performance tuning
prometheus.remote.storage.sample.ingestion.max.concurrent.requests	Maximum concurrent remote write requests	10	Controls parallelism of ingestion

Example: Configuring Sample Ingestion

remote_write:
  - url: http://remote-prometheus:9090/api/v1/write
    queue_config:
      capacity: 10000
      max_samples_per_send: 1000
      batch_send_deadline: 5s

Note: Ensure that the remote storage system can handle the configured batch size and concurrent requests to avoid ingestion delays or failures.

3. Checking WAL Corruption

WAL (Write-Ahead Log) corruption can lead to data loss or monitoring gaps in Prometheus. Corruptions typically occur due to disk issues, improper shutdowns, or bugs in the storage engine.

Check	Description	Command/Method
WAL file integrity	Verifies the integrity of WAL files by checking for corrupted blocks or inconsistent data.	`promtool check-wal /path/to/wal/directory`
WAL file size	Ensures WAL files are within expected size limits to prevent excessive disk usage.	`ls -lh /path/to/wal/directory`
WAL segment consistency	Checks that WAL segments are properly aligned and not missing any data.	`promtool check-wal --verbose /path/to/wal/directory`

Example: Running WAL Integrity Check

promtool check-wal /var/lib/prometheus/wal

Note: If corruption is detected, Prometheus will not start or may drop data. It's recommended to backup the WAL directory before running checks.

Warning: Corrupted WAL files may require manual recovery or reimporting data from backups.

4. Monitoring Rule Evaluation

Property	Description	Values	Notes
rule_files	Path to rule files	`["rules.yml"]`	Can be a glob pattern
evaluation_interval	How often to evaluate rules	`1m`	Default is 1 minute
rule_evaluation_duration_seconds	Duration of rule evaluation in seconds	`0.05`	Used for alerting and recording rules

Example: Rule Evaluation Configuration

rule_files:
  - rules.yml
evaluation_interval: 1m

Example: Recording Rule

groups:
  - name: example
    rules:
      - record: instance:cpu_usage:ratio
        expr: 1 - (node_cpu_seconds_total{mode="idle"} / node_cpu_seconds_total)

Example: Alerting Rule

groups:
  - name: example
    rules:
      - alert: HighCPUUsage
        expr: instance:cpu_usage:ratio > 0.8
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "High CPU usage detected"

Note: Rule evaluation happens on the Prometheus server and can be monitored via metrics like prometheus_rule_evaluation_failures_total.

Warning: Complex rules can lead to high resource usage and slow evaluation times.

5. Tracking Query Performance

Metric	Description	Notes
prometheus_engine_query_duration_seconds	Duration of query execution in seconds	Measures time taken to execute a query, including parsing and evaluation
prometheus_engine_query_samples	Total number of samples evaluated during query execution	Useful for identifying queries that are too expensive in terms of data points
prometheus_engine_queries	Number of queries currently being evaluated	Indicates current query load on the Prometheus instance
prometheus_engine_queries_concurrent_max	Maximum number of concurrent queries allowed	Defaults to 20; can be configured via --query.max-concurrency

Example: Querying for slow queries

rate(prometheus_engine_query_duration_seconds_sum[5m]) / rate(prometheus_engine_query_duration_seconds_count[5m]) > 1

Note: The prometheus_engine_query_duration_seconds histogram is useful for identifying queries that are taking longer than expected. It includes _sum and _count for calculating average query duration over time.

Example: Monitoring query sample count

prometheus_engine_query_samples > 10000

Warning: Queries that return too many samples can cause high memory usage and slow down Prometheus. Use max_samples limit to prevent such queries from overwhelming the system.

Query Performance Best Practices

Use rate() and irate() functions to calculate rates over time
Limit query complexity by avoiding complex aggregations or large data sets
Monitor prometheus_engine_queries_concurrent_max to prevent query overload
Use --query.max-concurrency to control concurrent query limits

6. Monitoring Storage Size

Configuration Option	Default Value	Description
storage.tsdb.retention.time	`15d`	How long to retain samples in the storage. Can be specified as a duration (e.g., `15d`, `30h`) or in milliseconds.
storage.tsdb.retention.size	`0`	Maximum size of the storage before old data is deleted. If set to 0, no limit is enforced.
storage.tsdb.min-block-duration	`2h`	Minimum duration of a block before it is considered for compaction.
storage.tsdb.max-block-duration	`10d`	Maximum duration of a block before it is considered for compaction.

Example: Setting Storage Retention Size

storage:
  tsdb:
    retention:
      size: 50GB

Note: When storage.tsdb.retention.size is set, Prometheus will delete the oldest data blocks once that limit is exceeded. This setting can be used in combination with storage.tsdb.retention.time to ensure that both time and size limits are respected.

Storage Size Monitoring Tips

Monitor prometheus_tsdb_head_chunks and prometheus_tsdb_head_series to understand current usage.
Use prometheus_tsdb_storage_blocks to track how many blocks are stored.
Set storage.tsdb.retention.size to prevent disk space exhaustion.

7. Checking Scrape Failures

Field	Description	Example
scrape_duration_seconds	Duration of the scrape in seconds	`0.05`
scrape_samples_scraped	Number of samples scraped from the target	`120`
scrape_samples_post_metric_relabeling	Number of samples after metric relabeling	`100`
scrape_series_added	Number of series added to the target	`50`
scrape_samples_dropped	Number of samples dropped due to relabeling or other reasons	`20`

Example: Checking scrape failures via Prometheus UI

prometheus --config.file=/etc/prometheus/prometheus.yml

Example: Querying scrape failures in Prometheus

rate(prometheus_target_scrape_pool_duration_seconds[5m])

Note: Use prometheus_target_scrape_pool_duration_seconds to detect slow scrapes or failures.

Example: Alerting on scrape failures

groups:
- name: scrape-failures
  rules:
  - alert: TargetDown
    expr: up == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Target is down"

8. Tracking Remote Write

Field	Description	Values	Notes
remote_write	Configures remote write endpoints	`[<remote_write>]`	Array of remote write configurations
url	Remote endpoint URL	`string`	Must be a valid HTTP(S) URL
name	Identifier for the remote write config	`string`	Used for labeling metrics
write_retries	Number of retry attempts on write failure	`int`	Default: 10
queue_config	Configuration for the write queue	`{}`	Controls batching and retry behavior
remote_timeout	Timeout for remote write requests	`duration`	Default: 30s

Example: Remote Write Configuration

remote_write:
  - url: https://prometheus.example.com/api/v1/write
    name: "remote-1"
    write_retries: 5
    remote_timeout: 1m
    queue_config:
      capacity: 10000
      max_shards: 100

Note: Remote write is a core feature for long-term storage and cross-prometheus data sharing.

Remote Write Key Features

Supports multiple remote endpoints
Built-in retry and queueing for reliability
Can be used with Prometheus, Cortex, or Thanos
Configurable batch sizes and timeouts

9. Monitoring Memory Usage

Configuration Option	Default Value	Description
storage.tsdb.max-block-duration	`2h`	Maximum duration a block can last before being closed and written to disk. Affects memory usage during compaction.
storage.tsdb.min-block-duration	`2h`	Minimum duration a block can last. Lower values increase memory usage due to more frequent compactions.
storage.tsdb.retention.time	`15d`	How long to keep data in memory before flushing to disk. Longer retention increases memory pressure.
storage.tsdb.head.chunks	`120`	Number of chunks to keep in memory for each series. Higher values increase memory usage.

Example: Memory Usage Configuration

storage:
  tsdb:
    max-block-duration: 2h
    min-block-duration: 2h
    retention.time: 15d
    head:
      chunks: 120

Note: Memory usage in Prometheus is primarily driven by the number of time series and their chunk size. Adjusting head.chunks can significantly impact memory consumption.

Memory Metric	Description	Monitoring Command
prometheus_engine_heap_size_bytes	Current heap size of the Prometheus engine.	`prometheus_engine_heap_size_bytes`
prometheus_engine_memory_usage_bytes	Total memory used by the Prometheus engine.	`prometheus_engine_memory_usage_bytes`
prometheus_tsdb_head_chunks	Number of chunks in the head block.	`prometheus_tsdb_head_chunks`
prometheus_tsdb_head_series	Number of series in the head block.	`prometheus_tsdb_head_series`

10. Checking Compaction Status

Property	Description	Value
prometheus.tsdb.compactions.total	Total number of compactions that were executed for the storage	`0`
prometheus.tsdb.compaction.duration.seconds	Duration of compaction runs	`0.0`
prometheus.tsdb.blocks.loaded	Number of currently loaded blocks	`0`

Example: Querying Compaction Metrics

curl -s http://localhost:9090/api/v1/query?query=prometheus.tsdb.compactions.total

Note: Compaction is the process of merging multiple chunks into larger chunks to optimize storage and query performance.

11. Tracking Time Series Count

Property	Description	Default Value	Notes
prometheus_tsdb_head_series	Current number of series in the head block	N/A	Exposes the total count of time series in Prometheus' head chunk
prometheus_tsdb_head_chunks	Current number of chunks in the head block	N/A	Represents the number of chunks stored in the head block
prometheus_tsdb_head_chunks_total	Total number of chunks created in the head block	N/A	Tracks all chunks created since Prometheus started

Example: Querying Series Count

curl -s http://localhost:9090/api/v1/query?query=prometheus_tsdb_head_series

Example: Monitoring Series Growth

groups:
- name: prometheus-alerts
  rules:
  - alert: HighSeriesCount
    expr: prometheus_tsdb_head_series > 100000
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High number of time series detected"

12. Monitoring Alert Queue

Property	Description	Default Value	Notes
prometheus_alert_queue_capacity	Capacity of the alert queue	10000	Number of alerts that can be queued before dropping
prometheus_alert_queue_length	Current length of the alert queue	0	Current number of alerts in the queue
prometheus_alert_queue_dropped_total	Total number of alerts dropped due to queue overflow	0	Counter for dropped alerts

Example: Alert Queue Metrics


prometheus_alert_queue_capacity 10000
prometheus_alert_queue_length 500
prometheus_alert_queue_dropped_total 25

Note: The alert queue is used to buffer alerts before they are sent to the Alertmanager. If the queue is full, new alerts will be dropped.