Optimizing Query Performance

1. Using Recording Rules

Recording rules allow you to precompute frequently used or expensive queries and store the results as new time series. This can significantly improve query performance by reducing the amount of data Prometheus needs to scan during queries.

Field	Description	Example
record	Name of the new time series to be created	`http_request_duration_seconds_bucket`
expr	PromQL expression to evaluate and store	`rate(http_request_duration_seconds_bucket[5m])`
labels	Optional labels to add to the recorded metric	`{job="api-server"}`

Example: Recording Rule Configuration

groups:
- name: http_rules
  rules:
  - record: http_request_duration_seconds_bucket
    expr: rate(http_request_duration_seconds_bucket[5m])
    labels:
      job: api-server

Note: Recording rules are evaluated at the same interval as the scrape interval, or less frequently if configured. They help reduce query load on Prometheus by caching results.

Warning: Avoid creating too many recording rules as they can increase memory usage and potentially cause performance issues.

2. Limiting Query Range

Option	Values	Description
range	`1h`, `1d`, `1w`	Defines the time range for the query. Example: `rate(cpu_usage{job="web"}[5m])` for the last 5 minutes.
start	`2024-01-01T00:00:00Z`	Start time for the query. Optional, defaults to current time minus range.
end	`2024-01-01T01:00:00Z`	End time for the query. Optional, defaults to current time.
step	`15s`, `1m`	Step size for the query. Determines how often to sample data.

Example: Limiting Query Range

rate(cpu_usage{job="web"}[5m]) 
  @ 2024-01-01T00:00:00Z 
  [1h]

Note: Using @ with a timestamp limits the query to that specific time window. This is useful for historical analysis or debugging.

3. Avoiding High-Cardinality Queries

Query Pattern	Description	Impact	Best Practice
count_values()	Counts occurrences of each value for a label	High cardinality if label has many unique values	Use `topk()` or `quantile()` instead
label_values()	Lists all values for a label	Can return thousands of unique values	Avoid in queries; use `group_left` or `group_right` with known values
rate(metric{label=~".*"})	Matches all values with regex	Can cause high cardinality and slow queries	Use specific label values or `count()` to limit scope

Example: High-cardinality query


rate(http_requests_total{method=~".*"}[5m])

Example: Optimized query


rate(http_requests_total{method="GET"}[5m])

Note: High-cardinality queries can cause Prometheus to use excessive memory and CPU, leading to degraded performance or out-of-memory errors.

4. Using Efficient Selectors

Selector Type	Description	Performance Impact	Best Practice
metric_name	Matches all time series with the given name	Fastest - no label matching	Use when you only need the metric name
metric_name{label="value"}	Matches time series with the given name and label value	Moderate - requires label matching	Use specific labels to reduce result set size
metric_name{label=~"regex"}	Matches time series with the given name and label matching regex	Slow - regex evaluation on all matching series	Avoid unless necessary; use exact matches when possible
metric_name{label!~"regex"}	Matches time series with the given name and label NOT matching regex	Very slow - evaluates all series	Avoid; consider filtering at ingestion or using alternative queries

Example: Efficient Selector Usage


# Fast query - no labels
up

# Efficient query - specific label match
http_requests_total{method="GET"}

# Less efficient - regex matching on labels
http_requests_total{method=~"POST|PUT"}

# Inefficient - negative regex match (avoid)
http_requests_total{method!~"GET|POST"}

Note: Label values should be kept as short and stable as possible to improve query performance.

Warning: Using regex patterns in label matching can significantly increase query execution time, especially on large datasets.

5. Implementing Query Timeouts

Configuration Option	Default Value	Description
query.timeout	`2m`	Maximum time a query is allowed to run before being cancelled
query.max-samples	`50000000`	Maximum number of samples a single query is allowed to load

Example: Setting Query Timeout in Prometheus Configuration

query:
  timeout: 30s
  max-samples: 100000000

Note: Setting a query timeout prevents long-running queries from consuming excessive resources and affecting other queries.

Warning: Setting the timeout too low may cause valid queries to be prematurely cancelled, especially for complex aggregations or large datasets.

6. Configuring Max Samples

Option	Values	Notes
max_samples	`1000000`	Maximum number of samples a single query can return
max_samples_per_query	`1000000`	Alias for `max_samples`, deprecated in v2.30+

Example: Setting max_samples in Prometheus configuration

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "prometheus.rules"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

# Configure max samples
query:
  max_samples: 5000000

Note: Setting max_samples too low can cause queries to be truncated, while setting it too high can lead to memory exhaustion or query timeouts.

7. Using Subqueries Efficiently

Subquery Type	Description	Best Practices	Notes
Instant Vector Subqueries	Queries that return a single value at a specific timestamp	Use with `rate()` or `increase()` for counter metrics	Can be inefficient if not scoped properly
Range Vector Subqueries	Queries that return a vector over a time range	Use `sum_over_time()`, `avg_over_time()` for aggregations	More resource-intensive than instant queries
Aggregation Subqueries	Queries that aggregate data over time windows	Use `group_left`, `group_right` for joins	Can cause high memory usage if not optimized

Example: Efficient Subquery with Aggregation

rate(http_requests_total[5m]) * 60

Example: Range Vector Subquery

sum_over_time(http_requests_total[1h]) / 3600

Note: Subqueries can significantly impact query performance. Always test with query_range endpoint to measure latency.

Warning: Avoid nested subqueries as they can lead to exponential query execution times.

8. Avoiding Regex Overuse

Pattern	Description	Impact	Better Alternative
/.foo./	Matches any string containing "foo"	Scans entire string for each match	`/.foo./` → `count_foo`
/.foo./	Matches any string containing "foo"	Can cause performance issues on large datasets	`count_foo` or `sum by (foo)`

Example: Inefficient Regex Usage


count_over_time({job="web"}[5m]) =~ ".*error.*"

Example: Efficient Alternative


count_over_time({job="web", error="true"}[5m])

Note: Regex patterns in queries can be expensive and should be avoided when direct metric filtering is possible.

9. Implementing Result Caching

Example: Using Prometheus Caching with Grafana

prometheus:
  enable_remote_write: true
  remote_write:
    - url: "http://grafana-agent:9090/api/prom/push"
      queue_config:
        capacity: 1000
        max_shards: 10
        min_shard_size: 100

Configuration Option	Description	Default Value	Notes
enable_remote_write	Enables remote write support for caching results	false	Required to enable caching
remote_write.url	URL to send cached metrics to	None	Must be a valid remote write endpoint
queue_config.capacity	Maximum number of samples in the queue	1000	Higher values allow more buffering
queue_config.max_shards	Maximum number of shards for parallel processing	10	Higher values increase concurrency
queue_config.min_shard_size	Minimum size of each shard in the queue	100	Controls batch size for writes

Note: Caching is implemented via remote write to a caching agent or storage system.

Warning: Ensure that the remote write endpoint supports caching to avoid performance degradation.

Best Practices for Caching

Use queue_config to optimize batch sizes and shard counts
Monitor queue capacity to prevent data loss during high load
Configure enable_remote_write to enable caching features

10. Monitoring Query Performance

Example: Using Prometheus UI to Monitor Query Performance

# Query to check query duration
rate(prometheus_engine_query_duration_seconds_sum[5m]) / rate(prometheus_engine_query_duration_seconds_count[5m])

# Example result:
# {instance="localhost:9090",job="prometheus"} 0.002345

Metric	Description	Unit	Notes
prometheus_engine_query_duration_seconds_sum	Total time spent evaluating queries in seconds	seconds	Used to calculate average query duration
prometheus_engine_query_duration_seconds_count	Number of queries evaluated	count	Used to calculate average query duration
prometheus_engine_query_duration_seconds_bucket	Bucketed query durations for histogram	seconds	Used to analyze query performance distribution

Note: The query duration metrics are exposed by Prometheus itself and can be used to monitor the performance of queries being executed.

Example: Querying Query Duration Histogram

# Query to get 99th percentile of query duration
histogram_quantile(0.99, sum by(le) (rate(prometheus_engine_query_duration_seconds_bucket[5m])))

# Example result:
# {instance="localhost:9090",job="prometheus"} 0.056789

Query Performance Monitoring Best Practices

Monitor query duration using prometheus_engine_query_duration_seconds_sum and prometheus_engine_query_duration_seconds_count
Use histogram quantiles to understand query performance distribution
Set up alerts for queries that exceed a certain duration threshold
Use rate() function to calculate query rates over time windows

11. Using Lookback Delta

Property	Description	Default Value	Notes
lookback_delta	Time range to look back for data when evaluating a query	`5m`	Controls how far back Prometheus searches for data points when evaluating a query. Affects query performance and accuracy.

Example: Configuring lookback_delta

query:
  lookback_delta: 10m

Note: Increasing lookback_delta can improve accuracy for long-running queries but may increase memory usage and query latency.

Warning: Setting lookback_delta too high can lead to excessive memory consumption and slow query execution.

12. Optimizing Aggregations

Aggregation Function	Description	Performance Notes	Example
sum()	Sums values across series	Can be expensive with many time series. Use `sum by (label)` to group	`sum by (job) (node_cpu_seconds_total)`
avg()	Averages values across series	Similar to sum; consider using `avg by (label)`	`avg by (instance) (http_requests_total)`
max()	Maximum value across series	Efficient, but grouping can reduce overhead	`max by (job) (up)`
min()	Minimum value across series	Similar to max; use grouping for efficiency	`min by (job) (up)`
count()	Counts number of values	Fast, but can be slow if used with many series	`count by (job) (up)`
stddev()	Standard deviation of values	Computationally intensive; avoid on large datasets	`stddev by (job) (node_cpu_seconds_total)`
quantile()	Quantile (e.g., 0.95) of values	Expensive; consider using `histogram_quantile()` for histograms	`quantile by (job) (0.95, http_request_duration_seconds)`

Note: Use group_left, group_right carefully to avoid expensive joins.

Warning: Aggregations over large numbers of time series can cause high memory usage and slow query execution.