Implementing Advanced PromQL Patterns
1. Calculating Percentage Change
| Expression | Description | Example |
|---|---|---|
| increase(metric[5m]) / increase(metric[5m]) * 100 | Percentage change over a fixed time window | increase(http_requests_total[5m]) / increase(http_requests_total[5m]) * 100 |
| rate(metric[5m]) / rate(metric[5m]) * 100 | Percentage change using rate function | rate(http_requests_total[5m]) / rate(http_requests_total[5m]) * 100 |
| (current_value - previous_value) / previous_value * 100 | Manual calculation using delta and offset | (http_requests_total - http_requests_total offset 5m) / http_requests_total offset 5m * 100 |
Example: Percentage Change in HTTP Requests
increase(http_requests_total[5m]) / increase(http_requests_total[5m]) * 100
Note: The
increase function is preferred over rate for percentage change calculations as it accounts for counter resets.
2. Implementing Anomaly Detection
Anomaly detection in Prometheus involves identifying unusual patterns or deviations from expected behavior in time-series data. This is typically achieved using PromQL queries that leverage statistical methods, thresholds, or machine learning models.
| Method | Description | Use Case | Example |
|---|---|---|---|
| Statistical Thresholds | Using standard deviations or percentiles to detect anomalies based on historical data distribution. | Detecting outliers in system metrics like CPU usage or memory consumption. | rate(node_cpu_seconds_total[5m])[1h:1m] > 2 * stddev(rate(node_cpu_seconds_total[5m])[1h:1m]) |
| Rate-Based Detection | Identifying anomalies by analyzing rate of change in metrics over time. | Monitoring sudden spikes or drops in network traffic or request rates. | abs(rate(http_requests_total[5m])) > abs(avg(rate(http_requests_total[5m])) + 3 * stddev(rate(http_requests_total[5m]))) |
| Machine Learning Models | Applying ML models trained on historical data to predict normal behavior and flag deviations. | Advanced anomaly detection for complex systems with multiple interdependent metrics. | predict_linear(node_cpu_seconds_total[5m], 3600) < node_cpu_seconds_total |
Example: Detecting CPU Usage Anomalies
rate(node_cpu_seconds_total[5m])[1h:1m] > 2 * stddev(rate(node_cpu_seconds_total[5m])[1h:1m])This query detects CPU usage anomalies by comparing current rates to historical standard deviations.
Example: Detecting Network Traffic Spikes
abs(rate(http_requests_total[5m])) > abs(avg(rate(http_requests_total[5m])) + 3 * stddev(rate(http_requests_total[5m])))This query identifies network traffic anomalies by detecting rates significantly higher than average plus three standard deviations.
3. Using Moving Averages
Moving averages are useful for smoothing out short-term fluctuations and highlighting longer-term trends in time series data. Prometheus provides several functions to calculate moving averages over a specified window.
| Function | Description | Parameters | Example |
|---|---|---|---|
| rate() | Calculates the per-second average rate of increase over a time window. | rate(metric[window]) |
rate(http_requests_total[5m]) |
| increase() | Calculates the increase over a time window, useful for counters. | increase(metric[window]) |
increase(cpu_usage_seconds_total[1h]) |
| avg_over_time() | Averages the values over a specified time window. | avg_over_time(metric[window]) |
avg_over_time(http_response_time_seconds[10m]) |
| max_over_time() | Reports the maximum value over a specified time window. | max_over_time(metric[window]) |
max_over_time(memory_usage_bytes[1h]) |
| min_over_time() | Reports the minimum value over a specified time window. | min_over_time(metric[window]) |
min_over_time(disk_io_bytes[5m]) |
Note: The
rate() function is particularly useful for counter metrics, as it accounts for resets in the counter values.
4. Calculating Ratio Metrics
| Metric | Description | Formula | Notes |
|---|---|---|---|
| http_requests_total | Total number of HTTP requests | rate(http_requests_total[5m]) |
Rate of requests over 5-minute window |
| http_requests_failed_total | Total number of failed HTTP requests | rate(http_requests_failed_total[5m]) |
Rate of failed requests over 5-minute window |
| error_rate | Ratio of failed to total requests | rate(http_requests_failed_total[5m]) / rate(http_requests_total[5m]) |
Expressed as a decimal (e.g., 0.02 = 2%) |
Example: Calculating Error Rate
rate(http_requests_failed_total[5m]) / rate(http_requests_total[5m])
Note: Ensure both metrics have the same time range for accurate ratio calculation.
Example: Error Rate with Label Filtering
rate(http_requests_failed_total{status_code!~"2.*"}[5m]) / rate(http_requests_total[5m])
5. Implementing Threshold Detection
Threshold detection in Prometheus involves identifying when metric values cross predefined limits. This is commonly achieved using PromQL functions and alerting rules.
| Function | Description | Usage Example |
|---|---|---|
| absent() | Checks for the absence of a metric over a time window. | absent(up{job="myapp"}) |
| rate() | Calculates the per-second average rate of increase of a counter. | rate(http_requests_total[5m]) > 100 |
| increase() | Calculates the increase in a counter over a time range. | increase(cpu_usage_seconds_total[1h]) > 3600 |
| clamp_max() | Clamps metric values to a maximum threshold. | clamp_max(cpu_usage_percent, 80) |
| clamp_min() | Clamps metric values to a minimum threshold. | clamp_min(memory_usage_bytes, 1024) |
Note: Thresholds should be based on historical data and system behavior to avoid false positives.
Warning: Using
rate() on counters without resets can lead to incorrect results if the counter is reset.
Threshold Detection Best Practices
- Use
rate()for counters to normalize increases over time. - Combine thresholds with
absent()to detect missing data or unavailability. - Apply
clamp_min()andclamp_max()to prevent extreme outliers from skewing averages. - Set thresholds dynamically using
histogram_quantile()for percentile-based alerts.
6. Using Absent for Alerting
The absent() function in PromQL is used to detect when a metric is missing or has no data points over a specified time range. It's particularly useful for alerting when expected metrics are not being collected.
| Function | Description | Return Value |
|---|---|---|
| absent(metric) | Returns 1 if the metric is missing or has no data points over the specified range, otherwise returns 0 | 1 or 0 |
Example: Alerting on Missing Metrics
absent(up{job="web-server"})This expression will return 1 if the up metric for the job web-server is missing or has no data points, indicating that the server might be down.
Example: Alerting on Missing Metrics with Time Range
absent(up{job="web-server"}[5m])This expression checks for the absence of the up metric over a 5-minute window. If no data is found, it returns 1.
Note: The
absent() function is particularly useful in alerting rules to detect when a metric is not being collected, which can indicate system issues or misconfigurations.
7. Calculating Request Rates
| Metric | Description | Usage |
|---|---|---|
| rate(http_requests_total[5m]) | Calculates the per-second rate of increase of http_requests_total over a 5-minute window | Useful for measuring request volume trends |
| irate(http_requests_total[5m]) | Calculates the per-second rate of increase using only the last two samples | Better for alerting due to its sensitivity to recent changes |
| increase(http_requests_total[1h]) | Calculates the total increase in http_requests_total over a 1-hour window | Useful for calculating total request counts over time windows |
Note: The
rate() function is preferred for long-term trends, while irate() is better for alerting due to its sensitivity to recent changes.
8. Implementing SLO Queries
Example: Basic SLO Query Using Counters
100 - (sum(increase(http_request_duration_seconds_count{job="api"}[1h])) by (job) / sum(increase(http_request_duration_seconds_count{job="api"}[1h])) by (job) * 100)| Component | Description | Notes |
|---|---|---|
| http_request_duration_seconds_count | Counter metric tracking request counts | Must be a counter to use increase() |
| increase() | Aggregation function to calculate counter increase over time | Essential for calculating rates from counters |
| 1h | Time window for the increase calculation | Adjust based on desired SLI window |
Note: SLOs are typically defined as a percentage of requests meeting a latency threshold. The example above calculates the percentage of requests that exceed a certain duration.
Example: SLO Query with Latency Threshold
100 - (sum(increase(http_request_duration_seconds_bucket{job="api",le="0.5"}[1h])) by (job) / sum(increase(http_request_duration_seconds_count{job="api"}[1h])) by (job) * 100)SLO Query Components
- Counter Metrics: Use
increase()to calculate counter deltas - Time Windows: Define the period over which to measure (e.g.,
1h) - Latency Buckets: Use
http_request_duration_seconds_bucketfor percentile calculations - Percentage Calculation: Apply basic math to convert counts into percentages
9. Using Quantile Functions
| Function | Description | Example | Notes |
|---|---|---|---|
| quantile() | Calculates the q-th quantile of values for a given metric | quantile(0.9, http_request_duration_seconds) |
q is between 0 and 1; 0.9 = 90th percentile |
| quantile_over_time() | Calculates the q-th quantile over a time range | quantile_over_time(0.95, http_request_duration_seconds[5m]) |
Useful for dynamic quantiles over time windows |
| histogram_quantile() | Calculates quantiles from histogram data | histogram_quantile(0.9, http_request_duration_seconds_bucket) |
Requires histogram bucket data; more accurate than quantile() |
Example: Calculating Percentiles
# 95th percentile of request duration over last hour
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[1h]))Note: Quantile functions are useful for performance analysis and SLO monitoring.
Warning:
quantile() and quantile_over_time() can be computationally expensive on large datasets.10. Calculating Saturation Metrics
| Metric Type | Description | Formula | Notes |
|---|---|---|---|
| cpu_saturation | Percentage of CPU usage relative to capacity | 100 * (1 - (node_cpu_seconds_total{mode="idle"} / node_cpu_seconds_total{mode="idle"} + node_cpu_seconds_total{mode="iowait"})) |
Use with rate() for accurate calculation over time |
| memory_saturation | Memory usage as a percentage of total available | 100 * (node_memory_bytes_used / node_memory_bytes_total) |
May require adjustment for buffers/cache |
| disk_saturation | Disk I/O wait time as a percentage of total time | 100 * (rate(node_disk_io_time_seconds_total[5m]) / 5) |
Use rate() to normalize over time windows |
Example: Calculating CPU Saturation
100 * (1 - (
rate(node_cpu_seconds_total{mode="idle"}[5m]) /
rate(node_cpu_seconds_total{mode="idle"}[5m] + node_cpu_seconds_total{mode="iowait"}[5m])
))Note: Saturation metrics help identify resource bottlenecks in system performance.
Warning: Ensure that
rate() intervals are appropriate for your data retention and scrape frequency.11. Implementing Cardinality Queries
Cardinality queries in Prometheus help you understand the number of unique time series for a given metric or label combination. High cardinality can impact performance and storage.
| Query | Description | Use Case |
|---|---|---|
| count by (job) (up) | Counts the number of time series grouped by job | Monitor how many instances are reporting for each job |
| count by (instance) (up) | Counts the number of time series grouped by instance | Identify how many instances are active in a scrape target |
| count by (job, instance) (up) | Counts the number of time series grouped by job and instance | Monitor unique combinations of job and instance |
Example: Identifying High Cardinality Metrics
count by (job, instance) (up)
This query helps identify how many unique instances are reporting for each job, which is useful in identifying high cardinality issues.
Example: Monitoring Label Values
count by (job) (up)
Use this to monitor the number of jobs reporting to Prometheus. If a job has too many instances, it may indicate a cardinality problem.
12. Using Label Functions for Grouping
| Function | Description | Example | Notes |
|---|---|---|---|
| label_replace() | Replaces a label value with a regex capture group | label_replace(up{job="prometheus"}, "new_label", "$1", "instance", "(.*):.*") |
Useful for extracting parts of label values |
| label_join() | Joins multiple label values into one | label_join(up{job="prometheus"}, "combined", ":", "job", "instance") |
Can be used to create composite keys |
| label_set() | Sets a new label or overrides an existing one | label_set(up{job="prometheus"}, "env", "prod") |
Useful for adding context to metrics |
| label_map() | Maps label values using a function | label_map(up{job="prometheus"}, "status", "up", "instance") |
Allows transformation of label values |
Example: Using label_replace()
label_replace(up{job="prometheus"}, "new_label", "$1", "instance", "(.*):.*")
Note: Label functions are powerful for transforming and enriching metrics data in queries.