Using PromQL Aggregation Operators
1. Calculating Sum
| Operator | Description | Example | Notes |
|---|---|---|---|
| sum() | Calculates the sum of all values in a vector | sum(prometheus_http_requests_total) |
Useful for aggregating metrics across all instances |
| sum by (label) | Calculates the sum of all values, grouped by a specific label | sum by (method) (prometheus_http_requests_total) |
Groups metrics by method to calculate totals per method |
| sum without (label) | Calculates the sum of all values, excluding a specific label | sum without (instance) (prometheus_http_requests_total) |
Aggregates metrics across instances, excluding instance label |
Example: Sum of HTTP Requests
sum(prometheus_http_requests_total)This query returns the total number of HTTP requests across all instances and methods.
Example: Sum by Method
sum by (method) (prometheus_http_requests_total)This query groups HTTP requests by method and calculates the sum for each method.
2. Computing Average
| Operator | Description | Example | Notes |
|---|---|---|---|
| avg() | Averages values across a vector | avg(http_requests_total) |
Aggregates over all series in the vector |
| avg_over_time() | Averages values over a time range | avg_over_time(http_requests_total[5m]) |
Useful for rate calculations over time windows |
| avg by () | Averages grouped by labels | avg by (method) (http_requests_total) |
Groups results by specified label values |
3. Finding Minimum and Maximum
| Operator | Description | Example | Notes |
|---|---|---|---|
| min() | Returns the minimum value among all values in a time series | min(prometheus_http_requests_total) |
Useful for identifying lowest observed values |
| max() | Returns the maximum value among all values in a time series | max(prometheus_http_requests_total) |
Useful for identifying highest observed values |
| min_over_time() | Calculates the minimum value over a specified time range | min_over_time(prometheus_http_requests_total[5m]) |
Useful for analyzing trends over time windows |
| max_over_time() | Calculates the maximum value over a specified time range | max_over_time(prometheus_http_requests_total[5m]) |
Useful for analyzing peak values over time windows |
Example: Using min() and max() with grouping
min by (instance) (prometheus_http_requests_total)
max by (instance) (prometheus_http_requests_total)
Note: These aggregation operators can be combined with
by, without, and other grouping modifiers to control how results are grouped.
4. Counting Series
| Operator | Description | Example | Notes |
|---|---|---|---|
| count | Counts the number of series in a vector | count(up) |
Returns a scalar value |
| count_values | Counts the number of series with each distinct value | count_values("value", metric) |
Groups by value and returns a vector |
| count_over_time | Counts the number of samples in a range vector | count_over_time(metric[5m]) |
Useful for time-series analysis |
Example: Using count_values
count_values("status", http_status)Counts how many times each status code appears in the http_status metric.
5. Computing Standard Deviation
| Operator | Description | Example | Notes |
|---|---|---|---|
| stddev | Calculates the population standard deviation of values in a vector | stddev(http_requests_total) |
Uses the formula: √(Σ(xi - μ)² / N) |
| stddev_over_time | Calculates the population standard deviation of values over a time range | stddev_over_time(http_requests_total[5m]) |
Applies to a range vector over the specified time window |
Example: Using stddev_over_time
stddev_over_time(http_requests_total[1h])Computes the standard deviation of HTTP request counts over the last hour.
Note: Standard deviation is useful for identifying variability in metrics like request rates or response times.
6. Using Group Operator
| Operator | Description | Example |
|---|---|---|
| group | Groups metrics by a specified label and applies an aggregation function to each group. | group by (job) (rate(http_requests_total[5m])) |
Note: The
group operator is useful for aggregating metrics across different dimensions, such as job or instance.
7. Implementing topk and bottomk
The topk and bottomk aggregation operators in PromQL are used to select the top or bottom k metrics based on their values. These operators are particularly useful for identifying the highest or lowest performing metrics in a system.
| Operator | Description | Example |
|---|---|---|
| topk | Returns the k largest values from a set of time series. | topk(5, http_requests_total) |
| bottomk | Returns the k smallest values from a set of time series. | bottomk(3, http_requests_total) |
Example: Using topk to identify the highest HTTP request counts
topk(5, http_requests_total)This query returns the top 5 time series with the highest values for http_requests_total.
Example: Using bottomk to identify the lowest HTTP request counts
bottomk(3, http_requests_total)This query returns the bottom 3 time series with the lowest values for http_requests_total.
Note: The
k parameter in both operators must be a positive integer. If k is zero or negative, the operator will return an empty result.
Warning: When using
topk or bottomk, the results are not ordered by any specific metric name or label. The operator simply returns the top or bottom k values based on their numeric value.
8. Using count_values
The count_values aggregation operator in PromQL counts the number of times each value appears for a given metric and returns a time series for each unique value, with the metric name replaced by the count of occurrences.
| Operator | Description | Example |
|---|---|---|
| count_values | Counts occurrences of each value for a given metric and returns a time series for each unique value. | count_values("value", http_requests_total) |
Example: Counting HTTP Request Values
count_values("status_code", http_requests_total)This query returns a time series for each unique status code in the http_requests_total metric, showing how many times each status code occurred.
Note: The
count_values operator is useful for identifying the distribution of values in a metric, particularly when analyzing error codes or status responses.
9. Aggregating by Labels
| Operator | Description | Example | Notes |
|---|---|---|---|
| sum by (label) | Sums metrics by the specified label | sum by (job) (up) |
Groups metrics by job and sums their values |
| avg by (label) | Averages metrics by the specified label | avg by (job) (http_requests_total) |
Groups metrics by job and averages their values |
| count by (label) | Counts metrics by the specified label | count by (job) (up) |
Groups metrics by job and counts them |
| max by (label) | Finds the maximum value by the specified label | max by (job) (http_requests_total) |
Groups metrics by job and finds the maximum value |
| min by (label) | Finds the minimum value by the specified label | min by (job) (http_requests_total) |
Groups metrics by job and finds the minimum value |
| stddev by (label) | Calculates standard deviation by the specified label | stddev by (job) (http_requests_total) |
Groups metrics by job and calculates standard deviation |
| stdvar by (label) | Calculates standard variance by the specified label | stdvar by (job) (http_requests_total) |
Groups metrics by job and calculates standard variance |
| quantile by (label) | Calculates a quantile by the specified label | quantile by (job) (0.95, http_requests_total) |
Groups metrics by job and calculates 95th percentile |
10. Aggregating without Labels
| Operator | Description | Example | Notes |
|---|---|---|---|
| sum | Returns the sum of all values in the vector | sum(metric) |
Does not consider labels |
| avg | Returns the average of all values in the vector | avg(metric) |
Does not consider labels |
| min | Returns the minimum value in the vector | min(metric) |
Does not consider labels |
| max | Returns the maximum value in the vector | max(metric) |
Does not consider labels |
| count | Returns the count of all values in the vector | count(metric) |
Does not consider labels |
| stddev | Returns the standard deviation of values in the vector | stddev(metric) |
Does not consider labels |
| stdvar | Returns the variance of values in the vector | stdvar(metric) |
Does not consider labels |
| absent | Returns 1 if the vector is empty, otherwise 0 | absent(metric) |
Does not consider labels |
11. Combining Aggregation Operators
Aggregation operators in PromQL can be combined to perform complex operations on time series data. These combinations allow for more sophisticated analysis and filtering of metrics.
| Operator | Description | Example | Notes |
|---|---|---|---|
| sum by (label) | Aggregates series by a specified label | sum by (job) (node_cpu_seconds_total) |
Groups metrics by job label and sums them |
| avg without (label) | Averages series excluding a specified label | avg without (instance) (node_cpu_seconds_total) |
Excludes instance label from averaging |
| count by (label) | Counts series by a specified label | count by (job) (node_cpu_seconds_total) |
Counts metrics grouped by job label |
| max by (label) | Finds maximum value per group | max by (job) (node_cpu_seconds_total) |
Groups metrics by job and finds max |
| min by (label) | Finds minimum value per group | min by (job) (node_cpu_seconds_total) |
Groups metrics by job and finds min |
| stddev by (label) | Calculates standard deviation per group | stddev by (job) (node_cpu_seconds_total) |
Groups metrics by job and computes stddev |
| quantile by (label) | Computes quantiles per group | quantile by (job) (0.95, node_cpu_seconds_total) |
Groups metrics by job and computes 95th percentile |
Example: Combining sum and avg
sum by (job) (node_cpu_seconds_total)
avg by (job) (node_cpu_seconds_total)These two expressions can be used together to understand both total CPU usage and average CPU usage per job.
Example: Using count and max together
count by (job) (node_cpu_seconds_total)
max by (job) (node_cpu_seconds_total)This combination helps identify how many metrics exist per job and the maximum value observed for each job.
12. Understanding Aggregation Behavior
| Operator | Description | Example | Notes |
|---|---|---|---|
| sum() | Sums all values for each group | sum by (job) (node_cpu_seconds_total) |
Groups metrics by job label and sums CPU seconds |
| avg() | Averages all values for each group | avg by (job) (node_cpu_seconds_total) |
Groups metrics by job label and averages CPU seconds |
| count() | Counts the number of values in each group | count by (job) (node_cpu_seconds_total) |
Groups metrics by job label and counts entries |
| max() | Reports the maximum value in each group | max by (job) (node_cpu_seconds_total) |
Groups metrics by job label and reports max CPU seconds |
| min() | Reports the minimum value in each group | min by (job) (node_cpu_seconds_total) |
Groups metrics by job label and reports min CPU seconds |
| stddev() | Calculates standard deviation for each group | stddev by (job) (node_cpu_seconds_total) |
Groups metrics by job label and computes standard deviation |
| stdvar() | Calculates variance for each group | stdvar by (job) (node_cpu_seconds_total) |
Groups metrics by job label and computes variance |
| quantile() | Calculates a quantile (e.g., 0.9) for each group | quantile(0.9, node_cpu_seconds_total) |
Reports 90th percentile of CPU seconds |
| topk() | Selects top k values in each group | topk(3, node_cpu_seconds_total) |
Selects top 3 CPU seconds values |
| bottomk() | Selects bottom k values in each group | bottomk(3, node_cpu_seconds_total) |
Selects bottom 3 CPU seconds values |
Example: Aggregation with grouping
sum by (job, instance) (node_cpu_seconds_total)This query groups metrics by job and instance, then sums the CPU seconds for each group.
Example: Aggregation with quantile
quantile(0.95, rate(node_cpu_seconds_total[5m]))This query calculates the 95th percentile of CPU usage rate over a 5-minute window.
Note: Aggregation operators can be combined with
by and without to control grouping labels.
Warning: Using
sum() on counters can lead to incorrect results if not properly handled with rate functions.