Working with PromQL Subqueries
1. Understanding Subquery Syntax
PromQL subqueries allow you to perform time-based operations on time series data, enabling complex aggregations and transformations over specific time windows.
| Component | Description | Example |
|---|---|---|
| subquery | Encapsulates a query with a time range and step | rate(http_requests_total[5m]) |
| range | Defines the time window over which to evaluate the query | [1h] |
| step | Interval between samples in the result | 1m |
@ operator can be used to evaluate a subquery at a specific timestamp.
2. Implementing Nested Aggregations
Nested aggregations in PromQL allow you to perform complex calculations by nesting aggregation functions inside one another. This is particularly useful for deriving insights from time-series data.
| Function | Description | Example |
|---|---|---|
| sum by () | Aggregates values by a specified label set. | sum by (job) (node_cpu_seconds_total) |
| avg by () | Calculates the average of values grouped by a label set. | avg by (job) (node_cpu_seconds_total) |
| max by () | Finds the maximum value within a group. | max by (job) (node_cpu_seconds_total) |
| min by () | Finds the minimum value within a group. | min by (job) (node_cpu_seconds_total) |
| count by () | Counts the number of values in a group. | count by (job) (node_cpu_seconds_total) |
| stddev by () | Calculates the standard deviation of values in a group. | stddev by (job) (node_cpu_seconds_total) |
| quantile by () | Calculates a quantile (e.g., 0.95) of values in a group. | quantile by (job) (0.95, node_cpu_seconds_total) |
Example: Nested Aggregation with Rate
Use rate() inside an aggregation to compute the average rate per job:
avg by (job) (rate(node_cpu_seconds_total[5m]))Example: Aggregating Over Time and Labels
Compute the sum of CPU usage across all jobs, grouped by instance:
sum by (instance) (node_cpu_seconds_total)3. Using Subqueries for Smoothing
Subqueries in PromQL can be used to smooth time series data by aggregating over a time window. This is particularly useful for reducing noise in metrics and making trends more visible.
| Operator | Description | Example |
|---|---|---|
| [<range>] | Applies a range vector selector to the result of a subquery | rate(http_requests_total[5m])[10m:1m] |
Example: Smoothing with rate() and window
rate(http_requests_total[5m])[10m:1m]This expression calculates the rate of HTTP requests over a 5-minute window, then aggregates it over a 10-minute window with 1-minute resolution.
Example: Using subqueries for trend analysis
avg_over_time(http_response_time_seconds[1h])[2h:30m]This expression computes the average response time over an hour, then averages it again over a 2-hour window with 30-minute resolution to smooth trends.
4. Calculating Moving Averages
Moving averages are a common way to smooth out fluctuations in time series data, making trends easier to identify. In PromQL, you can calculate moving averages using the rate() function in combination with sum_over_time() or avg_over_time().
| Function | Description | Example |
|---|---|---|
| rate() | Returns the per-second average rate of increase of a counter over a given time window. | rate(http_requests_total[5m]) |
| sum_over_time() | Returns the sum of values for a given time window. | sum_over_time(http_requests_total[1h]) |
| avg_over_time() | Returns the average value for a given time window. | avg_over_time(http_requests_total[1h]) |
5. Implementing Derivative Calculations
Derivative calculations in PromQL are used to compute the rate of change of a metric over time. This is particularly useful for understanding trends and identifying anomalies in system behavior.
| Function | Description | Parameters | Example |
|---|---|---|---|
| deriv() | Calculates the derivative of a metric using linear regression. | metric |
deriv(http_requests_total) |
| idelta() | Calculates the instantaneous delta between the last two samples. | metric |
idelta(cpu_usage) |
| rate() | Calculates the per-second average rate of increase of a metric. | metric, duration |
rate(http_requests_total[5m]) |
| irate() | Calculates the per-second rate of increase based on the last two samples. | metric, duration |
irate(http_requests_total[5m]) |
Example: Derivative of Counter Metric
rate(http_requests_total[1m])This expression calculates the per-second rate of increase for the http_requests_total counter over a one-minute window.
Example: Instantaneous Delta
idelta(cpu_usage)This computes the difference between the most recent and previous sample values for cpu_usage.
rate() function is preferred over deriv() for counter metrics as it accounts for resets and provides more accurate results.
idelta() on counters can lead to incorrect results due to counter resets. It's best suited for gauge metrics.
6. Using Subqueries with Functions
| Function | Description | Usage | Notes |
|---|---|---|---|
| rate() | Calculates the per-second average rate of increase of a counter over a time window. | rate(counter{job="api"}[5m]) |
Useful for calculating rates over time ranges. Can be used with subqueries to get rates over different intervals. |
| increase() | Calculates the increase in a counter over a time window. | increase(counter{job="api"}[5m]) |
Useful for calculating total increase over a time period. Can be used with subqueries to get increases over different intervals. |
| delta() | Calculates the difference between the last and first value in a time window. | delta(gauge{job="api"}[5m]) |
Useful for calculating the change in a gauge over time. Can be used with subqueries to get changes over different intervals. |
| deriv() | Calculates the derivative of a gauge over time. | deriv(gauge{job="api"}[5m]) |
Useful for calculating the rate of change in a gauge. Can be used with subqueries to get derivatives over different intervals. |
| clamp_min() | Clamps values to a minimum value. | clamp_min(gauge{job="api"}[5m], 0) |
Useful for ensuring values do not go below a certain threshold. Can be used with subqueries to apply clamping over different intervals. |
| clamp_max() | Clamps values to a maximum value. | clamp_max(gauge{job="api"}[5m], 100) |
Useful for ensuring values do not exceed a certain threshold. Can be used with subqueries to apply clamping over different intervals. |
7. Configuring Step Duration
Step duration in PromQL subqueries determines the time interval between samples when evaluating a range vector. It affects how frequently the query is evaluated over the specified time window.
| Parameter | Description | Default Value | Notes |
|---|---|---|---|
| step | Interval between samples in the range vector | 1s |
Must be greater than 0 and less than or equal to the range |
Example: Setting Step Duration in Subquery
rate(http_requests_total[5m]){job="api"}[1h:1m]step parameter in subqueries is specified as part of the range vector expression, not as a separate configuration option.
8. Optimizing Subquery Performance
| Optimization | Description | Best Practice |
|---|---|---|
| Range Selection | Choose appropriate time ranges to avoid excessive data processing. | Use range parameter wisely to limit query scope. |
| Step Size | Smaller steps increase precision but may impact performance. | Select a step size that balances accuracy and efficiency. |
| Query Caching | Prometheus caches results for subqueries to reduce redundant computation. | Enable caching where appropriate to avoid repeated evaluations. |
| Aggregation Functions | Use aggregation functions like sum, avg to reduce data volume. |
Apply aggregations early in subqueries to minimize intermediate results. |
9. Combining Subqueries with Offset
| Operator | Description | Example | Notes |
|---|---|---|---|
| offset | Shifts the time window of a subquery by a specified duration | rate(http_requests_total[5m] offset 10m) |
Useful for comparing current data with historical data |
offset operator shifts the time window of a subquery by a specified duration. This is useful for comparing current data with historical data, such as comparing the rate of requests in the last 5 minutes with the rate from 10 minutes ago.
10. Debugging Subquery Results
Example: Using rate() with subqueries to debug
rate(http_requests_total[5m] @ end() - http_requests_total[5m] @ start())| Operator | Description | Usage |
|---|---|---|
| subquery | Wraps a metric with a time range and optional timestamp | metric_name[time_range] @ timestamp |
| @ end() | Uses the end time of the query window | metric[5m] @ end() |
| @ start() | Uses the start time of the query window | metric[5m] @ start() |
Debugging Steps
- Use
@ end()to get the latest value of a metric - Use
@ start()to get the initial value of a metric - Compare these values to understand changes over time
- Apply
rate()or other functions to calculate rates or deltas
11. Using Default Step Interval
When using subqueries in PromQL, the default step interval is automatically derived from the range and the step size of the query. This ensures that subqueries can effectively process time series data at appropriate intervals.
| Parameter | Description | Default Value |
|---|---|---|
| step | The interval at which the subquery is evaluated | Calculated from the range and query duration |
1m:10s in the subquery defines a step interval of 10 seconds, which is used to evaluate the rate function over a 1-minute window.12. Implementing Multi-Level Subqueries
Multi-level subqueries in PromQL allow you to perform complex time series operations by nesting queries. These are particularly useful for aggregating data across multiple dimensions or applying transformations at different granularities.
| Feature | Description | Example |
|---|---|---|
| Subquery Nesting | Allows nesting of subqueries to build complex queries | rate(http_requests_total[5m])[1m:1m] |
| Aggregation Levels | Supports multi-level aggregations across time windows | sum by (job) (rate(http_requests_total[5m])[1h:5m]) |
| Time Windowing | Enables specifying different time windows for inner and outer queries | max_over_time(http_requests_total[1h:5m]) |
Example: Multi-Level Aggregation
sum by (job) (
rate(http_requests_total[5m])[1h:5m]
)This query calculates the rate of HTTP requests over a 5-minute window, then aggregates it hourly using a subquery.
Example: Nested Subqueries with Histograms
histogram_quantile(0.95, sum by (le) (
rate(request_duration_seconds_bucket[1m])[5m:1m]
))This query computes the 95th percentile of request duration using histogram buckets, with nested subqueries for rate calculation and aggregation.