Using PromQL Over-Time Functions

1. Calculating Average Over Time

Function	Description	Example	Notes
avg_over_time()	Calculates the average of values for each time series over a specified time range.	`avg_over_time(metric[5m])`	Useful for smoothing data over time windows.
avg_range()	Calculates the average over a range vector with a specified time window.	`avg_range(metric[1h])`	Alternative to `avg_over_time`, often used in queries with range vectors.

Example: Using avg_over_time to calculate average CPU usage over time

avg_over_time(node_cpu_seconds_total[5m])

2. Finding Min and Max Over Time

Function	Description	Example	Notes
min_over_time()	Returns the minimum value of a metric over a time range.	`min_over_time(http_requests_total[5m])`	Useful for identifying lowest values in a time window.
max_over_time()	Returns the maximum value of a metric over a time range.	`max_over_time(http_requests_total[5m])`	Useful for identifying peak values in a time window.
avg_over_time()	Returns the average value of a metric over a time range.	`avg_over_time(http_requests_total[5m])`	Useful for smoothing out fluctuations in metrics.
stddev_over_time()	Returns the standard deviation of a metric over a time range.	`stddev_over_time(http_requests_total[5m])`	Useful for measuring variability in metric values.
stdvar_over_time()	Returns the variance of a metric over a time range.	`stdvar_over_time(http_requests_total[5m])`	Alternative to standard deviation for measuring spread.

Note: These functions are useful for aggregating time-series data to understand trends and anomalies over a specified duration.

3. Computing Sum Over Time

Function	Description	Signature	Example
sum_over_time()	Calculates the sum of values within a given time range	`sum_over_time(metric[5m])`	Example: Sum over 5-minute window `sum_over_time(http_requests_total[5m])`
sum_rate()	Calculates the sum of the per-second rate of increase over a time range	`sum_rate(metric[5m])`	Example: Rate sum over 5-minute window `sum_rate(http_requests_total[5m])`

Note: sum_over_time includes all values in the window, while sum_rate accounts for rate changes over time.

4. Counting Samples Over Time

Function	Description	Syntax	Notes
count_over_time()	Counts the number of samples in the specified time range.	`count_over_time(metric[time_range])`	Useful for determining how many data points exist in a given window.
count_values()	Counts the number of distinct values in a vector.	`count_values(label_name, vector)`	Can be used to understand value distribution over time.
quantile_over_time()	Calculates the quantile of values over time.	`quantile_over_time(quantile, metric[time_range])`	Useful for performance analysis and latency trends.

Example: Using count_over_time()

count_over_time(http_requests_total[5m])

Example: Using count_values()

count_values("status", http_requests_total)

Note: count_over_time is particularly useful for identifying gaps in data or ensuring sufficient sampling frequency.

5. Using Quantile Over Time

The quantile_over_time function in PromQL calculates the quantile of a time series over a specified range. It is particularly useful for analyzing performance metrics or latency distributions.

Function	Parameters	Description	Example
quantile_over_time	`(quantile, range_vector)`	Calculates the quantile over a time range	`quantile_over_time(0.95, http_request_duration_seconds[5m])`

Example: Calculating 95th Percentile Over Time

quantile_over_time(0.95, http_request_duration_seconds[5m])

This query returns the 95th percentile of HTTP request duration over the last 5 minutes.

Note: The quantile_over_time function is useful for identifying performance bottlenecks and understanding latency distributions over time.

Warning: Ensure that the range vector is sufficiently long to provide meaningful quantile calculations. Short ranges may lead to inaccurate results.

6. Computing Standard Deviation Over Time

Function	Description	Example	Notes
stddev_over_time()	Calculates the standard deviation of values for each time series over a specified time range.	`stddev_over_time(http_requests_total[5m])`	Useful for identifying variability in metrics over time. Returns a scalar or vector of standard deviations.
stdvar_over_time()	Calculates the variance of values for each time series over a specified time range.	`stdvar_over_time(http_requests_total[5m])`	Similar to `stddev_over_time`, but returns the square of the standard deviation (variance).

Note: These functions are useful for anomaly detection and performance analysis. They aggregate over the specified range and return a single value per time series.

7. Finding Last Value Over Time

Function	Description	Example	Notes
last_over_time()	Returns the last value of a time series over a given range.	`last_over_time(http_requests_total[5m])`	Useful for getting the most recent value in a window.
max_over_time()	Returns the maximum value of a time series over a given range.	`max_over_time(http_requests_total[1h])`	Can be used to find peak values over a time window.
min_over_time()	Returns the minimum value of a time series over a given range.	`min_over_time(http_requests_total[1h])`	Useful for identifying lowest values in a time window.

Example: Finding Last Value Over Time

last_over_time(http_requests_total[5m])

This query returns the last recorded value of http_requests_total over the last 5 minutes.

Note: The last_over_time function is particularly useful for alerting when a metric stops updating or changes unexpectedly.

8. Finding First Value Over Time

Function	Description	Example	Notes
first_over_time()	Returns the first value in the specified time range for each time series.	`first_over_time(http_requests_total[5m])`	Useful for capturing initial values before a metric changes.

Example: Using first_over_time()

first_over_time(prometheus_http_requests_total[1h])

Note: The function evaluates the metric over a specified range and returns the first value observed in that window.

9. Using Absent Over Time

The absent_over_time() function in PromQL is used to detect when a time series is missing or absent over a specified time range. It returns a vector with the same labels as the input series, and a value of 1 if the series was absent during the time range, or 0 if it was present.

Function	Signature	Description
absent_over_time()	`absent_over_time(range_vector)`	Returns 1 if the series is absent over the specified time range, otherwise 0.

Example: Detecting Missing Metrics

absent_over_time(http_requests_total[5m])

This query checks if the http_requests_total metric is missing over a 5-minute window.

Note: The function returns a vector with a value of 1 for any series that was absent during the specified time range. If no data is found, it returns 0.

10. Calculating Changes Over Time

Function	Description	Example	Notes
increase()	Returns the increase in value over time for a counter metric	`increase(http_requests_total[5m])`	Useful for calculating total requests over a time window
rate()	Calculates the per-second average rate of increase for a counter	`rate(http_requests_total[5m])`	Accounts for counter resets and provides rate per second
delta()	Returns the difference between the last and first value in a range vector	`delta(cpu_usage_seconds_total[1h])`	Does not account for counter resets, use with caution
deriv()	Calculates the derivative of a metric over time	`deriv(cpu_usage_seconds_total[1h])`	Useful for identifying trends in continuous metrics

Example: Calculating Request Rate

rate(http_requests_total[1m])

This query returns the per-second rate of HTTP requests over the last minute.

Example: Tracking Counter Resets

increase(http_requests_total[5m])

This query sums up all increases in the counter over a 5-minute window, handling resets correctly.

11. Detecting Resets Over Time

When working with Prometheus counters, it's crucial to detect resets in counter values over time. Counter resets can occur due to metric reinitialization or system restarts, and they can skew calculations if not handled properly.

Function	Description	Usage
delta()	Returns the delta between the last two samples of a counter, including resets.	`delta(my_counter[5m])`
increase()	Calculates the increase in a counter over time, handling resets automatically.	`increase(my_counter[5m])`
rate()	Calculates the per-second rate of increase for a counter, handling resets.	`rate(my_counter[5m])`
resets()	Counts the number of resets in a counter over time.	`resets(my_counter[5m])`

Example: Using increase() to handle counter resets

increase(http_requests_total[1h])

This query returns the total increase in HTTP requests over the last hour, automatically accounting for any resets in the counter.

Example: Detecting resets with resets()

resets(http_requests_total[1h])

This query returns the number of times the http_requests_total counter has reset over the last hour.

Note: The increase() and rate() functions are preferred over delta() for counters because they handle resets gracefully, ensuring accurate calculations even when a counter resets.

12. Computing Derivative Over Time

Function	Description	Example	Notes
deriv()	Calculates the per-second derivative of the input time series, using linear regression over the last two data points.	`deriv(http_requests_total[5m])`	Useful for computing rates of change over time. Returns a rate per second.
irate()	Calculates the per-second derivative of the input time series, using only the last two data points.	`irate(http_requests_total[5m])`	More sensitive to spikes than `rate()`. Ideal for short-term rate calculations.
rate()	Calculates the per-second average rate of increase of the input time series over a window.	`rate(http_requests_total[5m])`	Smoothes out spikes and is more stable than `irate()`. Best for long-term trends.

Note: deriv() is less commonly used than rate() or irate(), but it can be useful for detecting anomalies in the rate of change over time.