Querying with PromQL Selectors
1. Selecting Metrics by Name
| Selector | Description | Example |
|---|---|---|
| metric_name | Matches all time series with the given metric name | up |
| metric_name{label="value"} | Matches time series with the given metric name and label value | http_requests_total{method="GET"} |
| metric_name{label!="value"} | Matches time series with the given metric name and label value not equal to specified | http_requests_total{method!="POST"} |
| metric_name{label=~"regex"} | Matches time series with the given metric name and label matching regex | http_requests_total{method=~"GET|POST"} |
| metric_name{label!~"regex"} | Matches time series with the given metric name and label not matching regex | http_requests_total{method!~"PUT|DELETE"} |
Example: Selecting a metric with label filters
http_requests_total{method="GET", handler="/api/users"}
2. Using Label Matchers
| Matcher Type | Description | Examples | Notes |
|---|---|---|---|
| = | Exact match | job="prometheus" |
Matches series where the label value exactly matches |
| != | Not equal | job!="prometheus" |
Matches series where the label value does not match |
| =~ | Regular expression match | job=~"prometheus|alertmanager" |
Matches series where the label value matches the regex |
| !~ | Not regex match | job!~"prometheus.*" |
Matches series where the label value does not match the regex |
Note: Label matchers can be combined with other selectors to form complex queries.
3. Selecting Instant Vectors
| Selector Type | Description | Example | Notes |
|---|---|---|---|
| metric_name | Selects all time series with the given metric name | up |
Matches all time series with the metric name up |
| metric_name{label_name="label_value"} | Selects time series with the given metric name and matching label value | http_requests_total{method="GET"} |
Matches all http_requests_total series with the label method="GET" |
| metric_name{label_name!="label_value"} | Selects time series with the given metric name and non-matching label value | http_requests_total{method!="POST"} |
Matches all http_requests_total series where the label method is not POST |
| metric_name{label_name=~"regex"} | Selects time series with the given metric name and matching label value using regex | http_requests_total{method=~"GET|POST"} |
Matches all http_requests_total series where the label method matches either GET or POST |
| metric_name{label_name!~"regex"} | Selects time series with the given metric name and non-matching label value using regex | http_requests_total{method!~"PUT|DELETE"} |
Matches all http_requests_total series where the label method does not match either PUT or DELETE |
Note: Label matching in PromQL is case-sensitive. If you're unsure about label values, use
label_values() to inspect them.
4. Selecting Range Vectors
| Selector | Description | Example | Notes |
|---|---|---|---|
| metric_name | Selects all time series with the given metric name | up |
Matches all time series with the metric name up |
| metric_name{label="value"} | Selects time series with the given metric name and matching label value | http_requests_total{method="GET"} |
Matches all http_requests_total series with the label method set to GET |
| metric_name{label!="value"} | Selects time series with the given metric name and non-matching label value | http_requests_total{method!="POST"} |
Matches all http_requests_total series where the label method is not set to POST |
| metric_name{label=~"regex"} | Selects time series with the given metric name and matching label regex | http_requests_total{method=~"GET|POST"} |
Matches all http_requests_total series where the label method matches either GET or POST |
| metric_name{label!~"regex"} | Selects time series with the given metric name and non-matching label regex | http_requests_total{method!~"PUT|DELETE"} |
Matches all http_requests_total series where the label method does not match either PUT or DELETE |
| metric_name{label="value", label2=~"regex"} | Selects time series with the given metric name and multiple label conditions | http_requests_total{method="GET", handler=~"/api/.*"} |
Matches all http_requests_total series where the label method is GET and the label handler matches the regex pattern /api/.* |
Example: Selecting a Range Vector with Time Range
up[5m]
Fetches the up metric for the last 5 minutes.
Example: Selecting a Range Vector with Offset
http_requests_total{method="GET"} offset 5m
Selects the http_requests_total metric with method=GET from 5 minutes ago.
Example: Selecting a Range Vector with Step
rate(http_requests_total[5m])[1m:10s]
Calculates the rate of http_requests_total over 5 minutes, then samples it every 10 seconds with a step of 1 minute.
Note: The
[5m] syntax selects a range vector over the last 5 minutes. It is used in combination with functions like rate(), increase(), and others to compute metrics over a time window.
5. Using Offset Modifier
The offset modifier in PromQL allows you to retrieve historical metric values from a specified time duration ago. It is particularly useful when you want to compare current data with past data, or when dealing with metrics that have a delay in reporting.
| Modifier | Description | Example |
|---|---|---|
| offset | Shifts the time window of a query by a specified duration. | rate(http_requests_total[5m]) offset 1h |
Example: Comparing Current and Historical Data
rate(http_requests_total[5m])
-
rate(http_requests_total[5m] offset 1h)This query calculates the difference between the current rate of HTTP requests and the rate from one hour ago, useful for detecting trends or anomalies.
Note: The
offset modifier does not change the evaluation time of the query; it only shifts the data points retrieved from the time series.
6. Implementing @ Modifier
The @ modifier in PromQL allows you to query the value of a metric at a specific timestamp. It is particularly useful for historical analysis or when you want to compare the current value of a metric with its value at a past point in time.
| Modifier | Description | Example |
|---|---|---|
| @ | Queries the value of a metric at a specific timestamp. | http_requests_total @ 1609459200 |
Example: Using @ Modifier with a Timestamp
http_requests_total @ 1609459200This query returns the value of http_requests_total at the Unix timestamp 1609459200 (which corresponds to 2021-01-01 00:00:00 UTC).
Example: Using @ Modifier with Time Range
http_requests_total @ (time() - 3600)This query returns the value of http_requests_total one hour ago from the current time.
Note: The
@ modifier can be used with any metric and supports all PromQL functions. It is particularly useful in combination with rate() or other time-series functions to analyze historical trends.
7. Understanding Time Durations
| Unit | Description | Examples |
|---|---|---|
| s | Seconds | 30s, 1m |
| m | Minutes | 5m, 1h |
| h | Hours | 2h, 1d |
| d | Days | 7d, 30d |
| w | Weeks | 1w, 2w |
| y | Years | 1y, 2y |
Note: Time durations in PromQL are used to define the window over which metrics are aggregated or sampled.
8. Combining Multiple Matchers
| Matcher | Description | Example |
|---|---|---|
| label=value | Matches metrics with a specific label value | up{job="prometheus"} |
| label!=value | Matches metrics where label does not equal value | up{job!="prometheus"} |
| label=~regex | Matches metrics where label matches regex pattern | up{job=~"prom.*"} |
| label!~regex | Matches metrics where label does not match regex pattern | up{job!~"prom.*"} |
Note: Matchers are combined with AND logic. All conditions must be true for a metric to be selected.
9. Using Regex Matchers
Regex matchers in PromQL allow you to filter metrics based on pattern matching. They are particularly useful for selecting metrics with similar naming conventions.
| Matcher | Description | Example |
|---|---|---|
| =~ | Matches metrics where the label value matches the provided regex pattern. | up{job=~"prometheus|alertmanager"} |
| !~ | Matches metrics where the label value does NOT match the provided regex pattern. | up{job!~"prometheus.*"} |
Example: Selecting Metrics with Regex
up{job=~"prometheus.*"}
This query selects all metrics with the label job that starts with prometheus.
Example: Excluding Metrics with Regex
up{job!~"alertmanager.*"}
This query selects all metrics with the label job that do not start with alertmanager.
Note: Regex patterns in PromQL are anchored at the beginning and end, meaning
prometheus will match prometheus at the start of a string but not in the middle.
10. Selecting All Series
| Selector | Description | Example | Notes |
|---|---|---|---|
| metric_name | Selects all time series with the given metric name | up |
Matches all time series with the metric name up |
| {} | Selects all time series (empty label matcher) | {} |
Matches all time series in the system |
| metric_name{} | Selects all time series with the given metric name and no labels | up{} |
Matches all time series with the metric name up and no labels |
11. Understanding Vector Types
| Type | Description | Example |
|---|---|---|
| instant vector | A set of time series at a single point in time | http_requests_total |
| range vector | A set of time series containing a range of data points over time | http_requests_total[5m] |
12. Using Negative Offset
Negative offsets in PromQL allow you to query data from the past relative to the current evaluation time. This is particularly useful for comparing current values with historical data.
| Offset Expression | Description | Example |
|---|---|---|
| metric_name[5m] | Query metric from 5 minutes ago | up[5m] |
| metric_name[1h] | Query metric from 1 hour ago | http_requests_total[1h] |
| metric_name[1d] | Query metric from 1 day ago | node_cpu_seconds_total[1d] |
Example: Comparing Current and Historical Data
rate(http_requests_total[5m])
-
rate(http_requests_total[5m] offset 1h)
Note: The
offset operator can be used with any time range selector and supports durations like 5m, 1h, or 1d.
Warning: When using negative offsets, ensure that the time range selector includes enough data points to avoid gaps in your results.