Working with Set Operators
1. Using Set Operators
Set operators in Prometheus allow you to perform set operations on metric series. These operators are useful for filtering, combining, and comparing time series data.
| Operator | Description | Example |
|---|---|---|
| group_left | Matches series from the left with one or more series from the right | rate(http_requests_total[5m]) by (method) group_left(handler) http_requests_total |
| group_right | Matches series from the right with one or more series from the left | rate(http_requests_total[5m]) by (method) group_right(handler) http_requests_total |
| ignoring | Matches series ignoring specified labels | rate(http_requests_total[5m]) by (method) ignoring(handler) http_requests_total |
| on | Matches series on specified labels only | rate(http_requests_total[5m]) by (method) on(handler) http_requests_total |
Example: Using group_left
rate(http_requests_total[5m]) by (method) group_left(handler) http_requests_total
2. Implementing Intersection
Intersection in Prometheus is achieved through matching labels between two series. The on() operator allows you to specify which labels should be matched for intersection.
| Operator | Description | Example |
|---|---|---|
| on() | Matches series on specified labels | a on(label) b |
| ignoring() | Matches series ignoring specified labels | a ignoring(label) b |
3. Using Union
The union operator in Prometheus combines multiple series into a single result set. This is useful for aggregating data from different sources or metrics.
| Operator | Description | Example |
|---|---|---|
| union | Merges multiple series into one | rate(http_requests_total[5m]) union rate(node_cpu_seconds_total[5m]) |
4. Implementing Difference
Difference in Prometheus is implemented using the unless operator. This operator excludes matching series from a query.
| Operator | Description | Example |
|---|---|---|
| unless | Excludes matching series from a query | a unless b |
5. Combining with Vector Matching
Vector matching allows you to combine metrics based on their labels. When using set operators, vector matching ensures that only compatible series are matched.
| Operator | Description | Example |
|---|---|---|
| group_left | Matches left series with right series, preserving labels from the left | a group_left(b) b |
| group_right | Matches right series with left series, preserving labels from the right | a group_right(b) b |
Example: Vector Matching with group_left
rate(http_requests_total[5m]) by (method) group_left(handler) http_requests_total
6. Using Set Operators for Filtering
Set operators can be used to filter metrics based on specific conditions. For example, you might want to filter out certain instances or jobs from your metrics.
| Operator | Description | Example |
|---|---|---|
| unless | Filters out matching series | a unless b |
| on() | Filters based on matching labels | a on(label) b |
7. Implementing Alert Correlation
Set operators are useful for correlating alerts. By combining metrics using set operations, you can create more complex alert conditions that depend on multiple factors.
| Operator | Description | Example |
|---|---|---|
| group_left | Matches left series with right series, preserving labels from the left | a group_left(b) b |
| group_right | Matches right series with left series, preserving labels from the right | a group_right(b) b |
Example: Alert Correlation using group_left
http_requests_total{method="GET"} group_left(handler) http_requests_total
8. Using Unless for Exclusion
The unless operator is used to exclude matching series from a query. This is particularly useful for filtering out unwanted data or creating exclusionary conditions.
| Operator | Description | Example |
|---|---|---|
| unless | Excludes matching series from a query | a unless b |
9. Combining Multiple Conditions
Multiple set operations can be combined to create complex filtering and matching logic. This allows for sophisticated querying of metrics data.
| Operator | Description | Example |
|---|---|---|
| group_left | Matches left series with right series, preserving labels from the left | a group_left(b) b |
| group_right | Matches right series with left series, preserving labels from the right | a group_right(b) b |
| unless | Excludes matching series from a query | a unless b |
Example: Combining Conditions
up{job="prometheus"} group_left(instance) up{job="node_exporter"} unless up{job="alertmanager"}
10. Optimizing Set Operations
To optimize set operations in Prometheus, ensure that queries are efficient and avoid unnecessary label matching. Use appropriate grouping and filtering to reduce the number of series processed.
| Technique | Description | Example |
|---|---|---|
| Label Filtering | Filter by labels to reduce the number of series | a{job="prometheus"} unless b{job="node_exporter"} |
| Grouping | Use grouping to match only necessary labels | a group_left(b) b |
11. Implementing Series Presence Checks
Series presence checks can be implemented using set operators to verify whether specific metrics or labels exist in the data.
| Operator | Description | Example |
|---|---|---|
| on() | Matches series on specified labels | a on(label) b |
| unless | Excludes matching series from a query | a unless b |
12. Using Vector Matching with Sets
Vector matching with sets allows you to combine metrics in a way that preserves label information and enables complex data transformations.
| Operator | Description | Example |
|---|---|---|
| group_left | Matches left series with right series, preserving labels from the left | a group_left(b) b |
| group_right | Matches right series with left series, preserving labels from the right | a group_right(b) b |