Writing PromQL Queries
1. Using PromQL Selectors (metric, labels)
| Selector Type | Description | Example |
|---|---|---|
| Metric Name | Matches all time series with the given name | up |
| Label Matchers | Filters time series by label values | up{job="prometheus"} |
| Label Match Operators | Defines how labels are matched | =, !=, =~, !~ |
Note: Label matchers are case-sensitive and support regex patterns using
=~ and !~.
2. Using Range Vectors ([5m], [1h])
| Range Vector Selector | Description | Examples | Notes |
|---|---|---|---|
| [5m] | Time range of 5 minutes | rate(cpu_usage[5m]) |
Used to calculate rates over a 5-minute window |
| [1h] | Time range of 1 hour | increase(memory_used[1h]) |
Useful for calculating increases over an hour |
| [1d] | Time range of 1 day | avg_over_time(http_requests[1d]) |
Helps in analyzing daily averages |
| [30s] | Time range of 30 seconds | max_over_time(cpu_load[30s]) |
Good for high-frequency data sampling |
Note: Range vectors must be used with functions that support time ranges, such as
rate(), increase(), and avg_over_time().
3. Using Instant Vectors (current values)
| Property | Description | Example |
|---|---|---|
| metric_name | Instant vector representing current values of a metric | up |
| metric_name{label="value"} | Instant vector with label filters | http_requests_total{method="GET"} |
| metric_name{label!="value"} | Instant vector excluding specific label values | http_requests_total{method!="POST"} |
Note: Instant vectors return a set of time series with the same timestamp, representing current values at that instant.
4. Applying Aggregation Operators
| Operator | Description | Example | Notes |
|---|---|---|---|
| sum | Sums all values in a group | sum by (instance) (node_cpu_seconds_total) |
Groups by instance and sums CPU seconds |
| avg | Averages all values in a group | avg by (job) (http_requests_total) |
Groups by job and averages requests |
| max | Reports the maximum value in a group | max by (host) (memory_usage_bytes) |
Groups by host and finds max memory usage |
| min | Reports the minimum value in a group | min by (service) (response_time_seconds) |
Groups by service and finds min response time |
| count | Counts the number of values in a group | count by (status) (http_status_code) |
Groups by status and counts occurrences |
| stddev | Calculates standard deviation in a group | stddev by (instance) (disk_io_time_seconds) |
Groups by instance and computes stddev |
| quantile | Calculates a quantile (e.g., 90th percentile) in a group | quantile(0.9, by (job) (http_response_time_seconds)) |
Groups by job and computes 90th percentile response time |
Note: Aggregation operators can be combined with
by, without, or group_left to control grouping behavior.
5. Using rate() Function (counter metrics)
| Property | Description | Example |
|---|---|---|
| rate(metric[5m]) | Calculates the per-second rate of increase for a counter metric over a 5-minute window | rate(http_requests_total[5m]) |
| rate(metric[1h]) | Calculates the per-second rate over a 1-hour window | rate(cpu_seconds_total[1h]) |
Example: Calculating Request Rate
rate(http_requests_total[5m])This query returns the per-second rate of HTTP requests over the last 5 minutes.
Note: The
rate() function is designed specifically for counter metrics and handles resets automatically.
Warning: Using
rate() on non-counter metrics can produce unexpected results.
6. Using increase() Function (total increase)
| Function | Description | Usage | Notes |
|---|---|---|---|
| increase() | Calculates the total increase of a metric over a given time range | increase(metric_name[5m]) |
Useful for calculating total increases over time windows |
Note: The
increase() function is particularly useful for counter metrics where you want to know the total increase over a specific time window.
7. Using histogram_quantile() (percentile calculations)
| Function | Signature | Description | Notes |
|---|---|---|---|
| histogram_quantile() | histogram_quantile(quantile, metric) |
Calculates the value at a given quantile from histogram data | Quantile must be between 0 and 1. Commonly used for latency percentiles like p95, p99 |
Example: Calculating 95th percentile latency
histogram_quantile(0.95, http_request_duration_seconds_bucket)Example: Using with labels
histogram_quantile(0.99, http_request_duration_seconds_bucket{method="GET"})
Note: The
histogram_quantile function assumes that the metric name ends with _bucket and uses the standard Prometheus histogram structure.
Warning: If the histogram data is not properly aligned or missing buckets,
histogram_quantile may return inaccurate results or NaN.
| Parameter | Type | Description | Valid Range |
|---|---|---|---|
| quantile | float | The desired quantile (e.g., 0.95 for 95th percentile) | 0 <= quantile <= 1 |
| metric | histogram vector | The histogram metric (must end in _bucket) |
Must be a histogram-type metric |
Key Points
- histogram_quantile() is used to calculate percentiles from histogram data
- Quantile values must be between 0 and 1 (e.g., 0.95 = 95th percentile)
- Works with standard Prometheus histogram metrics (ending in
_bucket) - Useful for monitoring latency and performance SLAs
8. Using Label Matching (=, !=, =~, !~)
| Operator | Description | Example | Notes |
|---|---|---|---|
| = | Matches labels with exact value | up{job="prometheus"} == 1 |
Standard equality match |
| != | Matches labels with different value | up{job!="prometheus"} == 1 |
Standard inequality match |
| =~ | Matches labels with regex pattern | up{job=~"prom.*"} == 1 |
Regex match (case-sensitive) |
| !~ | Matches labels that do not match regex pattern | up{job!~"prom.*"} == 1 |
Regex non-match (case-sensitive) |
9. Using Binary Operators (+, -, *, /)
| Operator | Description | Example | Notes |
|---|---|---|---|
| + | Addition operator | rate(http_requests_total[5m]) + rate(http_errors_total[5m]) |
Matches series with same labels and adds values |
| - | Subtraction operator | http_requests_total - http_errors_total |
Matches series with same labels and subtracts values |
| * | Multiplication operator | http_requests_total * 1000 |
Multiplies each value by scalar |
| / | Division operator | http_requests_total / http_errors_total |
Matches series with same labels and divides values |
Note: Binary operators can be used with scalar values, which will be applied to all matching time series.
10. Using Metric Browser (autocomplete)
| Feature | Description | Usage |
|---|---|---|
| Metric Name | Autocomplete for metric names | Type metric name or use dropdown to select |
| Label Matching | Filter metrics by labels | Use {label="value"} syntax |
| Function Suggestions | Autocomplete for PromQL functions | Type function name to see suggestions |
| Range Vector | Specify time range for metrics | Add [5m], [1h] etc. |
Note: The metric browser provides real-time suggestions as you type, helping to build complex queries quickly.
11. Using Recording Rules (pre-computed queries)
Example: Define a recording rule
groups:
- name: example-recording-rules
rules:
- record: instance:node_cpu_seconds:rate1m
expr: rate(node_cpu_seconds_total[1m])
| Property | Description | Values |
|---|---|---|
| name | Name of the rule group | example-recording-rules |
| rules | List of recording rules to define | Array of rule objects |
| record | Name of the metric to be created | instance:node_cpu_seconds:rate1m |
| expr | PromQL expression to compute the metric | rate(node_cpu_seconds_total[1m]) |
Note: Recording rules are evaluated at the same interval as the scrape interval, and their results can be used in dashboards or other queries.
12. Using Exemplars (link to traces)
| Property | Description | Usage |
|---|---|---|
| exemplar | Marker indicating a sample has an associated trace ID | Used in PromQL to link metrics to trace data via trace_id |
Note: Exemplars are only available when the underlying data source supports them, such as Prometheus with
-enable-feature=exemplar-storage.