Configuring Recording Rules
1. Creating Rule Groups
| Property | Description | Example |
|---|---|---|
| name | Name of the rule group | node_rules |
| interval | How often to evaluate rules in the group | 30s |
| rules | List of recording rules in the group | [{...}] |
Example: Basic Rule Group Configuration
groups:
- name: node_rules
interval: 30s
rules:
- record: node_cpu_seconds_total
expr: rate(node_cpu_seconds_total[5m])
2. Defining Recording Rules
| Property | Description | Example |
|---|---|---|
| name | The name of the rule | http_requests_total |
| expr | The PromQL expression to evaluate | rate(http_requests_total[5m]) |
| labels | Additional labels to add to the output series | {job="api-server"} |
Example: Basic Recording Rule
groups:
- name: http_rules
rules:
- record: http_requests_rate
expr: rate(http_requests_total[5m])
labels:
job: api-server
Note: Recording rules are evaluated at the same interval as the scrape target, and the resulting metrics are stored in the Prometheus storage.
3. Setting Evaluation Interval
| Property | Value | Description |
|---|---|---|
| evaluation_interval | 1m |
How often rules are evaluated. Default is 1m (1 minute). |
Example: Configuring Evaluation Interval
rule_files:
- "recording_rules.yml"
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
evaluation_interval: 30s
4. Adding Labels to Rules
| Label | Description | Example |
|---|---|---|
| rule_name | Unique identifier for the rule | up:1m |
| job | Job name from the scrape target | prometheus |
| instance | Instance identifier from the scrape target | localhost:9090 |
Example: Adding Labels to Recording Rule
groups:
- name: example
rules:
- record: up:1m
expr: up
labels:
job: "prometheus"
instance: "localhost:9090"5. Organizing Rule Files
Example: Rule file structure
prometheus/
├── rules/
│ ├── alerting_rules.yml
│ ├── recording_rules.yml
│ └── service_rules/
│ ├── frontend_rules.yml
│ └── backend_rules.yml
└── prometheus.yml
| File Type | Description | Best Practices |
|---|---|---|
| alerting_rules.yml | Contains alert definitions using alert blocks |
Group alerts by service or team; use descriptive names |
| recording_rules.yml | Defines recording rules using record blocks |
Organize by metric family or business logic; avoid over-recording |
| service_rules/ | Directory for service-specific rules | Use subdirectories to scope rules to specific services or teams |
Note: Rule files can be included in the main config using
rule_files directive.
Example: Including rule files in main config
rule_files:
- rules/alerting_rules.yml
- rules/recording_rules.yml
- rules/service_rules/*.yml
6. Naming Recording Rules
| Rule Name | Description | Best Practices |
|---|---|---|
| recording_rule_name | Unique identifier for the recording rule within the Prometheus configuration. | Use descriptive names that clearly indicate the metric being recorded and its purpose. Avoid generic names like rule1. |
| metric_name | The resulting metric name after applying the recording rule. | Follow Prometheus naming conventions: lowercase letters, numbers, and underscores only. Prefix with a namespace if needed (e.g., myapp_). |
Example: Naming Convention
groups:
- name: example_group
rules:
- record: myapp_http_requests_total
expr: sum by (method, handler) (http_requests_total)
Note: Rule names must be unique within a group and should reflect the purpose of the rule to ensure maintainability.
7. Using Recording Rules for Performance
| Rule Type | Description | Use Case | Performance Impact |
|---|---|---|---|
| Recording Rule | Transforms existing metrics into new time series | Reduce cardinality, pre-aggregate data | NEW Improved query performance |
| Aggregation Rule | Groups metrics by labels and applies aggregation functions | Summarize data for dashboards or alerts | Reduces data volume and query load |
Example: Recording Rule for CPU Usage
groups:
- name: cpu_recording_rules
rules:
- record: node_cpu_seconds_total:avg_rate5m
expr: avg(rate(node_cpu_seconds_total[5m])) by (cpu)
Note: Recording rules are evaluated at the same interval as the scrape, and can significantly reduce query load on Prometheus.
Warning: Avoid creating too many recording rules as they can increase memory usage and processing overhead.
Benefits of Recording Rules
- Reduce query latency by pre-computing results
- Lower cardinality for high-cardinality metrics
- Improve dashboard performance with cached aggregations
8. Implementing Multi-Level Recording
Multi-level recording rules allow you to aggregate and summarize metrics at different granularities, enabling efficient querying and reduced storage overhead. These rules can be chained to create hierarchical aggregations.
| Rule | Description | Example |
|---|---|---|
| recording_rule | Defines a new metric based on a PromQL expression | |
Example: Aggregating CPU Usage by Instance
groups:
- name: multi_level_recording
rules:
- record: instance:cpu_usage:rate1m
expr: rate(cpu_usage_seconds_total[1m])
- record: job:cpu_usage:rate1m
expr: sum by (job) (instance:cpu_usage:rate1m)
Note: The
record directive creates a new metric from the result of a PromQL expression. These rules are evaluated and stored at the configured interval.
Multi-Level Recording Benefits
- Reduces query complexity for frequently accessed aggregations
- Improves performance by pre-aggregating data
- Enables efficient storage of summarized metrics
9. Validating Rule Syntax
Example: Basic Rule Group Validation
# rules.yml
groups:
- name: example_group
rules:
- record: instance_cpu_time_seconds
expr: rate(cpu_time_seconds[5m])
| Rule Field | Description | Required | Valid Values |
|---|---|---|---|
| name | Unique identifier for the rule group | Yes | String |
| rules | List of recording or alerting rules | Yes | Array of rule objects |
| interval | How often to evaluate rules | No | Duration (e.g., 30s, 1m) |
Note: Rule names must be unique within a group and follow Prometheus naming conventions.
Example: Invalid Rule Expression
# Invalid rule - missing closing parenthesis
groups:
- name: invalid_group
rules:
- record: bad_metric
expr: sum(http_requests_total
Warning: Invalid rule syntax will cause the entire rule group to be skipped during evaluation.
| Validation Method | Description | Tool |
|---|---|---|
| promtool check rules | Command-line tool to validate rule files | promtool |
| API Validation | Send rule file to Prometheus via HTTP API | POST /api/v1/rules |
| Configuration Reload | Reload Prometheus config to validate rules | POST /-/reload |
10. Monitoring Rule Evaluation
| Property | Description | Default Value | Notes |
|---|---|---|---|
| evaluation_interval | How often rules are evaluated | 1m |
Configured in the Prometheus config file |
| rule_files | List of files containing recording rules | [] |
Each file is a YAML list of rules |
| external_labels | Labels to add to all metrics | {} |
Used for cross-tenant identification |
Example: Rule Evaluation Configuration
rule_files:
- "recording_rules.yml"
evaluation_interval: 30s
external_labels:
cluster: "prod"
region: "us-west-1"
Note: Rule evaluation happens on the Prometheus server, and results are stored as time series data.
Rule Evaluation Process
- Prometheus evaluates rules at the configured
evaluation_interval - Each rule is evaluated against the current data in the time series
- Results are stored as new time series with labels applied
- These can be used for alerting or visualization in Grafana
11. Testing Recording Rules
Example: Test recording rules with Prometheus console
curl -X GET "http://localhost:9090/api/v1/query?query=rate(http_requests_total[5m])"
| Rule | Expression | Notes |
|---|---|---|
| http_requests_total | rate(http_requests_total[5m]) |
Calculates the per-second rate of HTTP requests over a 5-minute window |
Note: Ensure the rule is defined in a valid PromQL expression and tested with
promtool before applying to production.
Warning: Incorrect rule definitions may lead to high cardinality metrics or performance degradation.
12. Implementing Rule Dependencies
Example: Rule Group with Dependency
groups:
- name: "application_rules"
rules:
- record: "app:http_requests_total:rate1m"
expr: "rate(http_requests_total[1m])"
- record: "app:http_errors_total:rate1m"
expr: "rate(http_errors_total[1m])"
- record: "app:http_error_rate"
expr: "app:http_errors_total:rate1m / app:http_requests_total:rate1m"
labels:
severity: "critical"
| Rule Type | Description | Dependency | Notes |
|---|---|---|---|
| Recording Rule | Creates new time series from expressions | Base metrics or other recording rules | Can reference other recording rules in expressions |
| Alerting Rule | Triggers alerts based on expressions | Base metrics or recording rules | May depend on recording rules for complex logic |
Note: Rule groups are evaluated in order, so dependencies should be defined in the same group or earlier groups.