Skip to main content
Mapagam

Configuring Alerting Rules

1. Defining Alert Rules

Field Description Example
alert Name of the alert rule HighRequestLatency
expr PromQL expression that determines when the alert fires rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m]) > 1
for Duration for which the condition must hold before firing 10m
labels Additional labels to attach to the alert { severity = "page" }
annotations Human-readable information about the alert { summary = "High request latency", description = "The average request latency exceeds 1 second." }

Example: Basic Alert Rule Definition

groups:
- name: http-alerts
  rules:
  - alert: HighRequestLatency
    expr: rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m]) > 1
    for: 10m
    labels:
      severity: page
    annotations:
      summary: "High request latency"
      description: "The average request latency exceeds 1 second."
Note: The for duration allows for alert deduplication and prevents flapping alerts.

2. Setting Alert Duration

Property Values Description
for duration Duration for which the alert must remain active before firing

Example: Setting Alert Duration

groups:
- name: example-group
  rules:
  - alert: HighRequestRate
    expr: rate(http_requests_total[5m]) > 100
    for: 10m
    labels:
      severity: page
    annotations:
      summary: "High request rate detected"
Note: The for property defines how long a condition must persist before the alert is triggered. It helps avoid noisy alerts from temporary spikes.

3. Adding Alert Labels

Label Name Description Example Value
alertname Unique identifier for the alert InstanceDown
severity Level of urgency (info, warning, error, critical) critical
job Service or job name generating the alert prometheus
instance Specific instance or host prometheus-01

Example: Alert Rule with Labels

groups:
- name: example-group
  rules:
  - alert: InstanceDown
    expr: up == 0
    labels:
      severity: critical
      job: prometheus
      instance: "prometheus-01"
    annotations:
      summary: "Instance {{ $labels.instance }} down"
Note: Alert labels are used by the Alertmanager to group and route alerts.

4. Adding Alert Annotations

Annotation Description Example
summary Short description of the alert "CPU usage is above 80% for more than 5 minutes"
description Detailed explanation of the alert condition and impact "The instance {{ $labels.instance }} has been running at over 80% CPU for more than 5 minutes. This may lead to performance degradation."
runbook_url Link to documentation or runbook for resolving the alert "https://internal.docs.example.com/alerts/cpu-high"
dashboard_url Link to dashboard for visualizing the alert context "https://grafana.example.com/d/abc123/cpu-monitoring"
severity Alert severity level (e.g., critical, warning, info) "warning"

Example: Alert Rule with Annotations

groups:
- name: node_rules
  rules:
  - alert: HighCpuUsage
    expr: node_cpu_seconds_total{mode="idle"} < 0.2
    for: 5m
    annotations:
      summary: "High CPU usage detected"
      description: "The instance {{ $labels.instance }} has been running at over 80% CPU for more than 5 minutes."
      runbook_url: "https://internal.docs.example.com/alerts/cpu-high"
      dashboard_url: "https://grafana.example.com/d/abc123/cpu-monitoring"
      severity: "warning"
Note: Annotations are optional but highly recommended for better context and faster response times during alert investigations.

5. Using Templates in Annotations

Prometheus supports templating within alert annotations using Go's text/template library. This allows for dynamic content in alert messages based on metric values.

Template Field Description Example
annotations Map of key-value pairs for additional alert information {{ $value := .Value }} Value: {{ $value }}

Example: Template in Annotation

groups:
- name: example-group
  rules:
  - alert: HighRequestRate
    expr: rate(http_requests_total[5m]) > 100
    annotations:
      summary: "High request rate detected"
      description: "The {{ .Labels.instance }} instance is experiencing high request rates. Current rate: {{ .Value }} requests/second."
Note: Template variables like .Value and .Labels are automatically provided by Prometheus during alert evaluation.
Warning: Complex templates can impact alerting performance. Keep them simple and efficient.

6. Implementing Alert Severity

Severity Level Description Typical Use Case Example Expression
critical System is down or severely degraded, requiring immediate attention. Service unavailability, critical system failure up == 0
error Important issues that require prompt investigation but not immediate action. High error rate, service degradation http_requests_total{status="5xx"} > 100
warning Conditions that may lead to problems or require monitoring. High latency, increasing error rate http_response_time_seconds > 5
info Non-critical conditions that are useful for tracking or informational purposes. Service status, metric threshold crossing processes_running > 1000

Example: Alert Rule with Severity

groups:
- name: example-alerts
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Instance {{ $labels.instance }} down"
      description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes."
Note: Severity levels can be customized based on your organization's alerting strategy. Commonly used levels include critical, error, warning, and info.

7. Creating Alert Dependencies

Alert dependencies in Prometheus are managed through the use of ALERT rules and the record feature to create derived metrics that can be used as conditions for other alerts. This allows for complex alerting logic where one alert might depend on the state of another.

Component Description Usage
Alert Dependency Condition where an alert is triggered only if another alert is active. Use record rules to create a metric indicating the state of a dependent alert.

Example: Alert Dependency Using Recording Rules

# Define a recording rule that captures the state of an alert
- name: alert_dependency_example
  rules:
  - record: alert_state:high_cpu_usage
    expr: up == 1 and cpu_usage > 0.8

# Create an alert that depends on the previous rule
  - alert: high_cpu_usage_alert
    expr: cpu_usage > 0.8
    for: 5m

  - alert: dependent_alert
    expr: alert_state:high_cpu_usage == 1
    for: 5m
Note: The record rule creates a new metric that can be referenced in other alerting rules. This approach is useful for creating complex dependencies or cascading alerts.

8. Configuring Alert Thresholds

Threshold Type Description Example Notes
static Fixed value threshold for alerts http_requests_total > 1000 Simple comparison, no dynamic adjustment
dynamic Threshold based on metric history or statistical analysis rate(http_requests_total[5m]) > 100 Requires time-series data analysis
percentile Threshold based on percentile of metric values http_response_time_seconds{quantile="0.95"} > 2.0 Useful for latency-based alerts

Example: Static Threshold Alert Rule

groups:
- name: http-alerts
  rules:
  - alert: HighRequestRate
    expr: http_requests_total > 1000
    for: 5m
    labels:
      severity: page
    annotations:
      summary: "High request rate detected"
      description: "The number of HTTP requests has exceeded 1000 in the last 5 minutes."

Example: Dynamic Threshold Using Rate

groups:
- name: http-alerts
  rules:
  - alert: HighRequestRate
    expr: rate(http_requests_total[5m]) > 100
    for: 5m
    labels:
      severity: page
    annotations:
      summary: "High request rate detected"
      description: "The rate of HTTP requests has exceeded 100 per minute in the last 5 minutes."
Note: The for clause defines how long a condition must hold before the alert fires, preventing flapping.
Warning: Avoid setting thresholds too low or too high, as they can lead to false positives or negatives respectively.

9. Using Absent Function

The absent() function in Prometheus is used to detect when a time series is missing or has no data points for a given time range. It returns a vector with a single sample value of 1 if the time series is absent, and 0 if it is present.

Function Description Return Value
absent(vector) Checks if the input vector is absent (no data points) over the evaluation time window. Vector with value 1 if absent, 0 if present

Example: Detecting Missing Metrics

absent(up{job="web-server"})
Note: The absent() function is particularly useful in alerting rules to detect when a service or metric stops reporting data.

10. Validating Alert Rules

Example: Basic Alert Rule Validation

# alert_rules.yml
groups:
  - name: example-group
    rules:
      - alert: HighRequestLatency
        expr: rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m]) > 1
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "High request latency detected"
Field Description Required Example
alert Name of the alert rule Yes HighRequestLatency
expr PromQL expression that determines when the alert fires Yes rate(http_request_duration_seconds_sum[5m]) > 1
for Duration for which the condition must hold before firing No 10m
labels Additional labels to attach to the alert No { severity: page }
annotations Metadata for the alert, such as summary and description No { summary: "High request latency" }
Note: Alert rule expressions must be valid PromQL. Use the promtool command-line tool to validate rules.

Example: Using promtool to Validate Rules

promtool check rules alert_rules.yml

Validation Steps

  • Ensure all alert names are unique within a group
  • Verify that expr is valid PromQL syntax
  • Confirm that for durations are in Prometheus time format (e.g., 5m, 1h)
  • Validate that labels and annotations follow YAML syntax
  • Run promtool check rules to detect any syntax or semantic errors

11. Testing Alert Expressions

Expression Description Example
alert Defines an alert rule with a name and expression alert: "HighRequestRate"
expr Expression that evaluates to a boolean or numeric value expr: rate(http_requests_total[5m]) > 100
for Duration for which the condition must hold before firing for: 10m
labels Additional labels to attach to the alert labels: severity: critical
annotations Annotations provide additional information about the alert annotations: summary: "High request rate detected"

Example: Testing Alert Expression

groups:
- name: example-alerts
  rules:
  - alert: HighRequestRate
    expr: rate(http_requests_total[5m]) > 100
    for: 10m
    labels:
      severity: critical
    annotations:
      summary: "High request rate detected"
Note: Use the Prometheus web UI or promtool to test alert expressions before deploying them.

Testing Steps

  1. Write the alert rule in a YAML file
  2. Validate the rule using promtool check rules <filename>
  3. Deploy the rule to Prometheus
  4. Monitor the alert in the Prometheus UI or Alertmanager

12. Implementing Alerting Best Practices

Key Principles

  • Alerts should be actionable and relevant to the system's health
  • Avoid noisy alerts that do not lead to meaningful responses
  • Use appropriate thresholds and time windows to prevent false positives
  • Implement silencing or inhibition rules to manage alert storms
Best Practice Description Implementation
Alert Frequency Minimize alert fatigue by ensuring alerts are not too frequent Use for clause in rules to delay alerting until conditions persist
Alert Severity Define severity levels (e.g., critical, warning, info) to prioritize responses Use severity label in alert rules
Alert Inhibition Suppress related alerts when a more severe alert is active Configure inhibition rules in alerts.yml
Silencing Alerts Temporarily disable alerts during maintenance windows or known issues Use silence matcher in Alertmanager UI or API

Example: Alert Rule with Severity

groups:
- name: example-group
  rules:
  - alert: HighRequestLatency
    expr: rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m]) > 1
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "High request latency detected"

Example: Inhibition Rule

inhibit_rules:
- source_match:
    severity: critical
  target_match:
    severity: warning
  equal: [alertname]

Was this article helpful?

Related Cheat Sheets