Configuring Grafana Alerting

1. Understanding Unified Alerting

Grafana's Unified Alerting is a centralized alerting system that allows you to define, manage, and respond to alerts across multiple data sources in a unified way.

Feature	Description	Status
Alert Rule	A definition of a condition that triggers an alert when met.	NEW
Notification Policy	Defines how and where alerts are sent (e.g., email, Slack, webhook).	NEW
Contact Point	Configuration for a specific notification method (e.g., Slack, email).	NEW
Silence	Temporarily suppress alert notifications for a specified time period.	NEW
Mute Timing	Define time ranges during which alerts should not be sent.	NEW

Example: Creating an Alert Rule

name: High CPU Usage Alert
condition: avg(cpu_usage) > 80
for: 5m
labels:
  severity: critical
annotations:
  summary: "CPU usage is above 80%"

Note: Unified Alerting requires Grafana v9.0 or higher.

Key Concepts

Alert Rule: Defines when to trigger an alert based on a query.
Notification Policy: Determines how to route and handle alerts.
Contact Point: Specifies where notifications should be sent (e.g., Slack, email).
Silence: Temporarily suppresses alerts to avoid noise during maintenance windows.

2. Creating Alert Rules

Property	Value	Description
name	`"High CPU Usage"`	Name of the alert rule
condition	`"A"`	Reference to the query (e.g., `A`)
evaluationInterval	`30s`	How often to evaluate the alert
for	`5m`	Duration to wait before firing the alert
annotations	`{ summary: "CPU usage over 80%" }`	Additional metadata for the alert
labels	`{ severity: "warning" }`	Tags to categorize the alert

Example: Creating an Alert Rule via API

{
  "name": "High CPU Usage",
  "condition": "A",
  "evaluationInterval": "30s",
  "for": "5m",
  "annotations": {
    "summary": "CPU usage over 80%"
  },
  "labels": {
    "severity": "warning"
  }
}

Note: Alert rules can be created via the Grafana UI or API. The for field defines how long a condition must be true before the alert fires.

Warning: Ensure that queries used in alert conditions are optimized to prevent performance issues.

3. Defining Alert Queries

Property	Description	Example
query	The query to execute for the alert condition	`rate(http_requests_total[5m]) > 100`
datasource	The data source to use for the query	`prometheus`
refId	Reference ID for the query in the alert	`A`

Example: Defining a Prometheus Alert Query

query:
  refId: A
  datasource: prometheus
  expr: rate(http_requests_total[5m]) > 100

Note: The query must return a numeric value for alerting to work properly.

4. Setting Alert Conditions

Condition	Description	Example
Query	Defines the data source query to evaluate for the alert	`SELECT mean("usage") FROM "cpu" WHERE $timeFilter GROUP BY time(1m)`
Time Range	Specifies the time window over which to evaluate the condition	`5m` or `1h`
Operator	Defines how to compare the query result to the threshold	`>`, `<`, `=`, `!=`
Threshold	The value to compare the query result against	`80`

Example: CPU Usage Alert

{
  "conditions": [
    {
      "query": {
        "model": {
          "refId": "A",
          "datasourceUid": "L7b3V2g4k",
          "rawSql": "SELECT mean(\"usage\") FROM \"cpu\" WHERE $timeFilter GROUP BY time(1m)"
        }
      },
      "operator": "gt",
      "threshold": {
        "value": 80,
        "type": "absolute"
      }
    }
  ]
}

Note: The rawSql field uses the data source's native query language (e.g., InfluxQL for InfluxDB).

5. Configuring Evaluation Interval

Property	Value	Description
alerting.evaluation_interval	`10s`	How often to evaluate alerts. Default is 10 seconds.
alerting.max_evaluations	`10`	Maximum number of evaluations per alert rule. Default is 10.

Example: Setting Evaluation Interval

[alerting]
evaluation_interval = 30s
max_evaluations = 5

Note: The evaluation interval affects how frequently Grafana checks the conditions defined in alert rules. Lower intervals provide more responsive alerts but increase system load.

6. Setting Pending Period

The pending period in Grafana alerting defines the time window during which an alert must remain in a firing state before it is considered "firing" and triggers notifications. This helps prevent noisy alerts from being sent for transient issues.

Property	Values	Description
alert.pending.period	`0s`	Duration to wait before firing an alert. Default is `0s`, meaning no delay.

Example: Configuring Pending Period in Grafana Configuration

[alerting]
pending.period = 30s

Note: Setting a pending period can help reduce alert fatigue by ensuring that only persistent issues trigger notifications.

7. Using Alert Expressions

Alert expressions in Grafana define the conditions under which an alert should trigger. They are written using the expr field in the alert rule configuration.

Field	Description	Example
expr	The PromQL expression that evaluates to a time series or scalar value.	`rate(http_requests_total[5m]) > 10`
for	Duration to wait before triggering the alert after the condition is met.	`10m`
labels	Additional labels to attach to the alert.	`{severity="critical"}`
annotations	Annotations provide additional context or metadata for the alert.	`{summary="High request rate", description="The number of HTTP requests exceeded 10 per minute."}`

Example: Basic Alert Expression

rate(http_requests_total[5m]) > 10

Example: Alert Expression with For Duration

rate(http_requests_total[5m]) > 10
for: 10m
labels:
  severity: critical
annotations:
  summary: "High request rate"
  description: "The number of HTTP requests exceeded 10 per minute."

Note: Alert expressions are evaluated in the context of a Grafana dashboard or through the Alerting UI. The expression must return a scalar or time series that can be evaluated for triggering conditions.

8. Adding Alert Labels

Label Key	Label Value	Description
alertname	`string`	Unique identifier for the alert rule
severity	`critical \| error \| warning \| info`	Level of alert severity
service	`string`	Service or component related to the alert
team	`string`	Team responsible for the alert

Example: Adding Labels to an Alert Rule

name: "High CPU Usage"
labels:
  severity: "warning"
  service: "web-server"
  team: "backend"
annotations:
  summary: "CPU usage is above 80%"

Note: Alert labels are used for grouping and filtering alerts in Grafana's alerting system.

9. Adding Alert Annotations

Field	Description	Example
alertName	Name of the alert rule	`"High CPU Usage"`
dashboardId	ID of the dashboard where the alert is configured	`12345`
panelId	ID of the panel within the dashboard	`67890`
time	Time when the alert was triggered (ISO 8601 format)	`"2025-04-05T14:30:00Z"`
state	Current state of the alert (e.g., 'alerting', 'ok')	`"alerting"`

Example: Adding Alert Annotations via API

const annotation = {
  alertName: "High CPU Usage",
  dashboardId: 12345,
  panelId: 67890,
  time: "2025-04-05T14:30:00Z",
  state: "alerting"
};

fetch('/api/annotations', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': 'Bearer <your-api-token>'
  },
  body: JSON.stringify(annotation)
});

10. Setting No Data Handling

Option	Values	Description
no_data_state	`OK`, `ALERTING`, `NO_DATA`	State to set when no data is returned by the query

Example: Configuring no_data_state in a dashboard

{
  "panels": [
    {
      "id": 1,
      "type": "graph",
      "targets": [
        {
          "expr": "up",
          "legendFormat": "Service Status"
        }
      ],
      "alert": {
        "no_data_state": "ALERTING"
      }
    }
  ]
}

Note: When no_data_state is set to ALERTING, the alert will trigger if no data is returned from the query.

11. Setting Error Handling

Property	Description	Default Value	Notes
alerting.error_handling	Controls how Grafana handles errors during alert evaluation	`fail`	Valid values: `fail`, `ignore`
alerting.error_handling.ignore	When set to true, Grafana ignores errors during alert evaluation and continues processing	`false`	Use with caution as it may hide critical issues

Example: Configuring Error Handling

[alerting]
error_handling = ignore

Note: Setting alerting.error_handling = ignore can mask underlying issues in alert rules or data sources, potentially leading to silent failures. It is recommended to use this setting only when debugging or in controlled environments.

12. Creating Alert Rule Groups

Property	Description	Example
name	Name of the alert rule group	`"Database Alerts"`
interval	How often to evaluate the rules in the group (in seconds)	`60`
rules	List of alert rules within this group	`[{...}]`

Example: Creating an Alert Rule Group

{
  "name": "Database Alerts",
  "interval": 60,
  "rules": [
    {
      "name": "High DB Load",
      "query": "sum by (instance) (rate(db_requests[5m])) > 100",
      "condition": "GT",
      "threshold": 100,
      "for": "5m"
    }
  ]
}

Note: Alert rule groups are defined in JSON format and must be valid according to Grafana's alerting schema.