Managing Alert Notifications

1. Creating Contact Points

Field	Description	Example
name	Name of the contact point	`Slack Alert Channel`
type	Notification type (e.g., email, webhook, Slack)	`slack`
settings	Configuration settings for the contact point	`{ "url": "https://hooks.slack.com/services/..." }`

Example: Slack Contact Point Configuration

{
  "name": "Slack Alert Channel",
  "type": "slack",
  "settings": {
    "url": "https://hooks.slack.com/services/T0123456789/XXXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXX"
  }
}

Note: Contact points must be configured in the Grafana configuration file or via the UI. For Slack, you need to create a Slack app and get the webhook URL.

2. Configuring Email Notifications

Property	Value	Description
smtp.host	`localhost`	SMTP server host address
smtp.port	`587`	SMTP server port (usually 587 or 465)
smtp.from_address	`grafana@yourdomain.com`	Email address to send alerts from
smtp.from_name	`Grafana Alerts`	Display name for the sender email
smtp.password	`secret`	Password for SMTP authentication
smtp.auth_type	`plain`	Authentication method (plain, login, cram-md5)

Example: Email Alert Configuration

{
  "name": "Email Alert",
  "type": "email",
  "settings": {
    "addresses": "admin@example.com"
  }
}

Note: Ensure your SMTP server allows sending from the Grafana instance's IP address.

Warning: Store SMTP credentials securely using Grafana's secure configuration options.

3. Configuring Slack Notifications

Property	Value	Description
slack.url	`https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX`	Slack webhook URL for incoming messages
slack.channel	`#alerts`	Channel to send alert notifications to
slack.username	`Grafana`	Username to display in Slack messages
slack.icon_emoji	`:chart_with_upwards_trend:`	Emoji to use as the icon in Slack messages

Example: Slack Notification Configuration

{
  "name": "Slack Alert",
  "type": "slack",
  "settings": {
    "url": "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX",
    "channel": "#alerts",
    "username": "Grafana",
    "icon_emoji": ":chart_with_upwards_trend:"
  }
}

Note: Ensure the Slack webhook URL is correctly configured and has appropriate permissions to post messages.

Warning: Slack notifications may be rate-limited by the Slack API. Monitor your notification volume to avoid hitting limits.

4. Configuring PagerDuty Notifications

Property	Value	Description
name	`PagerDuty`	Name of the notification channel
type	`pagerduty`	Notification type identifier
settings.integrationKey	`string`	Integration key from PagerDuty (required)
settings.autoResolve	`boolean`	Automatically resolve incidents when alert clears (default: `true`)
settings.severity	`string`	Severity level for alerts (e.g., `info`, `error`, `warning`)

Example: PagerDuty Channel Configuration

{
  "name": "PagerDuty Alerts",
  "type": "pagerduty",
  "settings": {
    "integrationKey": "pd_integration_key_12345",
    "autoResolve": true,
    "severity": "error"
  }
}

Note: The integration key can be found in the PagerDuty service settings under "Integration Settings".

5. Configuring Webhook Notifications

Property	Value	Description
name	`"Webhook"`	Name of the notification channel
type	`"webhook"`	Notification type identifier
settings.url	`"https://example.com/webhook"`	URL to send alert data
settings.httpMethod	`"POST"`	HTTP method to use (GET, POST, PUT)
settings.basicAuth	`false`	Enable basic authentication
settings.tlsSkipVerify	`false`	Skip TLS certificate verification

Example: Webhook Configuration

{
  "name": "Webhook",
  "type": "webhook",
  "settings": {
    "url": "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX",
    "httpMethod": "POST",
    "basicAuth": false,
    "tlsSkipVerify": false
  }
}

Note: The webhook URL must be accessible from the Grafana server. Ensure that any required authentication or TLS settings are configured correctly.

6. Configuring Microsoft Teams

Property	Value	Description
url	`https://outlook.office.com/webhook/...`	The incoming webhook URL from Microsoft Teams
messageTemplate	`<h3>{{alert_name}}</h3><p>Status: {{status}}</p>`	Customize the message format using Go template syntax
includeImage	`true`	Whether to include alert image in the message

Example: Microsoft Teams Webhook Configuration

{
  "name": "Teams Alert",
  "type": "teams",
  "settings": {
    "url": "https://outlook.office.com/webhook/...",
    "messageTemplate": "<h3>{{alert_name}}</h3><p>Status: {{status}}</p>",
    "includeImage": true
  }
}

Note: The webhook URL must be obtained from Microsoft Teams channel settings.

Warning: Ensure the webhook URL is kept secure and not exposed in logs or version control.

7. Configuring OpsGenie Integration

Configuration Option	Description	Default Value	Notes
opsGenie.apiKey	API key for OpsGenie integration	None	Required for authentication
opsGenie.apiUrl	OpsGenie API endpoint URL	`https://api.opsgenie.com`	Change only if using a custom endpoint
opsGenie.alertPriority	Priority level for alerts sent to OpsGenie	`P3`	Valid values: P1, P2, P3, P4, P5
opsGenie.autoClose	Whether to auto-close alerts in OpsGenie when resolved in Grafana	`false`	Set to `true` to close alerts automatically

Example: OpsGenie Integration Configuration

{
  "opsGenie": {
    "apiKey": "your-opsgenie-api-key",
    "apiUrl": "https://api.opsgenie.com",
    "alertPriority": "P3",
    "autoClose": true
  }
}

Note: Ensure the API key has appropriate permissions to create and manage alerts in OpsGenie.

Warning: Misconfiguring the OpsGenie API key can lead to alert delivery failures.

8. Creating Notification Policies

Notification policies in Grafana define how and when alerts are sent to various contact points. They allow you to configure routing rules based on alert labels and severity levels.

Field	Description	Example Value
receiver	The contact point to send the alert to	`email-notifications`
group_by	Labels to group alerts by before sending	`[instance, job]`
group_wait	Time to wait before sending the first alert in a group	`30s`
group_interval	Time to wait between sending grouped alerts	`5m`
repeat_interval	Time to wait before resending an alert that has been resolved and re-fired	`4h`

Example: Basic Notification Policy Configuration

route:
  receiver: email-notifications
  group_by: [instance, job]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

Note: The root route is the top-level policy that all alerts first match against. You can define sub-routes to handle specific alert labels or severity levels.

Example: Sub-route for Critical Alerts

route:
  receiver: default
  routes:
    - match:
        severity: critical
      receiver: pagerduty
      group_by: [alertname]
      group_wait: 10s
      group_interval: 1m
      repeat_interval: 1h

9. Using Label Matchers

Property	Description	Example
alertRule.labels	Labels to match against alert instances	`{"severity": "critical", "team": "backend"}`
alertRule.annotations	Additional metadata for alerts	`{"dashboard_url": "https://grafana.example.com/d/abc123"}`

Example: Matching Alerts by Labels

name: Critical DB Alert
expr: up == 0
for: 5m
labels:
  severity: critical
  team: backend
annotations:
  summary: "Database is down"
  description: "The database instance at {{ $labels.instance }} has been down for more than 5 minutes."

Note: Label matchers are used in alerting rules to filter and route alerts based on their labels.

10. Setting Notification Grouping

Property	Description	Default Value	Notes
group_wait	Duration to wait before sending the first notification for a new group of alerts	`30s`	Used to batch alerts together for efficiency
group_interval	Duration to wait before resending a group of alerts that have already been sent	`5m`	Prevents spamming repeated notifications
repeat_interval	Duration to wait before resending a group of alerts that have already been sent	`4h`	Used to re-alert after a long period of inactivity

Example: Alertmanager Configuration

route:
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

Note: These settings apply to the Alertmanager configuration and affect how Grafana groups and sends notifications.

11. Configuring Repeat Interval

Property	Value	Description
repeatInterval	`1h`	How often to resend the alert notification if the alert is not resolved
repeatInterval	`24h`	Default repeat interval for recurring alerts

Example: Setting repeat interval in alert rule

name: "High CPU Usage"
expr: "cpu_usage > 80"
for: "5m"
repeatInterval: "2h"
labels:
  severity: "warning"
annotations:
  summary: "CPU usage is above 80%"

Note: Setting repeatInterval to 0s disables repeating notifications.

12. Testing Contact Points

Field	Description	Example
contactPoint.name	Name of the contact point to test	`email-notifications`
contactPoint.type	Type of contact point (e.g., email, webhook)	`email`

Example: Testing Email Contact Point

{
  "contactPoint": {
    "name": "email-notifications",
    "type": "email"
  },
  "message": "Test alert notification"
}

Note: Ensure the contact point is configured with valid email settings before testing.

Warning: Testing contact points sends a real notification to the configured endpoint.