Integrating Grafana OnCall

1. Understanding Grafana OnCall

Feature	Description	Details
Alerting	Manages alerting workflows	Integrates with Grafana's alerting engine to send alerts via various channels
Incident Management	Handles incident response workflows	Enables teams to collaborate on resolving incidents with detailed timelines and status updates
Escalation Policies	Defines alert escalation paths	Allows defining who gets notified and when based on time or severity
On-Call Scheduling	Manages shift schedules	Supports rotating shifts and on-call assignments with calendar integration
Integration with Grafana	Seamless integration with Grafana dashboards	Enables direct linking from dashboard alerts to OnCall incidents

Example: Alert Configuration in Grafana

{
  "name": "High CPU Usage Alert",
  "condition": "A > 90",
  "evaluationInterval": "1m",
  "for": "5m",
  "labels": {
    "severity": "critical"
  },
  "annotations": {
    "summary": "CPU usage exceeds threshold",
    "description": "The CPU usage has been above 90% for more than 5 minutes"
  }
}

Note: Grafana OnCall is designed to work with Grafana's alerting engine, allowing for seamless integration between monitoring and incident management.

2. Connecting Grafana to OnCall

Configuration	Description	Value/Example
oncall_url	URL of the Grafana OnCall instance	`https://oncall.example.com`
api_key	API key for authentication with OnCall	`oncall-api-key-here`
grafana_url	URL of the Grafana instance	`https://grafana.example.com`

Example: OnCall Integration Configuration

{
  "oncall_url": "https://oncall.example.com",
  "api_key": "oncall-api-key-here",
  "grafana_url": "https://grafana.example.com"
}

Note: Ensure the API key has appropriate permissions to access and manage alerting rules in Grafana.

3. Configuring Escalation Chains

Field	Description	Example
name	Name of the escalation chain	`"Critical Alert Escalation"`
steps	List of steps in the escalation chain	`[{ "users": ["admin"], "wait": 300 }]`

Example: Basic Escalation Chain Configuration

{
  "name": "Critical Alert Escalation",
  "steps": [
    {
      "users": ["admin"],
      "wait": 300
    },
    {
      "users": ["oncall_team"],
      "wait": 600
    }
  ]
}

Note: Each step in the escalation chain defines a set of users to notify and how long to wait before escalating.

Step Field	Description	Values
users	List of user IDs to notify in this step	`["user1", "user2"]`
wait	Time in seconds to wait before escalating	`300`
repeat	Number of times to repeat this step (optional)	`2`

Example: Escalation with Repeat

{
  "name": "Repeat Escalation",
  "steps": [
    {
      "users": ["admin"],
      "wait": 300,
      "repeat": 2
    }
  ]
}

Warning: Setting wait to zero will escalate immediately without delay.

Escalation Chain Summary

Each step defines a set of users and a wait time
Escalation occurs sequentially through each step
Steps can be repeated to increase notification frequency

4. Setting Up Schedules

Schedule Property	Value/Description	Notes
name	`"Team A Schedule"`	Unique identifier for the schedule
timeZone	`"America/New_York"`	Timezone used for schedule calculations
shifts	`[ { "start": "2025-04-01T09:00:00Z", "end": "2025-04-01T17:00:00Z", "rotation": { "type": "hourly", "duration": 8 } } ]`	Shifts define working hours and rotation logic

Example: Creating a Basic Schedule

{
  "name": "DevOps OnCall",
  "timeZone": "UTC",
  "shifts": [
    {
      "start": "2025-04-01T00:00:00Z",
      "end": "2025-04-01T08:00:00Z",
      "rotation": {
        "type": "daily",
        "duration": 1
      }
    }
  ]
}

Note: Schedules can be assigned to teams or users via the Grafana UI or API.

Warning: Ensure timezone consistency across all schedules to avoid misalignment in shift assignments.

5. Configuring Alert Routes

Property	Value	Description
route.group_by	`[]`	Groups alerts by specified labels (e.g., `["alertname"]`)
route.matchers	`[]`	List of matchers to filter alerts (e.g., `[{label: "severity", value: "critical"}]`)
route.receiver	`default`	Name of the contact point to send alerts to
route.repeat_interval	`4h`	Time to wait before resending an alert
route.group_wait	`30s`	Time to wait before sending the first group of alerts
route.group_interval	`5m`	Time to wait between sending grouped alerts

Example: Basic Route Configuration

route:
  group_by: [&#34;alertname&#34;]
  matchers:
    - label: &#34;severity&#34;
      value: &#34;critical&#34;
  receiver: &#34;slack-notifications&#34;
  repeat_interval: 4h
  group_wait: 30s
  group_interval: 5m

Note: Alert routes are processed in order, so more specific routes should come before general ones.

6. Using Alert Grouping

Property	Description	Default Value	Notes
grouping_key	Key used to group alerts in OnCall	`None`	Can be any field from the alert payload
grouping_timeout	Time in seconds to wait before creating a new alert group	`300`	Defaults to 5 minutes
grouping_strategy	Strategy used to group alerts (e.g., by source, severity)	`source`	Options: `source`, `severity`, `custom`

Example: Alert Grouping Configuration

{
  "grouping_key": "alertname",
  "grouping_timeout": 600,
  "grouping_strategy": "severity"
}

Note: Alert grouping helps reduce alert fatigue by consolidating similar alerts into a single notification.

Warning: Incorrect grouping configuration can lead to missed critical alerts.

7. Managing Incidents

Property	Description	Values
incident.id	Unique identifier for the incident	`string`
incident.status	Current status of the incident	`open`, `acknowledged`, `resolved`
incident.created_at	Timestamp when the incident was created	`ISO 8601 datetime`
incident.assigned_to	User or team assigned to handle the incident	`string`

Example: Create Incident

const incident = {
  title: "Database connection timeout",
  status: "open",
  created_at: "2025-04-05T10:00:00Z",
  assigned_to: "db-team"
};

// Send to Grafana OnCall API
fetch('/api/incidents', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify(incident)
});

Example: Update Incident Status

const update = {
  status: "acknowledged"
};

fetch('/api/incidents/12345', {
  method: 'PUT',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify(update)
});

Note: Incident status updates are logged and can be used for audit trails.

Incident Lifecycle

Open: Incident is reported and awaiting response.
Acknowledged: Team has acknowledged the incident and is working on it.
Resolved: Incident has been fixed and closed.

8. Configuring Mobile App

Setting	Description	Value
Mobile App URL	URL to access the Grafana OnCall mobile app	`https://oncall.grafana.net`
Push Notifications	Enable or disable push notifications for alerts	`true`
Silence Alerts	Allow silencing alerts from the mobile app	`true`
Re-notify Interval	Time interval (in minutes) to re-notify users	`30`

Example: Configure Mobile App Settings

{
  "push_notifications": true,
  "silence_alerts": true,
  "re_notify_interval": 30
}

Note: Ensure that the mobile app has proper permissions to receive push notifications.

9. Using Slack Integration

Property	Value	Description
slack_integration.enabled	`true`	Enable or disable Slack integration
slack_integration.bot_token	`xoxb-...`	Slack bot token for authentication
slack_integration.app_token	`xapp-...`	Slack app-level token for events

Example: Configure Slack Integration

[slack_integration]
enabled = true
bot_token = xoxb-123456789012-123456789012-abc123def456
app_token = xapp-1-A1B2C3D4E5F6G7H8I9J0K1L2M3N4O5P6Q7R8S9T0U1V2W3X4Y5Z6

Note: Ensure the Slack bot has appropriate permissions in the workspace to post messages and manage channels.

Slack Integration Capabilities

Post alert notifications to Slack channels
Allow users to acknowledge or snooze alerts via Slack
Integrate with Slack's user groups for dynamic on-call scheduling

10. Viewing Insights and Metrics

Insight/Metric	Description	Access Method	Notes
Alerts	Active and resolved alerts from integrated systems	Dashboard, Alert list view	Includes severity levels and timestamps
Incidents	Active incidents and their status	OnCall dashboard, Incident timeline	Tracks incident lifecycle from detection to resolution
Response Time	Average response time for alerts and incidents	Metrics panel, Grafana dashboard	Can be filtered by team or service
Uptime	System uptime and availability metrics	Dashboard, Service health panel	Shows SLA compliance and downtime history
Team Performance	Metrics on team response and resolution times	OnCall analytics, Team performance dashboard	Includes leaderboards and trend analysis

Example: Viewing Alert Metrics in Grafana

// Query to fetch alert metrics from Grafana
const alertMetricsQuery = {
  "query": "sum by (alertname) (rate(alerts[5m]))",
  "format": "time_series"
};

// Example response structure
{
  "alertname": "High CPU Usage",
  "value": 2.5,
  "timestamp": "2026-01-15T10:00:00Z"
}

Note: Metrics are updated in real-time and can be filtered by time range, team, or service.

Key Metrics Overview

Alerts: Real-time alert status and count
Incidents: Incident tracking and resolution progress
Response Time: Average time to acknowledge and resolve alerts
Uptime: System availability and SLA compliance
Team Performance: Team efficiency and incident handling metrics