Configuring Notification Templates

1. Using Template Variables

Variable	Description	Example Value
{{ .Alerts.Firing }}	List of firing alerts	`[{name: "High CPU", status: "firing"}]`
{{ .Alerts.Resolved }}	List of resolved alerts	`[{name: "High CPU", status: "resolved"}]`
{{ .Status }}	Alert status (firing or resolved)	`firing`
{{ .Alerts.Count }}	Total number of alerts	`5`

Example: Alert Template with Variables

templates:
  - /etc/prometheus/templates/alerts.tmpl

# Template file: alerts.tmpl
{{ range .Alerts }}
Alert: {{ .Labels.alertname }}
Status: {{ .Status }}
Summary: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
{{ end }}

2. Implementing Conditional Logic

Example: Using Go Templates for Conditional Logic

{{ if eq .Status "firing" }}
  Alert: {{ .Alerts.0.Name }} is firing
{{ else if eq .Status "resolved" }}
  Alert: {{ .Alerts.0.Name }} has resolved
{{ else }}
  Alert: {{ .Alerts.0.Name }} is pending
{{ end }}

Template Function	Description	Usage
eq	Checks equality between two values	`{{ if eq .Status "firing" }}`
ne	Checks inequality between two values	`{{ if ne .Status "resolved" }}`
lt, le, gt, ge	Comparison operators for numbers	`{{ if gt .Value 100 }}`
and, or	Logical operators for combining conditions	`{{ if and (gt .Value 10) (lt .Value 100) }}`
not	Negates a boolean condition	`{{ if not .Alerts.0.Fired }}`

Note: Template conditions are evaluated using the Go template engine, which supports a wide range of conditional and logical operators.

Example: Nested Conditional Logic in Alert Templates

{{ range .Alerts }}
  {{ if eq .Status "firing" }}
    {{ if gt .Value 1000 }}
      Critical: {{ .Name }} is at {{ .Value }}
    {{ else }}
      Warning: {{ .Name }} is at {{ .Value }}
    {{ end }}
  {{ else if eq .Status "resolved" }}
    Resolved: {{ .Name }} has returned to normal
  {{ end }}
{{ end }}

3. Using Range Loops

Range loops in Prometheus notification templates allow you to iterate over a set of values or labels, enabling dynamic content generation based on metric data.

Property	Description	Usage
range	Defines the range of values to loop over	`{{ range $value := .Values }}`
index	Accesses the current index in the loop	`{{ $index := .Index }}`
value	Accesses the current value in the loop	`{{ $value := .Value }}`

Example: Iterating Over Labels

{{ range $label, $value := .Labels }}
Label: {{ $label }} = {{ $value }}
{{ end }}

Example: Processing Alert Values

{{ range $index, $value := .Values }}
Value {{ $index }}: {{ $value }}
{{ end }}

4. Formatting Values

Format	Description	Example
{{.Value}}	Raw value of the metric	`42.5`
{{.Labels}}	All labels as a map	`map[instance:localhost:9090 job:prometheus]`
{{.Labels.<label_name>}}	Specific label value	`localhost:9090`
{{.Alerts}}	List of alerts in the group	`[{name: HighCPUUsage, state: firing}]`
{{.GroupKey}}	Unique identifier for the alert group	`{alertname=HighCPUUsage}`
{{.CommonLabels}}	Labels common to all alerts in the group	`map[alertname:HighCPUUsage]`
{{.CommonAnnotations}}	Annotations common to all alerts in the group	`map[summary:CPU usage is high]`
{{.Status}}	Status of the alert (firing, resolved)	`firing`

Example: Using Value Formatting in Templates

templates:
  - "alert.tmpl"

template:
  - name: "alert.tmpl"
    content: |
      {{ range .Alerts }}
        Alert: {{ .Labels.alertname }}
        Value: {{ .Value }}
        Status: {{ .Status }}
      {{ end }}

5. Accessing External Labels

Label	Description	Usage
external_labels	Labels to be added to all metrics	Used for identifying the source or environment of metrics
job	Job name from scrape configuration	Identifies the target job for scraping
instance	Instance identifier from scrape configuration	Distinguishes between different instances of the same job

Example: Using external labels in a template

templates:
  - "{{ .ExternalLabels.job }}"

Note: External labels are defined in the Prometheus configuration and are added to all metrics before scraping.

6. Using Template Functions

Function	Description	Example
title	Extracts the title from an alert	`{{ title .Alerts }}`
message	Extracts the message from an alert	`{{ message .Alerts }}`
labels	Accesses label values from an alert	`{{ .Alert.Labels.job }}`
annotations	Accesses annotation values from an alert	`{{ .Alert.Annotations.description }}`
externalURL	Provides the external URL of the Prometheus instance	`{{ .ExternalURL }}`
alertStatus	Determines the status of an alert (firing, resolved)	`{{ alertStatus .Alerts }}`

Example: Using Template Functions in Alertmanager Configuration

receivers:
- name: "email"
  email_configs:
  - to: "team@example.com"
    subject: "{{ title .Alerts }}"
    text: "Alert: {{ message .Alerts }}\nLabels: {{ labels .Alerts }}"

Note: Template functions are part of the Go template engine. Ensure that all functions used in templates are compatible with the version of Go used by Prometheus.

7. Implementing Multi-Line Templates

Example: Multi-line template with line breaks

templates:
  - /etc/prometheus/templates/alerts.tmpl

Template Field	Description	Usage
template	Multi-line template content	Used in alerting rules to format notifications

Note: Template files must be valid YAML or JSON and support multi-line strings using | or > syntax.

Example: Template with multi-line message

templates:
  - |
    {{ range .Alerts }}
    Alert: {{ .Annotations.summary }}
    Description: {{ .Annotations.description }}
    Severity: {{ .Labels.severity }}
    {{ end }}

8. Adding URLs to Annotations

Field	Description	Example
annotations	Map of annotation key-value pairs	`{"url": "https://example.com/alerts/$(alertname)"}`

Example: Adding URL to Alert Annotations

groups:
- name: example-group
  rules:
  - alert: HighRequestRate
    expr: rate(http_requests_total[5m]) > 1000
    annotations:
      summary: "High request rate detected"
      url: https://example.com/alerts/$(alertname)

Note: The $(alertname) template variable is replaced with the alert name at runtime.

9. Using Custom Template Functions

Prometheus provides a way to extend template functionality using custom functions through the template package in Go. These functions can be used to enhance alert templates with additional logic or formatting capabilities.

Function	Description	Usage Example
custom.funcName	Custom function to process alert data	`{{ custom.funcName .Alerts }}`

Example: Defining a Custom Template Function

func customFunc(alerts []Alert) string {
    // Custom logic here
    return fmt.Sprintf("Processed %d alerts", len(alerts))
}

Note: Custom template functions must be registered with the Prometheus template engine to be used in alerting rules.

10. Testing Templates

Example: Template Validation

# prometheus.yml
global:
  evaluation_interval: 1m
  scrape_interval: 15s

rule_files:
  - "alert.rules"

# Template file for notifications
template_files:
  - "notification.tmpl"

Field	Description	Example
template_files	List of template files to load	`["notification.tmpl"]`

Example: Alert Template

{{ define "alert.template" }}
<div class="alert">
  <h3>{{ .CommonLabels.alertname }}</h3>
  <p><strong>Status:</strong> {{ .Status }}</p>
  <p><strong>Start Time:</strong> {{ .CommonLabels.start_time }}</p>
</div>
{{ end }}

Note: Template files must be valid Go template syntax. Use promtool check rules to validate rule files and templates.

Template Testing Workflow

Write template in .tmpl file
Validate with promtool check rules
Load template in Prometheus config
Trigger test alert to verify rendering

11. Implementing Runbook URLs

Field	Description	Example
runbook_url	URL template for runbooks associated with alerts	`https://mycompany.com/runbooks/{alertname}`
runbook_url_template	Template variable for alert name in runbook URL	`{alertname}`

Example: Alertmanager Configuration with Runbook URL

route:
  group_by: [alertname]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: team-email
  routes:
  - match:
      severity: critical
    receiver: team-email
    continue: true
    runbook_url: https://mycompany.com/runbooks/{alertname}

receivers:
- name: team-email
  email_configs:
  - to: team@example.com

Note: The {alertname} placeholder is replaced with the actual alert name at runtime.

12. Using Dashboard Links

Property	Description	Example
dashboard.url	URL template for the Prometheus dashboard	`http://localhost:9090`
dashboard.link	Link to the Prometheus dashboard with optional query parameters	`http://localhost:9090/graph?g0.expr=up&g0.tab=0`

Example: Dashboard URL Template

templates:
  - "dashboard.tmpl"

Example: Dashboard Link in Alert

receivers:
  - name: "web.hook"
    webhook_configs:
      - url: "http://localhost:8080/alert"
        send_resolved: true
        http_config:
          bearer_token: "secret"
        <span class="badge badge--new">NEW</span> dashboard_link: "http://localhost:9090/graph?g0.expr=up&amp;g0.tab=0"

Note: The dashboard_link field is available in Prometheus v2.35.0+.