Skip to main content
Mapagam

Working with Alertmanager Integration

1. Configuring Alertmanager URL

Property Value Description
alertmanager.url http://localhost:9093 Base URL of the Alertmanager instance
alertmanager.timeout 10s Timeout for requests to Alertmanager

Example: Alertmanager URL Configuration

alertmanager:
  url: http://alertmanager.example.com:9093
  timeout: 15s

2. Setting Up Static Alertmanagers

Property Value Description
alertmanager.url http://localhost:9093 URL of the Alertmanager instance
alertmanager.timeout 10s Timeout for requests to Alertmanager
alertmanager.retries 3 Number of retry attempts for failed alerts

Example: Static Alertmanager Configuration

alertmanager:
  static_configs:
    - targets:
        - alertmanager.example.com:9093
Note: Ensure the Alertmanager URL is accessible from the Prometheus server.

3. Using Service Discovery for Alertmanagers

Service Discovery (SD) in Prometheus allows automatic detection and configuration of Alertmanagers. This enables dynamic management of Alertmanager instances without manual updates to the Prometheus configuration.

Service Discovery Type Description Configuration Example Notes
File SD Uses a JSON file to define Alertmanager instances. { "targets": [ "alertmanager1:9093", "alertmanager2:9093" ] } Static file-based discovery. Useful for environments with stable Alertmanager instances.
Consul SD Discovers Alertmanagers via Consul catalog. { "server": "localhost:8500", "services": [ "alertmanager" ] } Requires Consul agent running. Supports dynamic updates.
DNS SD Discovers Alertmanagers via DNS SRV records. { "names": [ "_alertmanager._tcp.example.com" ] } Useful in environments where Alertmanagers are registered via DNS.
Kubernetes SD Discovers Alertmanagers running in Kubernetes. { "role": "endpoints", "namespace": "monitoring" } Requires Prometheus to run in Kubernetes. Supports label-based filtering.
EC2 SD Discovers Alertmanagers running on EC2 instances. { "region": "us-east-1", "filters": [ { "name": "tag:Role", "values": [ "alertmanager" ] } ] } Useful in AWS environments. Requires appropriate IAM permissions.

Example: File-based Service Discovery

{
  "targets": [
    "alertmanager1:9093",
    "alertmanager2:9093"
  ],
  "labels": {
    "job": "alertmanager"
  }
}

Example: Kubernetes Service Discovery

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
            - monitoring

alerting:
  alertmanagers:
    - kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
              - monitoring
Note: When using Kubernetes SD, ensure that the Prometheus instance has appropriate RBAC permissions to access the Alertmanager endpoints.

4. Configuring Alert Relabeling

Field Description Example
action The action to perform on the label. Valid values: replace, keep, drop, hashmod, labelmap, labeldrop, labelkeep. replace
source_labels List of label names to use as input for the relabeling action. [__name__, instance]
separator Separator to use when joining multiple source labels. ;
target_label Label name to write the result to. Required for replace, hashmod, and labelmap. job
regex Regular expression to match the source labels. Used in replace, keep, and drop. ^.*$
replacement Replacement string for replace action. Supports regex capture groups. $1

Example: Relabeling job name

relabel_configs:
  - source_labels: [job]
    regex: '(.+)-prod'
    target_label: job
    replacement: '$1'

Example: Dropping labels

relabel_configs:
  - source_labels: [__name__]
    regex: '^alert_(.*)

5. Setting Alert Timeout

Property Value Description
alertmanager.timeout 10s Timeout for sending alerts to Alertmanager
alertmanager.retry_interval 5s Interval between retries for failed alert sends

Example: Configuring Alert Timeout

alertmanager:
  timeout: 10s
  retry_interval: 5s

6. Configuring Alert Queue Capacity

Configuration Option Description Default Value Notes
alertmanager.alert-queue.capacity Maximum number of alerts that can be queued for sending to Alertmanager 10000 Adjust based on alert volume and system resources
alertmanager.alert-queue.max-blocks Maximum number of blocks to keep in memory before dropping alerts 100 Used to prevent memory exhaustion under high alert load

Example: Setting Alert Queue Capacity

alertmanager:
  alert-queue:
    capacity: 15000
    max-blocks: 150
Note: Increasing alert-queue.capacity allows more alerts to be queued but may increase memory usage. Adjust alert-queue.max-blocks to control how many blocks of alerts are retained in memory.

7. Using External Labels for Routing

Label Name Description Usage Example
alertname Unique identifier for the alert Used in routing rules to match specific alerts alertname="HighErrorRate"
severity Severity level of the alert (e.g., critical, warning) Used to route alerts based on severity severity="critical"
job Job name from the Prometheus scrape configuration Used to group alerts by job or service job="web-server"
instance Instance identifier (host:port) Used to route alerts to specific instances instance="localhost:9090"

Example: Alertmanager Configuration with External Labels

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'team-email'
  routes:
  - match:
      severity: critical
    receiver: 'critical-email'
    group_wait: 10s
receivers:
- name: 'team-email'
  email_configs:
  - to: 'team@example.com'
- name: 'critical-email'
  email_configs:
  - to: 'critical@example.com'
Note: External labels are applied to all alerts before they are sent to Alertmanager, allowing for consistent routing and grouping across different Prometheus instances or configurations.

8. Implementing Alert Batching

Alert batching in Prometheus is a mechanism to group multiple alerts into a single notification, reducing the number of alert notifications sent to Alertmanager. This is particularly useful for managing alert fatigue and improving the efficiency of alert handling.

Configuration Parameter Description Default Value Notes
group_wait Duration to wait before sending the first alert batch 30s Time to wait for additional alerts before sending the batch
group_interval Duration to wait before sending a new batch of alerts 5m Time to wait before sending a new batch after the first one
repeat_interval Duration to wait before resending a batch of alerts 4h Time to wait before resending the same alert batch

Example: Alertmanager Configuration for Batching

route:
  group_by: [alertname]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
Note: The group_by parameter determines how alerts are grouped. Alerts with the same value for the specified labels will be batched together.

9. Configuring Basic Auth

Property Value Description
alertmanager.basic_auth.username "admin" Username for basic authentication with Alertmanager
alertmanager.basic_auth.password "secret" Password for basic authentication with Alertmanager

Example: Basic Auth Configuration

alertmanager:
  basic_auth:
    username: "admin"
    password: "secret"

10. Setting Up TLS

Configuration Option Description Default Value Notes
web.tls.cert-file Path to the certificate file for TLS None Required when TLS is enabled
web.tls.key-file Path to the private key file for TLS None Required when TLS is enabled
web.tls.client-ca-file Path to the CA file for client certificate verification None Optional, enables mutual TLS
web.tls.min-version Minimum TLS version to support TLS12 Valid values: TLS10, TLS11, TLS12, TLS13
web.tls.cipher-suites List of TLS cipher suites to allow Default set Comma-separated list of cipher suite names

Example: Alertmanager TLS Configuration

web:
  tls:
    cert-file: /etc/alertmanager/certs/tls.crt
    key-file: /etc/alertmanager/certs/tls.key
    client-ca-file: /etc/alertmanager/certs/ca.crt
    min-version: TLS12
    cipher-suites:
      - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
      - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
Note: Ensure that the certificate and key files are readable by the Alertmanager process. Mutual TLS is optional but recommended for production environments.

11. Monitoring Alert Delivery

Property Description Values Notes
alertmanager.url URL of the Alertmanager instance http://alertmanager:9093 Must be accessible from Prometheus server
alertmanager.timeout Timeout for requests to Alertmanager 10s Default is 10 seconds
alertmanager.retries Number of retries for failed alert sends 3 Defaults to 3 retries

Example: Configuring Alertmanager URL

alertmanager:
  static_configs:
  - targets:
    - alertmanager:9093
Note: Ensure that the Alertmanager endpoint is reachable from Prometheus to receive alerts.

Alert Delivery Monitoring

  • Verify Alertmanager URL is correctly configured
  • Check timeout and retry settings for reliability
  • Monitor alert delivery status in Prometheus UI

12. Implementing Alert Deduplication

Alert deduplication in Prometheus ensures that duplicate alerts are not sent to Alertmanager, reducing noise and improving alert management efficiency.

Configuration Description Default Value
alertmanager.alerts.deduplication Enables or disables alert deduplication in Alertmanager true
alertmanager.alerts.deduplication.timeout Time window (in seconds) to consider alerts as duplicates 30s

Example: Alertmanager Configuration with Deduplication

alertmanager:
  config:
    receivers:
    - name: "team-a"
      email_configs:
      - to: "team-a@example.com"
    route:
      group_by: [alertname]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 1h
      receiver: "team-a"
      deduplication:
        enabled: true
        timeout: 30s
Note: Deduplication is enabled by default in Alertmanager. It groups alerts based on labels and ignores duplicates within the specified timeout window.
Warning: If deduplication is disabled, repeated alerts may flood the system and cause alert fatigue.
action: drop

Was this article helpful?

Related Cheat Sheets