Implementing Kibana Alerts and Actions

1. Understanding Alerting Framework

Component	Description	Status
Trigger	Condition that initiates an alert	NEW
Condition	Logic that evaluates data to determine alert state	NEW
Action	Response to an alert, e.g., send email or webhook	NEW
Throttle	Rate limiting for repeated alerts	NEW

Example: Define a simple alert trigger

{
  "trigger": {
    "threshold": 90,
    "metric": "cpu_utilization"
  }
}

Note: The Kibana Alerting framework is built on top of Elasticsearch and supports various trigger conditions like thresholds, anomalies, and custom queries.

2. Creating Alert Rules

Field	Description	Example
name	Name of the alert rule	`"High CPU Usage"`
enabled	Boolean to enable or disable the rule	`true`
throttle	Time window to suppress alerts (e.g., `"1h"`)	`"5m"`
actions	List of actions to execute when alert triggers	`[{ "actionRef": "webhook-1", "frequency": "5m" }]`

Example: Creating a Basic Alert Rule

{
  "name": "High CPU Usage",
  "enabled": true,
  "throttle": "5m",
  "actions": [
    {
      "actionRef": "webhook-1",
      "frequency": "5m"
    }
  ]
}

Note: Alert rules must be associated with a saved search or query in Kibana.

3. Configuring Alert Conditions

Condition Type	Description	Parameters	Notes
Threshold	Triggers alert when metric exceeds or falls below a defined threshold	`threshold`, `operator`	Supports `above`, `below`, `equal`, `not_equal`
Metric Aggregation	Aggregates metrics over a time window to detect anomalies or trends	`aggregation`, `window`	Useful for detecting patterns in time-series data
Query Threshold	Triggers based on query results count or value	`query`, `threshold`	Can be used to monitor logs or events

Example: Threshold Alert Configuration

{
  "name": "High CPU Usage Alert",
  "condition": {
    "type": "threshold",
    "threshold": 80,
    "operator": "above"
  },
  "time_window": "5m",
  "actions": [
    {
      "name": "Send Email",
      "action_type": "email",
      "params": {
        "to": "admin@example.com"
      }
    }
  ]
}

Note: Alert conditions can be combined using logical operators (AND/OR) to create more complex rules.

4. Setting Up Connectors

Example: Creating a Webhook Connector

{
  "name": "Webhook Alert",
  "type": "webhook",
  "config": {
    "url": "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX",
    "method": "POST",
    "headers": {
      "Content-Type": "application/json"
    }
  }
}

Property	Description	Values	Notes
name	Name of the connector	`string`	Must be unique within the Kibana instance
type	Connector type	`webhook`, `email`, `slack`, `service_now`	Available connector types depend on installed plugins
config	Configuration object for the connector	`object`	Contains specific settings for the connector type

Note: Connectors are managed via Kibana's Actions UI or API. Ensure that the connector configuration is valid before saving.

Warning: Webhook URLs should be secured and not exposed in logs or public code repositories.

Connector Types Overview

Webhook: Sends HTTP requests to external services
Email: Sends email notifications via SMTP
Slack: Posts messages to Slack channels
ServiceNow: Integrates with ServiceNow incident management

5. Using Index Threshold Alerts

Property	Description	Default Value	Notes
name	Unique name for the alert	None	Required field
enabled	Enables or disables the alert	false	Boolean value
throttle	Time window to suppress repeated alerts	None	ISO 8601 duration format (e.g., "5m")
condition.type	Condition type for the alert	None	Must be "threshold"
condition.query	Query to evaluate for alerting	None	Must be a valid KQL or Lucene query
condition.threshold	Threshold value for triggering alert	None	Number or array of numbers
condition.comparison	Comparison operator to use	None	One of: "gt", "gte", "lt", "lte", "eq", "ne"

Example: Creating an Index Threshold Alert

{
  "name": "High Error Rate Alert",
  "enabled": true,
  "throttle": "5m",
  "condition": {
    "type": "threshold",
    "query": "status:500",
    "threshold": 10,
    "comparison": "gt"
  },
  "actions": [
    {
      "id": "action-1",
      "name": "Send Email",
      "params": {
        "to": "admin@example.com",
        "subject": "High Error Rate Alert",
        "body": "The error rate exceeded 10 in the last 5 minutes."
      }
    }
  ]
}

Note: The throttle setting prevents duplicate alerts from being triggered within the specified time window.

Warning: Ensure that the query used in condition.query is optimized to avoid performance issues on large indices.

6. Creating ES Query Alerts

Example: Basic ES Query Alert Configuration

{
  "name": "High Error Rate Alert",
  "enabled": true,
  "throttle": "5m",
  "alertTypeId": "query",
  "params": {
    "size": 100,
    "timeWindowSize": 5,
    "timeWindowUnit": "m",
    "threshold": {
      "gt": 10
    },
    "query": {
      "language": "lucene",
      "query": "status:500"
    }
  },
  "actions": [
    {
      "group": "default",
      "id": "action-1",
      "params": {
        "message": "High error rate detected in logs."
      }
    }
  ]
}

Property	Description	Values
name	Name of the alert	`string`
enabled	Whether the alert is active	`true` or `false`
throttle	Minimum time between alert executions	`5m`, `1h`, etc.
alertTypeId	Identifier for the alert type	`query`
params	Configuration parameters for the alert	`object`
params.size	Maximum number of documents to return	`number`
params.timeWindowSize	Time window size in units	`number`
params.timeWindowUnit	Unit of time window (e.g., m, h)	`m`, `h`
params.threshold	Threshold condition to trigger alert	`{ "gt": 10 }`
params.query	ES query to run for alerting	`{ "language": "lucene", "query": "status:500" }`
actions	Actions to execute when alert triggers	`array`

Note: The query parameter supports both Lucene and KQL query languages.

Warning: Ensure that the size parameter does not exceed Elasticsearch's max_result_window (default 10,000) to avoid performance issues.

7. Configuring Alert Frequency

Property	Value	Description
frequency	`1m`	How often the alert checks for conditions. Can be `1m`, `5m`, `1h`, etc.
timeWindow	`5m`	The time window over which the alert evaluates data. For example, `5m` means the last 5 minutes.
expectedFrequency	`1m`	Expected frequency of data ingestion. Helps in determining if data is missing or delayed.

Example: Setting Alert Frequency in Kibana

{
  "name": "High CPU Usage Alert",
  "enabled": true,
  "frequency": "5m",
  "timeWindow": "10m",
  "expectedFrequency": "1m"
}

Note: Setting frequency too low can cause performance issues in Kibana. It's recommended to use 5m or higher for most use cases.

8. Using Action Variables

Variable	Description	Example Value
alert.id	Unique identifier for the alert	`abc123`
alert.name	Name of the alert	`High CPU Usage Alert`
alert.status	Current status of the alert (active, acknowledged, etc.)	`active`
alert.start_time	Timestamp when the alert started	`2025-04-01T10:00:00Z`
alert.end_time	Timestamp when the alert ended (if applicable)	`2025-04-01T11:00:00Z`
alert.duration	Duration of the alert in milliseconds	`3600000`
alert.severity	Severity level of the alert (e.g., critical, warning)	`critical`
alert.threshold	Threshold value that triggered the alert	`90%`
alert.context	Contextual data related to the alert	`{ "host": "server01", "cpu_percent": 95 }`

Example: Using Action Variables in a Slack Message

const message = `Alert <strong>${alert.name}</strong> has been triggered with severity <strong>${alert.severity}</strong>. The alert started at ${alert.start_time} and is currently <strong>${alert.status}</strong>. Context: ${JSON.stringify(alert.context)}`;

Note: Action variables are dynamically substituted when the action is executed, allowing for flexible and context-aware alert responses.

9. Managing Alert History

Field	Description	Values
alertId	Unique identifier for the alert	`string`
status	Current status of the alert	`active`, `inactive`, `error`
timestamp	ISO timestamp when the alert was triggered	`datetime`
message	Alert message or description	`string`
severity	Severity level of the alert	`low`, `medium`, `high`, `critical`

Example: Retrieve Alert History

const alertHistory = await kibanaClient.alerts.getHistory({
  alertId: "alert-123"
});

Note: Alert history is retained based on Kibana's elasticsearch index settings.

Warning: Historical alert data may be purged based on retention policies.

10. Muting Alerts

Property	Description	Values
mute	Flag to mute the alert	`true` or `false`
muteUntil	Time until which the alert is muted	`ISO 8601 date string`

Example: Mute an Alert

{
  "mute": true,
  "muteUntil": "2026-12-31T23:59:59Z"
}

Note: Muting an alert prevents it from sending notifications but does not stop the underlying rule from running.

11. Setting Up Alert Tags

Tag Name	Description	Usage	Examples
alert.status	Indicates the current status of an alert	Used for filtering and monitoring alerts	`active`, `inactive`, `throttled`
alert.severity	Severity level of the alert (e.g., critical, warning)	Helps prioritize alert response	`critical`, `high`, `medium`, `low`
alert.source	Origin of the alert (e.g., log, metric)	Used to categorize alerts	`logs`, `metrics`, `apm`
alert.category	Logical grouping of alerts (e.g., system, application)	Facilitates alert organization	`system`, `application`, `security`

Example: Creating an Alert with Tags

{
  "name": "High CPU Usage Alert",
  "tags": [
    {
      "name": "alert.status",
      "value": "active"
    },
    {
      "name": "alert.severity",
      "value": "high"
    },
    {
      "name": "alert.source",
      "value": "metrics"
    }
  ]
}

12. Testing Alert Actions

Example: Test Alert Action Configuration

{
  "name": "Test Slack Action",
  "type": "slack",
  "config": {
    "url": "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
  }
}

Field	Description	Values
name	Action name for identification	`string`
type	Integration type (e.g., slack, email)	`slack`, `email`, `webhook`
config	Configuration parameters for the action	`object`

Testing Workflow

Go to Kibana > Alerting > Actions
Create or select an existing action
Click "Test" button in the action details
Review test results and logs in the UI

Note: Testing actions ensures proper configuration and connectivity before alerts are triggered.