Skip to main content
Mapagam

Implementing Alerting and Watcher

1. Understanding Watcher

Watcher is a component of the ELK Stack (Elasticsearch, Logstash, Kibana) that enables users to define and automate alerting rules based on data in Elasticsearch. It allows for the creation of "watch" configurations that can monitor indices, execute queries, and trigger actions when specific conditions are met.

Property Description Values
id Unique identifier for the watch "my-watch"
trigger Defines when the watch should be executed { "schedule": { "interval": "5m" } }
input Defines the data source for the watch { "search": { "request": { "indices": ["logs-*"], "body": { "query": { "match": { "status": "error" } } } } } }
condition Defines the condition under which the watch should execute actions { "compare": { "ctx.payload.hits.total": { "gt": 0 } } }
actions Defines what actions to take when the condition is met { "email": { "to": "admin@example.com", "subject": "Error Alert" } }

Example: Basic Watch Configuration

{
  "id": "error-alert",
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": ["logs-*"],
        "body": {
          "query": {
            "match": {
              "status": "error"
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "actions": {
    "email": {
      "to": "admin@example.com",
      "subject": "Error Alert"
    }
  }
}
Note: Watcher requires a valid Elasticsearch cluster and Kibana instance to function. The watch configuration must be stored in the .watcher-history index or via Kibana's UI.

Key Features of Watcher

  • Triggering Mechanisms: Supports scheduled, manual, and continuous triggers.
  • Input Types: Can pull data from Elasticsearch, HTTP endpoints, or other sources.
  • Condition Logic: Allows complex condition checks using scripting or built-in comparisons.
  • Action Types: Supports email, webhook, Slack, and more via action templates.

2. Creating Watches

Property Description Example
id Unique identifier for the watch "my-watch"
trigger Defines when the watch should run 
{
  "schedule": {
    "interval": "5m"
  }
}
input Defines the data source for the watch 
{
  "search": {
    "request": {
      "indices": ["logs-*"],
      "body": {
        "query": {
          "match": {
            "level": "ERROR"
          }
        }
      }
    }
  }
}
condition Condition to determine if the watch should execute actions 
{
  "compare": {
    "ctx.payload.hits.total.value": {
      "gt": 0
    }
  }
}
actions Defines what to do when the watch triggers 
{
  "email": {
    "to": ["admin@example.com"],
    "subject": "High error count detected",
    "body": "Detected {{ctx.payload.hits.total.value}} errors in the last 5 minutes."
  }
}

Example: Creating a Watch

{
  "id": "error-alert",
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": ["logs-*"],
        "body": {
          "query": {
            "match": {
              "level": "ERROR"
            }
          },
          "size": 0,
          "aggs": {
            "error_count": {
              "value_count": {
                "field": "message"
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.aggregations.error_count.value": {
        "gt": 10
      }
    }
  },
  "actions": {
    "email": {
      "to": ["admin@example.com"],
      "subject": "High error count detected",
      "body": "Detected {{ctx.payload.aggregations.error_count.value}} errors in the last 5 minutes."
    }
  }
}

3. Using Schedule Triggers

Property Description Values Notes
schedule Defines when the watch runs interval, cron Required field
interval Time interval between executions 1m, 5h, 1d Used with schedule
cron Cron expression for scheduling 0 0 * * * Used with schedule

Example: Interval-based Schedule

{
  "schedule": {
    "interval": "10m"
  }
}

Example: Cron-based Schedule

{
  "schedule": {
    "cron": "0 0 * * *"
  }
}

4. Configuring Input Sources

Example: Using a Search Input

{
  "input": {
    "search": {
      "request": {
        "body": {
          "query": {
            "match": {
              "status": "error"
            }
          }
        }
      }
    }
  }
}
Input Type Description Configuration Notes
search Queries data from Elasticsearch request.body, indices Most common input type for alerts
http Fetches data from an HTTP endpoint url, method Useful for external data sources
simple Static input without dynamic data payload Good for testing or fixed values
Note: The search input is the most commonly used input type in Watcher, enabling real-time monitoring of Elasticsearch indices.

5. Defining Conditions

Condition Type Description Example
compare Compares a field value to a threshold using operators like >, <, == 
{
  "ctx.payload.aggregations.avg_response_time.value": {
    "gt": 500
  }
}
exists Checks if a field exists in the document 
{
  "ctx.payload.hits.total": {
    "exists": true
  }
}
range Checks if a field value is within a specified range 
{
  "ctx.payload.aggregations.avg_cpu_usage.value": {
    "gte": 80,
    "lt": 95
  }
}
script Runs a custom script to evaluate the condition 
{
  "script": {
    "source": "return ctx.payload.aggregations.avg_memory_usage.value > 1024"
  }
}

Example: Using Compare Condition

{
  "condition": {
    "compare": {
      "ctx.payload.aggregations.avg_response_time.value": {
        "gt": 500
      }
    }
  },
  "transform": {
    "script": {
      "source": "return 'High response time detected: ' + ctx.payload.aggregations.avg_response_time.value"
    }
  }
}

6. Setting Up Actions

Action Type Description Configuration Example Notes
email Sends an email notification 
{
  "email": {
    "to": ["admin@example.com"],
    "subject": "Alert: High CPU Usage",
    "body": "CPU usage exceeded 80% at {{ctx.execution_time}}"
  }
}
 Requires email server configuration in Elasticsearch
webhook Posts alert data to a webhook URL 
{
  "webhook": {
    "url": "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX",
    "method": "POST"
  }
}
 Supports custom headers and body formatting
slack Sends alert to Slack channel 
{
  "slack": {
    "channel": "#alerts",
    "message": "High memory usage detected on {{ctx.watch_id}}"
  }
}
 Requires Slack incoming webhook setup
index Creates a new document in an index 
{
  "index": {
    "index": "alert-log",
    "doc_type": "alert",
    "doc_id": "{{ctx.watch_id}}"
  }
}
 Useful for storing alert history

Example: Email Action Configuration

{
  "actions": {
    "send_email": {
      "email": {
        "to": ["ops@example.com"],
        "subject": "Elasticsearch Alert: {{ctx.watch_id}}",
        "body": "The following condition was met:\n{{ctx.payload.hits.total}} hits found"
      }
    }
  }
}

Example: Slack Webhook Action

{
  "actions": {
    "send_to_slack": {
      "webhook": {
        "url": "https://hooks.slack.com/services/XXX/YYY/ZZZ",
        "method": "POST",
        "body": "{\n  \"text\": \"Alert triggered: {{ctx.watch_id}}\"\n}"
      }
    }
  }
}

7. Using Transform

Example: Basic Transform Configuration

{
  "id": "transform-example",
  "source": {
    "index": "logs-*"
  },
  "dest": {
    "index": "transformed-logs"
  },
  "transform": {
    "script": {
      "source": "ctx.response = ctx._source.response_time"
    }
  }
}
Property Description Values
id Unique identifier for the transform string
source.index Source index pattern to transform string
dest.index Destination index to store transformed data string
transform.script.source Script to transform the data string
Note: Transforms are executed asynchronously and can be scheduled to run at regular intervals.
Warning: Ensure that the destination index has appropriate mappings to handle transformed data.

8. Testing Watches

Example: Simple Watch Test

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": ["logs-*"],
        "body": {
          "query": {
            "match": {
              "status": "error"
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "actions": {
    "log-error": {
      "logging": {
        "text": "Found {{ctx.payload.hits.total}} error logs"
      }
    }
  }
}
Property Description Values
trigger.schedule.interval Schedule interval for watch execution 1m, 5m, 1h
input.search.request.indices Indices to search in logs-*, app-logs-2024*
condition.compare.ctx.payload.hits.total Condition to compare against gt 0, eq 10
Note: Use ctx.payload.hits.total to access the result count from search input.

Testing Workflow

  1. Create a watch with a simple condition
  2. Use the elasticsearch:watcher API to execute it manually
  3. Check logs for action execution
  4. Adjust conditions or actions based on output

Watch vs Trigger

  • Watch: Complete alerting unit with input, condition, and actions
  • Trigger: Schedule or event that causes a watch to run

9. Managing Watch History

Field Description Values
watch_id Unique identifier for the watch string
execution_time Time when the watch was executed datetime
status Execution status of the watch success, failure, error
triggered Whether the watch was triggered true, false
result Result of the watch execution object

Example: Retrieve Watch History

const history = await client.watcher.getHistory({
  watch_id: 'my-watch'
});

console.log(history);
Note: Watch history is stored in Elasticsearch and can be queried using the _watcher/history endpoint.
Warning: Watch history is limited by default to 100 entries. Configure watcher.history settings to increase this limit.

10. Acknowledging Alerts

Field Description Values
alert_id Unique identifier for the alert string
acknowledged_by User who acknowledged the alert string
acknowledged_at Timestamp of acknowledgment datetime

Example: Acknowledge Alert via API

PUT /_watcher/acknowledge_alert/<alert_id>
{
  "acknowledged_by": "john.doe",
  "acknowledged_at": "2026-04-05T10:00:00Z"
}
Note: Acknowledging an alert does not close it; it only marks that the alert has been reviewed.

11. Using Chained Inputs

Input Type Description Usage Example
chain Allows chaining multiple inputs together in a sequence 
{
  "input": {
    "chain": [
      {
        "search": {
          "request": {
            "indices": ["logs-*"],
            "body": {
              "query": {
                "match": {
                  "level": "error"
                }
              }
            }
          }
        }
      },
      {
        "search": {
          "request": {
            "indices": ["logs-*"],
            "body": {
              "query": {
                "range": {
                  "@timestamp": {
                    "gte": "now-1h"
                  }
                }
              }
            }
          }
        }
      }
    ]
  }
}
search Executes a search query to fetch data from Elasticsearch 
{
  "input": {
    "search": {
      "request": {
        "indices": ["logs-*"],
        "body": {
          "query": {
            "match": {
              "message": "exception"
            }
          }
        }
      }
    }
  }
}
http Fetches data from an HTTP endpoint 
{
  "input": {
    "http": {
      "request": {
        "host": "localhost",
        "port": 9200,
        "path": "/_cluster/health"
      }
    }
  }
}

Example: Chained Search Inputs

{
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "chain": [
      {
        "search": {
          "request": {
            "indices": ["logs-*"],
            "body": {
              "query": {
                "match": {
                  "level": "error"
                }
              },
              "size": 10
            }
          }
        }
      },
      {
        "search": {
          "request": {
            "indices": ["logs-*"],
            "body": {
              "query": {
                "range": {
                  "@timestamp": {
                    "gte": "now-1h"
                  }
                }
              },
              "size": 5
            }
          }
        }
      }
    ]
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total.value": {
        "gt": 0
      }
    }
  },
  "actions": {
    "log-alert": {
      "logging": {
        "text": "Found {{ctx.payload.hits.total.value}} errors in the last hour"
      }
    }
  }
}

12. Debugging Watch Failures

Issue Description Solution
Watcher not triggering Watch fails to execute despite conditions being met Check Watcher logs in Elasticsearch for errors or warnings. Ensure the watch's condition is correctly defined and that the input query returns results.
Incorrect alerting thresholds Alerts fire when they shouldn't or don't fire when they should Review the watch's condition logic and thresholds. Use debug to inspect values during execution.
Timeout errors Watcher times out during execution Increase watcher.execution.timeout setting in Elasticsearch config or optimize the watch's query and actions.
Action failures Alerts are triggered but actions (e.g., email, webhook) fail Check action configuration and logs. Ensure the action URL or email settings are valid.

Example: Debugging Watcher Logs

GET /_cluster/health
GET /_cat/nodes?v
GET /_watcher/stats

Example: Inspecting Watcher Execution

{
  "watch_id": "my_watch",
  "debug": {
    "enabled": true
  }
}
Note: Use debug in the watch definition to inspect variables and conditions during execution.
Warning: Avoid complex queries or actions in watches that may cause timeouts or resource exhaustion.

Was this article helpful?

Related Cheat Sheets