Managing Cluster Health and Monitoring

1. Checking Cluster Health

Health Status	Description	Indicator
green	All primary and replica shards are allocated	✓
yellow	All primary shards are allocated, but some replica shards are unassigned	⚠
red	Some primary shards are unassigned	✗

Example: Using Elasticsearch API to Check Cluster Health

GET /_cluster/health

{
  "cluster_name": "elasticsearch",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 3,
  "number_of_data_nodes": 3,
  "active_primary_shards": 10,
  "active_shards": 20,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0
}

Example: Using cURL to Check Cluster Health with Verbose Output

curl -X GET "localhost:9200/_cluster/health?pretty"

{
  "cluster_name": "elasticsearch",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 3,
  "number_of_data_nodes": 3,
  "active_primary_shards": 10,
  "active_shards": 20,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0
}

Note: The status field in the response indicates the overall health of the cluster. A green status means all shards are allocated, while a red status indicates that some primary shards are missing.

Warning: If the cluster status is red, it indicates a critical issue that requires immediate attention to restore data availability.

2. Viewing Cluster Stats

Stat	Description	Example Value
cluster_name	Name of the Elasticsearch cluster	`my-cluster`
status	Overall health status of the cluster (green, yellow, red)	`green`
number_of_nodes	Total number of nodes in the cluster	`3`
active_shards	Number of active shards	`42`
relocating_shards	Number of shards currently relocating	`0`
initializing_shards	Number of shards initializing	`0`
unassigned_shards	Number of shards not assigned to any node	`0`
pending_tasks	Number of pending tasks in the cluster	`1`

Example: Get Cluster Stats

GET /_cluster/stats

Example: Get Cluster Health

GET /_cluster/health

Note: The status field indicates the overall health of the cluster. A green status means all primary and replica shards are allocated, a yellow status indicates that all primary shards are allocated but some replica shards are unassigned, and a red status means that some primary shards are unassigned.

3. Monitoring Node Stats

Property	Description	Example
os.cpu.percent	Percentage of CPU used by the node	`25`
os.mem.used_in_bytes	Memory used by the node in bytes	`1073741824`
os.mem.free_in_bytes	Free memory available in bytes	`2147483648`
transport.rx_packets	Number of packets received	`1200`
transport.tx_packets	Number of packets transmitted	`1300`
http.active_connections	Active HTTP connections	`50`

Example: Fetching Node Stats via API

GET /_nodes/stats

Example: Fetching Specific Node Stats

{
  "nodes": {
    "node_id": {
      "os": {
        "cpu": {
          "percent": 25
        },
        "mem": {
          "used_in_bytes": 1073741824,
          "free_in_bytes": 2147483648
        }
      },
      "transport": {
        "rx_packets": 1200,
        "tx_packets": 1300
      },
      "http": {
        "active_connections": 50
      }
    }
  }
}

4. Checking Shard Allocation

Property	Description	Values
shard_allocation	Controls how shards are allocated across nodes	`balanced`, `primary`, `manual`
shard_allocation.enable	Enables or disables shard allocation	`true`, `false`
shard_allocation.allow_rebalance	Allows rebalancing of shards	`true`, `false`

Example: Shard Allocation Status

GET /_cluster/allocation/explain

Example: Shard Allocation Settings

{
  "transient": {
    "cluster.routing.allocation.enable": "all"
  },
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "rack"
  }
}

5. Viewing Pending Tasks

Field	Description	Example
pending_tasks	Lists all pending tasks in the cluster	`GET /_cat/pending_tasks?v`
priority	Priority level of the task (HIGH, NORMAL, LOW)	`HIGH`
time_in_queue	Duration the task has been pending	`1m`
source	Originating source of the task	`cluster-formation`

Example: Fetch pending tasks

GET /_cat/pending_tasks?v

Note: Pending tasks can indicate cluster instability or high load.

6. Using Task Management API

Task ID	Properties	Description
task_id	`string`	Unique identifier for the task
status	`string`	Current status of the task (e.g., `running`, `completed`, `failed`)
start_time	`datetime`	Time when the task started
end_time	`datetime`	Time when the task ended
duration	`integer`	Duration of the task in milliseconds

Example: Retrieve Task Information

const taskInfo = {
  task_id: "task-12345",
  status: "running",
  start_time: "2025-04-01T10:00:00Z",
  end_time: "2025-04-01T10:05:00Z",
  duration: 300000
};

console.log(taskInfo);

Example: Cancel a Task

curl -X POST "http://localhost:9200/_tasks/task-12345/_cancel"

Note: The Task Management API allows you to monitor and manage long-running operations in the cluster.

7. Monitoring Thread Pools

Property	Description	Default Value	Notes
thread_pool.search.queue_size	Queue size for search thread pool	`1000`	Controls how many search tasks can be queued before rejection
thread_pool.search.max_concurrent_queries	Maximum concurrent search queries	`10`	Set to `-1` for unlimited
thread_pool.index.queue_size	Queue size for index thread pool	`200`	Controls how many indexing tasks can be queued before rejection
thread_pool.index.max_concurrent_operations	Maximum concurrent indexing operations	`10`	Set to `-1` for unlimited
thread_pool.bulk.queue_size	Queue size for bulk thread pool	`50`	Controls how many bulk operations can be queued before rejection
thread_pool.bulk.max_concurrent_operations	Maximum concurrent bulk operations	`10`	Set to `-1` for unlimited

Example: Configuring Thread Pool Settings

thread_pool:
  search:
    queue_size: 2000
    max_concurrent_queries: 20
  index:
    queue_size: 400
    max_concurrent_operations: 15
  bulk:
    queue_size: 100
    max_concurrent_operations: 20

Note: Adjusting thread pool sizes can significantly impact cluster performance. Monitor queue depths and rejection rates to tune appropriately.

8. Checking Index-Level Stats

Property	Description	Example
index	Name of the index	`my-index`
docs.count	Number of documents in the index	`12500`
docs.deleted	Number of deleted documents in the index	`500`
store.size	Size of the index on disk	`2.4 GB`
segments.count	Number of segments in the index	`15`

Example: Fetching Index Stats

GET /my-index/_stats

Example: Fetching Specific Index Stats

GET /my-index/_stats/docs,store

Note: The _stats API provides a wealth of information about index performance and storage usage, useful for capacity planning and troubleshooting.

9. Using Hot Threads API

Property	Description	Default Value	Notes
hot_threads	Identifies threads that are consuming a high amount of CPU or I/O resources	Disabled	Enabled via `elasticsearch.yml`
interval	How often to check for hot threads	`5s`	Configurable in `elasticsearch.yml`
max_threads	Maximum number of threads to report	`3`	Can be adjusted for more detailed reporting

Example: Enabling Hot Threads API

cluster:
  routing:
    allocation:
      enable_threads: true

Example: Querying Hot Threads API

GET /_nodes/hot_threads

Note: The Hot Threads API is useful for identifying performance bottlenecks in Elasticsearch clusters.

Warning: Frequent querying of the Hot Threads API can impact cluster performance.

10. Configuring Cluster Settings

Setting	Description	Default Value	Notes
cluster.name	Name of the Elasticsearch cluster	`elasticsearch`	Must be unique across clusters
node.name	Name of the node	Auto-generated	Must be unique within the cluster
network.host	Network interface to bind to	`127.0.0.1`	Use `_all_` to bind to all interfaces
http.port	HTTP port to listen on	`9200`	Can be changed to avoid conflicts
transport.port	Transport port for inter-node communication	`9300`	Must be accessible between nodes in a cluster
discovery.seed_hosts	Hosts to join the cluster	`[]`	Required in a multi-node setup
cluster.initial_master_nodes	Nodes eligible to become master in a new cluster	`[]`	Required for 7.x+ clusters in a new setup

Example: Basic Cluster Configuration

cluster.name: my-cluster
node.name: node-1
network.host: 0.0.0.0
http.port: 9200
transport.port: 9300
discovery.seed_hosts: ["host1:9300", "host2:9300"]
cluster.initial_master_nodes: ["node-1", "node-2"]

Note: For a single-node cluster, network.host can be set to localhost or 127.0.0.1.

Warning: Changing cluster.name or node.name after a node has joined a cluster will cause it to leave the cluster.

11. Monitoring Ingest Statistics

Statistic	Description	Metrics	Notes
elasticsearch.cluster.ingest.total	Total number of documents ingested	Count	Tracks all documents processed by the ingest pipeline
elasticsearch.cluster.ingest.failed	Number of failed ingest operations	Count	Indicates failures in processing or pipeline errors
elasticsearch.cluster.ingest.current	Currently active ingest operations	Count	Real-time view of ongoing ingestion tasks
elasticsearch.cluster.ingest.time	Total time spent in ingest processing	Milliseconds	Measures performance overhead of ingestion

Example: Ingest Statistics Query

{
  "aggs": {
    "ingest_stats": {
      "stats": {
        "field": "elasticsearch.cluster.ingest.total"
      }
    }
  }
}

Note: These statistics can be accessed via the /_cluster/stats endpoint or through Kibana's monitoring dashboards.

12. Using Cat APIs

API	Description	Example
/_cat/indices	Lists all indices with basic information	`GET /_cat/indices?v=true&pretty`
/_cat/nodes	Displays information about cluster nodes	`GET /_cat/nodes?v=true&pretty`
/_cat/master	Shows the master node details	`GET /_cat/master?v=true&pretty`
/_cat/plugins	Lists installed plugins on each node	`GET /_cat/plugins?v=true&pretty`
/_cat/shards	Shows shard allocation and status	`GET /_cat/shards?v=true&pretty`
/_cat/health	Displays cluster health status	`GET /_cat/health?v=true&pretty`

Example: Retrieve Cluster Health

GET /_cat/health?v=true&pretty

Example: List All Indices

GET /_cat/indices?v=true&pretty

Note: The ?v=true parameter adds headers to the response for better readability.