Configuring Cluster Performance

1. Configuring Number of Shards

Parameter	Description	Default Value	Notes
index.number_of_shards	Number of primary shards for an index	1	Cannot be changed after index creation
index.number_of_replicas	Number of replica shards per primary shard	1	Can be adjusted after index creation

Example: Setting shard count during index creation

{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "name": {
        "type": "text"
      }
    }
  }
}

Note: Too many shards can lead to increased overhead and reduced performance. Aim for shard sizes between 10GB and 50GB.

2. Setting Number of Replicas

Property	Default	Description	Notes
index.number_of_replicas	`1`	Number of replica shards per primary shard	Controls redundancy and availability of data

Example: Setting Number of Replicas

{
  "settings": {
    "number_of_replicas": 2
  }
}

Note: Setting number_of_replicas to 0 disables replication for the index.

Warning: Increasing replica count beyond available nodes can lead to unassigned shards.

3. Tuning Merge Policy

Parameter	Description	Default Value	Notes
merge.policy	Merge policy used to merge segments during index creation	`log_byte_size`	Controls how segments are merged based on size or count
merge.policy.max_merge_size	Maximum size of a merged segment	`5gb`	Prevents merging of very large segments
merge.policy.min_merge_size	Minimum size of a segment to be considered for merging	`2mb`	Segments smaller than this are not merged
merge.policy.segments_per_tier	Number of segments to keep in each tier before merging	`10`	Higher values increase memory usage but reduce merge frequency

Example: Configuring Merge Policy in elasticsearch.yml

index:
  merge:
    policy:
      max_merge_size: 10gb
      min_merge_size: 5mb
      segments_per_tier: 15

Note: Adjusting merge policy can significantly impact indexing performance and disk I/O. Monitor cluster health after changes.

4. Using Force Merge

Option	Values	Notes
force_merge	`POST /_forcemerge`	Merges segments in the index to reduce segment count
max_num_segments	`1`	Maximum number of segments to merge to
only_expunge_deletes	`false`	Only merge segments with deleted documents
flush	`true`	Flush after force merge

Example: Force Merge Index

POST /my_index/_forcemerge?max_num_segments=1&amp;flush=true

Note: Force merge can be resource-intensive and should be performed during low-traffic periods.

Warning: Force merge may cause performance degradation if used excessively.

5. Configuring Query Cache

Property	Description	Default Value	Notes
indices.cache.query.enable	Enables or disables query cache	`true`	Set to `false` to disable query cache entirely
indices.cache.query.size	Size of the query cache	`10%`	Can be specified in bytes (e.g., `1gb`) or percentage of heap
indices.cache.query.expire	Expiration time for cached queries	`10m`	Time after which cached queries are evicted

Example: Enabling Query Cache

{
  "indices": {
    "cache": {
      "query": {
        "enable": true,
        "size": "2gb",
        "expire": "30m"
      }
    }
  }
}

Note: Query cache is most effective for read-heavy workloads with repeated queries.

Warning: Enabling query cache can increase heap usage significantly. Monitor memory usage carefully.

6. Using Field Data Cache

Property	Description	Default Value	Notes
indices.fielddata.cache.size	Maximum size of the field data cache	40%	Can be set as a percentage or absolute value (e.g., 1gb)
indices.fielddata.cache.expire	Expiration time for field data cache entries	-	Set to a time value like 60m (60 minutes)

Example: Setting field data cache size

{
  "indices": {
    "fielddata": {
      "cache": {
        "size": "2gb"
      }
    }
  }
}

Note: Field data cache is used for terms, aggregations, and sorting on fields that are not stored in the index. It is important to size this cache appropriately to avoid excessive memory usage.

7. Optimizing Mapping

Mapping Property	Description	Values	Notes
type	Defines the data type of the field	`text`, `keyword`, `integer`, `date`, `boolean`	Use `text` for full-text search, `keyword` for exact matches
analyzer	Defines how to analyze text fields	`standard`, `english`, `custom`	Use `standard` for general text, `english` for English-specific analysis
index	Controls whether the field is indexed for search	`true`, `false`	Set to `false` for fields that should not be searchable
doc_values	Enables or disables storing field data for aggregations	`true`, `false`	Set to `false` for fields not used in aggregations to save memory

Example: Defining a Text Field with Custom Analyzer

{
  "properties": {
    "title": {
      "type": "text",
      "analyzer": "english",
      "index": true,
      "doc_values": true
    }
  }
}

Example: Disabling Indexing for a Non-Searchable Field

{
  "properties": {
    "timestamp": {
      "type": "date",
      "index": false,
      "doc_values": true
    }
  }
}

8. Configuring Circuit Breakers

Property	Description	Default Value	Notes
indices.breaker.fielddata.limit	Threshold for fielddata circuit breaker	60%	Prevents excessive memory usage during fielddata operations
indices.breaker.request.limit	Threshold for request circuit breaker	40%	Prevents excessive memory usage during query execution
indices.breaker.fielddata.limit	Threshold for fielddata circuit breaker	60%	Prevents excessive memory usage during fielddata operations
indices.breaker.request.limit	Threshold for request circuit breaker	40%	Prevents excessive memory usage during query execution

Example: Setting Circuit Breaker Limits

indices:
  breaker:
    fielddata:
      limit: 70%
    request:
      limit: 50%

Note: Circuit breakers prevent out-of-memory errors by limiting memory usage in Elasticsearch operations.

9. Tuning HTTP Thread Pool

Property	Default Value	Description
http.thread_pool.queue_size	`1000`	Maximum number of queued HTTP requests before rejecting new ones
http.thread_pool.threads	`4`	Number of threads in the HTTP thread pool
http.thread_pool.type	`fixed`	Thread pool type: `fixed`, `cached`, or `bounded`

Example: Adjusting HTTP Thread Pool Settings

http:
  thread_pool:
    threads: 8
    queue_size: 2000
    type: fixed

Note: Increasing http.thread_pool.threads can improve throughput under high load, but may increase memory usage and context switching overhead.

10. Configuring Search Thread Pool

Property	Default Value	Description
thread_pool.search.queue_size	`1000`	Maximum number of search tasks that can be queued before rejecting new requests.
thread_pool.search.threads	`4`	Number of threads in the search thread pool. Defaults to 4 or number of CPU cores, whichever is higher.
thread_pool.search.type	`fixed`	Scheduling type for the search thread pool. Options: `fixed`, `dynamic`.

Example: Adjusting Search Thread Pool Settings

thread_pool.search.queue_size: 2000
thread_pool.search.threads: 8
thread_pool.search.type: fixed

Note: Increasing threads beyond the number of CPU cores can lead to context switching overhead and reduced performance.

11. Using Coordinating-Only Nodes

Example: Configuring a Coordinating-Only Node

node.role: [&#34;coordinating_only&#34;]
cluster.name: &#34;my-cluster&#34;
network.host: &#34;0.0.0.0&#34;
http.port: 9200

Property	Value	Description
node.role	`["coordinating_only"]`	Assigns the node to be a coordinating-only node
cluster.name	`"my-cluster"`	Name of the cluster to join
network.host	`"0.0.0.0"`	Host interface to bind to
http.port	`9200`	HTTP port for API access

Note: Coordinating-only nodes do not store data or participate in cluster state management. They are ideal for load distribution and API access.

12. Implementing Hot-Warm-Cold Architecture

Example: Index Lifecycle Management (ILM) Policy

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "30d"
          }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "allocate": {
            "number_of_replicas": 1
          },
          "set_priority": {
            "priority": 50
          }
        }
      },
      "cold": {
        "min_age": "90d",
        "actions": {
          "freeze": {},
          "set_priority": {
            "priority": 0
          }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Phase	Description	Actions	Notes
hot	Active data, frequently accessed	`rollover`	Optimized for write-heavy workloads
warm	Less frequently accessed, but still searchable	`allocate`, `set_priority`	Reduced replicas, optimized for read-heavy workloads
cold	Archived data, rarely accessed	`freeze`, `set_priority`	Freeze index to reduce memory usage
delete	Archived data, no longer needed	`delete`	Permanently removes index after retention period

Note: The rollover action in the hot phase is used to split large indices into smaller ones based on size or age.

Warning: Ensure that the freeze action is compatible with your data access patterns. Frozen indices cannot be updated or written to.