Advanced Performance Tuning

1. Tuning OS Settings

Setting	Description	Recommended Value	Notes
vm.max_map_count	Maximum number of memory map areas a process can have	`262144`	Required for Elasticsearch to function properly
fs.file-max	Maximum number of file handles that the system can open	`65536`	Should be higher than the number of open files in use
net.core.somaxconn	Maximum number of connections that can be queued for acceptance	`1024`	Higher values may help with high connection loads
sysctl.vm.swappiness	Controls how aggressively the kernel swaps memory pages	`1`	Lower values reduce swapping, better for performance
sysctl.vm.dirty_ratio	Percentage of system memory that can be filled with dirty pages before flushing begins	`15`	Higher values can cause performance issues under heavy write loads

Example: Setting vm.max_map_count

sudo sysctl -w vm.max_map_count=262144

Note: These settings should be applied to /etc/sysctl.conf or /etc/sysctl.d/99-elasticsearch.conf to persist across reboots.

2. Configuring Kernel Parameters

Parameter	Description	Recommended Value	Notes
vm.max_map_count	Maximum number of memory map areas a process can have	`262144`	Required for Elasticsearch to function properly
fs.file-max	Maximum number of file handles that the system can open	`65536`	Should be higher than the number of open files in use
net.core.somaxconn	Maximum number of connections that can be queued for acceptance	`1024`	Higher values may help with high connection loads
net.ipv4.ip_unprivileged_port_start	Lowest port number that can be bound by unprivileged processes	`0`	Allows binding to low ports if needed
sysctl.vm.swappiness	Controls how aggressively the kernel swaps memory pages	`1`	Lower values reduce swapping, better for performance

Example: Setting Kernel Parameters

sudo sysctl -w vm.max_map_count=262144
sudo sysctl -w fs.file-max=65536
sudo sysctl -w net.core.somaxconn=1024
sudo sysctl -w net.ipv4.ip_unprivileged_port_start=0
sudo sysctl -w vm.swappiness=1

Note: These settings may need to be made persistent across reboots by adding them to /etc/sysctl.conf or /etc/sysctl.d/99-elasticsearch.conf.

3. Using Memory-Mapped Files

Property	Description	Default Value	Notes
elasticsearch.index.mapper.memory_map.enabled	Enables memory-mapped files for index mapping	`false`	Requires `bootstrap.memory_lock` to be enabled
elasticsearch.index.mapper.memory_map.size	Maximum size of memory-mapped files	`1gb`	Set to `0` to disable

Example: Enabling Memory-Mapped Files

elasticsearch:
  index:
    mapper:
      memory_map:
        enabled: true
        size: 2gb

Note: Memory-mapped files can significantly improve performance for large indices by reducing disk I/O. Ensure sufficient system memory and file descriptor limits are configured.

4. Optimizing Disk I/O

Configuration	Values	Description
indices.store.throttle.type	`node`	Throttling type for store operations. NEW in 8.10
indices.store.throttle.max_bytes_per_second	`20mb`	Maximum bytes per second for store operations
indices.store.throttle.type	`node`	Throttling type for store operations. NEW in 8.10

Example: Disk I/O Throttling Configuration

indices:
  store:
    throttle:
      type: node
      max_bytes_per_second: 20mb

Key Recommendations

Use SSDs for better I/O performance
Throttle store operations to prevent disk saturation
Monitor I/O wait times and adjust throttling limits accordingly

Note: Throttling is particularly important in multi-tenant environments to prevent one index from overwhelming the storage subsystem.

5. Tuning Network Settings

Setting	Description	Default Value	Recommended Value
net.core.rmem_max	Maximum receive socket buffer size in bytes	134217728 (128 MB)	268435456 (256 MB)
net.core.wmem_max	Maximum send socket buffer size in bytes	134217728 (128 MB)	268435456 (256 MB)
net.ipv4.tcp_rmem	Minimum, default, and maximum TCP receive buffer sizes	4096 87380 6291456	4096 65536 268435456
net.ipv4.tcp_wmem	Minimum, default, and maximum TCP send buffer sizes	4096 16384 6291456	4096 65536 268435456
net.ipv4.tcp_mtu_probing	Enables MTU probing to avoid fragmentation	0 (disabled)	1 (enabled)
net.ipv4.ip_forward	Enables IP forwarding for routing	0 (disabled)	1 (enabled) if needed for routing

Example: Apply network settings via sysctl

sudo sysctl -w net.core.rmem_max=268435456
sudo sysctl -w net.core.wmem_max=268435456
sudo sysctl -w net.ipv4.tcp_rmem="4096 65536 268435456"
sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 268435456"
sudo sysctl -w net.ipv4.tcp_mtu_probing=1

Note: These settings should be applied to all nodes in the ELK cluster to ensure consistent performance tuning.

6. Using Dedicated Master Nodes

Note: Dedicated master nodes are essential for maintaining cluster stability and performance in large-scale deployments.

Configuration	Description	Default Value	Notes
node.master	Enables or disables the node from becoming a master node	`true`	Set to `false` for dedicated data nodes
node.data	Enables or disables the node from storing data	`true`	Set to `false` for dedicated master nodes
node.ingest	Enables or disables the node from performing ingest operations	`true`	Set to `false` for dedicated master nodes
node.ml	Enables or disables machine learning capabilities on the node	`false`	Set to `false` for dedicated master nodes

Example: Dedicated Master Node Configuration

node:
  master: true
  data: false
  ingest: false
  ml: false

cluster:
  routing:
    allocation:
      include:
        data: false

Benefits of Dedicated Master Nodes

Improved cluster stability and reduced risk of split-brain scenarios
Better resource allocation for master-eligible tasks
Enhanced scalability for large clusters

Warning: Ensure that at least 3 dedicated master nodes are configured for high availability. Having fewer than 3 can lead to cluster instability.

7. Implementing Data Tiers

Tier	Description	Use Case	Configuration
Hot Tier	Active data with high query frequency	Real-time analytics, monitoring dashboards	`{ "index.lifecycle.name": "hot" }`
Warm Tier	Archived data with moderate query frequency	Historical trend analysis, compliance reporting	`{ "index.lifecycle.name": "warm" }`
Cold Tier	Infrequently accessed data	Long-term archival, legal holds	`{ "index.lifecycle.name": "cold" }`
Delete Tier	Data that is no longer needed	Automated cleanup of expired logs	`{ "index.lifecycle.name": "delete" }`

Example: Setting up a Data Tier Policy

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "30d"
          }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "allocate": {
            "number_of_replicas": 1
          }
        }
      },
      "cold": {
        "min_age": "90d",
        "actions": {
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Note: Data tiers are managed via Index Lifecycle Management (ILM) policies in Elasticsearch.

Benefits of Data Tiers

Improved query performance by keeping active data on faster storage
Reduced storage costs through optimized data placement
Automated lifecycle management for log retention

8. Using Index Sorting

Property	Description	Values	Notes
index.sort.field	Sort field for index	`timestamp`	Must be a numeric or date field
index.sort.order	Sort order	`asc`, `desc`	Default is `asc`
index.sort.mode	Sort mode	`min`, `max`	Applies to multi-value fields
index.sort.nested	Nested field to sort on	`field.nested`	Used for nested object sorting

Example: Configure index sorting

{
  "settings": {
    "index": {
      "sort": {
        "field": "timestamp",
        "order": "asc"
      }
    }
  }
}

Note: Index sorting is most effective for date and numeric fields. It can significantly improve query performance for range queries.

9. Optimizing Shard Size

Parameter	Description	Default	Notes
index.number_of_shards	Number of primary shards for an index	1	Must be set at index creation time
index.number_of_replicas	Number of replica shards per primary shard	1	Can be changed after index creation
index.routing.allocation.total_shards_per_node	Maximum shards allowed per node	-1 (unlimited)	Helps control shard distribution

Note: Shard size affects performance and resource usage. Too many small shards can cause overhead, while too few large shards can lead to uneven distribution.

Example: Setting shard count during index creation

{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "name": { "type": "text" },
      "timestamp": { "type": "date" }
    }
  }
}

Example: Changing replica count after index creation

PUT /my-index/_settings
{
  "index": {
    "number_of_replicas": 2
  }
}

Best Practices for Shard Size

Aim for 20-40GB per shard
Use fewer shards with more data rather than many small shards
Consider using index templates for consistent shard configuration
Monitor cluster health and adjust shard count based on node capacity

10. Configuring Slow Log Thresholds

Setting	Default Value	Description
index.indexing.slowlog.threshold.query.warn	`10s`	Warning threshold for query operations
index.indexing.slowlog.threshold.query.info	`5s`	Info threshold for query operations
index.indexing.slowlog.threshold.query.debug	`2s`	Debug threshold for query operations
index.indexing.slowlog.threshold.query.trace	`500ms`	Trace threshold for query operations
index.indexing.slowlog.threshold.flush.warn	`10s`	Warning threshold for flush operations
index.indexing.slowlog.threshold.flush.info	`5s`	Info threshold for flush operations
index.indexing.slowlog.threshold.flush.debug	`2s`	Debug threshold for flush operations
index.indexing.slowlog.threshold.flush.trace	`500ms`	Trace threshold for flush operations
index.indexing.slowlog.threshold.merge.warn	`10s`	Warning threshold for merge operations
index.indexing.slowlog.threshold.merge.info	`5s`	Info threshold for merge operations
index.indexing.slowlog.threshold.merge.debug	`2s`	Debug threshold for merge operations
index.indexing.slowlog.threshold.merge.trace	`500ms`	Trace threshold for merge operations

Example: Setting Slow Log Thresholds

{
  "index": {
    "indexing": {
      "slowlog": {
        "threshold": {
          "query": {
            "warn": "10s",
            "info": "5s",
            "debug": "2s",
            "trace": "500ms"
          },
          "flush": {
            "warn": "10s",
            "info": "5s",
            "debug": "2s",
            "trace": "500ms"
          },
          "merge": {
            "warn": "10s",
            "info": "5s",
            "debug": "2s",
            "trace": "500ms"
          }
        }
      }
    }
  }
}

Note: Slow log thresholds can be set at the index or cluster level. Adjust these values based on your system's performance characteristics and monitoring needs.

11. Using Adaptive Replica Selection

Adaptive Replica Selection (ARS) is a feature in Elasticsearch that dynamically selects the best replica shard to serve read requests based on performance metrics. This helps optimize query latency and resource utilization.

Configuration Option	Description	Default Value	Notes
index.routing.allocation.awareness.attributes	Defines the attributes used for awareness during shard allocation.	`null`	Must be set for ARS to function properly in multi-zone environments.
index.routing.allocation.awareness.force	Enforces awareness constraints for shard allocation.	`null`	Can be set to `zone` or specific attribute names.
index.routing.allocation.adaptive_replica_selection.enabled	Enables or disables Adaptive Replica Selection.	`false`	Set to `true` to enable ARS.
index.routing.allocation.adaptive_replica_selection.threshold	Threshold for considering replica shards for read requests.	`50`	Replica shards with latency below this threshold are selected.

Example: Enabling Adaptive Replica Selection

{
  "settings": {
    "index.routing.allocation.adaptive_replica_selection.enabled": true,
    "index.routing.allocation.awareness.attributes": [
      "zone"
    ],
    "index.routing.allocation.awareness.force.zone.values": [
      "us-west-1a",
      "us-west-1b"
    ]
  }
}

Note: ARS is available starting from Elasticsearch version 7.10.

Warning: Ensure that all nodes in the cluster have consistent awareness settings to avoid shard allocation issues.

12. Implementing Query Result Caching

Configuration Option	Default Value	Description	Notes
elasticsearch.cache.query.enabled	`true`	Enables or disables query result caching	Set to `false` to disable caching entirely
elasticsearch.cache.query.ttl	`1h`	Time-to-live for cached query results	Can be specified in seconds, minutes, hours, or days
elasticsearch.cache.query.max_size	`100mb`	Maximum size of the query cache	Cache will evict least recently used entries when this limit is reached

Example: Enabling Query Caching in elasticsearch.yml

elasticsearch:
  cache:
    query:
      enabled: true
      ttl: 1h
      max_size: 100mb

Note: Query caching is most effective for queries that are repeated frequently with the same parameters. It's not recommended for highly dynamic queries or those involving random data.

Warning: Setting max_size too low may cause frequent cache misses, while setting it too high can lead to memory pressure on the cluster.