Optimizing Indexing Performance

1. Tuning Refresh Interval

Setting	Default Value	Description	Impact
index.refresh_interval	`1s`	How often the index is refreshed to make new documents searchable	Lower values increase search latency but improve near real-time searchability
index.refresh_interval	`-1`	Disables automatic refreshes; requires manual refresh via API	Best for batch indexing or when search latency is not critical

Example: Setting refresh interval to 30 seconds

{
  "index": {
    "refresh_interval": "30s"
  }
}

Note: Setting refresh_interval to -1 disables automatic refreshes, which can significantly reduce CPU and I/O load but makes documents searchable only after a manual refresh.

2. Tuning Bulk Request Size

Setting	Value	Description
index.bulk.request.size	`1000`	Number of documents per bulk request
index.bulk.max.requests	`50`	Maximum concurrent bulk requests
index.bulk.max.size	`10mb`	Maximum size of a bulk request

Example: Configuring Bulk Request Size in Elasticsearch

{
  "index": {
    "bulk": {
      "request": {
        "size": 1000,
        "max.requests": 50,
        "max.size": "10mb"
      }
    }
  }
}

Note: Adjusting bulk request size can significantly impact indexing throughput and memory usage. Larger requests reduce overhead but increase latency.

3. Using Async Indexing

Option	Values	Notes
index.async	`true`	Enables asynchronous indexing
index.async.delay	`500ms`	Delay before starting async index operation
index.async.max_concurrent	`10`	Maximum concurrent async indexing operations

Example: Configuring Async Indexing

index:
  async:
    enabled: true
    delay: 500ms
    max_concurrent: 10

Note: Async indexing improves performance by offloading index creation to background threads.

Warning: Setting max_concurrent too high may lead to resource exhaustion.

4. Increasing Indexing Buffer

Property	Value	Description
indexing.buffer.size	`512mb`	Size of the buffer used for indexing operations
indexing.thread.count	`4`	Number of threads used for indexing

Example: Configuring Indexing Buffer

indexing:
  buffer:
    size: 512mb
  thread:
    count: 4

Note: Increasing the buffer size can improve indexing throughput but may increase memory usage.

5. Reducing Replica Count

Reducing the number of replica shards in an Elasticsearch index can significantly improve indexing performance by decreasing the load on cluster nodes and reducing resource contention.

Parameter	Description	Default Value	Notes
index.number_of_replicas	Number of replica shards per primary shard	`1`	Set to `0` for performance optimization in development or low-availability environments

Example: Setting Replica Count to Zero

{
  "settings": {
    "number_of_replicas": 0
  }
}

Note: Setting number_of_replicas to zero reduces fault tolerance and availability. Only use in environments where high availability is not required.

Warning: Reducing replica count to zero may lead to data loss if a node fails and there are no replicas to recover from.

6. Optimizing Document Structure

Field	Type	Recommendation
_id	`string`	Use auto-generated IDs unless you have a specific need for custom IDs. Custom IDs can lead to uneven distribution of documents across shards.
timestamp	`date`	Ensure timestamps are in ISO 8601 format for optimal parsing and querying. Use `date` type for efficient time-based queries.
message	`text`	Use `text` type for full-text search. Avoid using `text` fields for filtering or sorting unless necessary.
status	`keyword`	Use `keyword` for exact matches and aggregations. This field is ideal for filtering logs by status code or level.

Example: Optimized Document Structure

{
  "_id": "abc123",
  "timestamp": "2026-04-05T12:34:56.789Z",
  "message": "User login successful",
  "status": "INFO"
}

Note: Avoid deeply nested objects in your documents. Elasticsearch performs better with flattened, denormalized structures.

Warning: Using text fields for filtering or sorting can cause performance degradation. Prefer keyword fields for such operations.

7. Using Routing

Property	Description	Values	Notes
_routing	Used to control document routing to specific shards	`string`	Improves query performance by reducing the number of shards to search
_routing.required	Enforces that routing value must be provided when indexing documents	`true` or `false`	Defaults to `false`

Example: Indexing with Routing

{
  "_routing": "user-123",
  "message": "Hello World"
}

Example: Querying with Routing

{
  "query": {
    "match": {
      "message": "Hello World"
    }
  },
  "routing": "user-123"
}

Note: When using routing, ensure that the same routing value is used for indexing and querying to avoid data inconsistency.

8. Tuning Thread Pool Settings

Setting	Description	Default Value	Notes
thread_pool.index.queue_size	Number of queued index requests before rejecting new ones	200	Higher values allow more concurrent indexing but increase memory usage
thread_pool.index.threads	Number of threads in the index thread pool	1	Adjust based on CPU cores and I/O capacity
thread_pool.index.type	Thread pool type (e.g., fixed, scaling)	fixed	Fixed thread pool size for predictable performance
thread_pool.search.queue_size	Number of queued search requests before rejecting new ones	1000	Higher values allow more concurrent searches but increase memory usage
thread_pool.search.threads	Number of threads in the search thread pool	1	Adjust based on CPU cores and I/O capacity

Example: Updating Thread Pool Settings

{
  "thread_pool": {
    "index": {
      "queue_size": 300,
      "threads": 2
    },
    "search": {
      "queue_size": 1500,
      "threads": 2
    }
  }
}

Note: Adjust thread_pool.index.threads and thread_pool.search.threads based on your system's CPU cores to avoid overloading the system.

9. Configuring Translog Durability

Setting	Values	Description
index.translog.durability	`request` \| `sync` \| `async`	Controls how often the translog is flushed to disk. `request` flushes after every indexing request, `sync` ensures durability with each operation, and `async` flushes asynchronously for better performance.

Example: Setting translog durability to async

{
  "settings": {
    "index": {
      "translog": {
        "durability": "async"
      }
    }
  }
}

Note: async mode provides better performance but may lose recent operations in case of a crash.

10. Disabling Swapping

Configuration	Value	Description
vm.swappiness	`0`	Disables swapping to disk for better performance

Example: Setting vm.swappiness to 0

# Temporary setting
echo 0 &gt; /proc/sys/vm/swappiness

# Permanent setting in /etc/sysctl.conf
echo "vm.swappiness = 0" &gt;&gt; /etc/sysctl.conf
sysctl -p

Note: Disabling swapping can lead to OutOfMemory errors if system memory is insufficient.

11. Optimizing Ingest Pipelines

Property	Description	Default Value	Notes
pipeline.id	Unique identifier for the pipeline	None	Required field
pipeline.description	Description of the pipeline's purpose	None	Optional
pipeline.processors	List of processors to apply to the document	None	Required field
pipeline.on_failure	Actions to take if a processor fails	None	Optional

Example: Basic Ingest Pipeline Configuration

{
  "id": "my-pipeline",
  "description": "Process log data for analysis",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{TIMESTAMP}"]
      }
    },
    {
      "date": {
        "field": "timestamp",
        "formats": ["yyyy-MM-dd HH:mm:ss"]
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "field": "error",
        "value": "Pipeline failed"
      }
    }
  ]
}

Note: Use grok and date processors to parse and format log data efficiently.

Best Practices for Ingest Pipeline Optimization

Minimize the number of processors in a single pipeline
Use if conditions to apply processors only when necessary
Pre-process data to reduce the load on downstream processors
Monitor pipeline performance using elasticsearch monitoring tools

12. Monitoring Indexing Rate

Metric	Description	Unit	Collection Method
indexing.rate	Number of documents indexed per second	docs/sec	Through Elasticsearch Monitoring API or Logstash metrics
indexing.throughput	Total volume of data indexed per second	MB/sec	Calculated from document size and indexing.rate
indexing.delay	Time between document ingestion and indexing completion	ms	Measured via timestamp differences in logs or Elasticsearch stats

Example: Fetching Indexing Rate via Elasticsearch API

GET /_cluster/stats

Note: The indexing.rate is a key indicator of indexing performance and can help identify bottlenecks in your ELK pipeline.