Optimizing Search Performance

1. Using Filter Context vs Query Context

Context	Description	Use Case	Performance Impact
Filter Context	Used for filtering documents based on boolean conditions. Does not calculate relevance scores.	When you need to filter results without scoring, such as "find all logs with status 200"	Highly optimized, caches results, faster execution
Query Context	Used for full-text search and relevance scoring. Calculates TF-IDF scores.	When you need to rank results by relevance, such as "find logs containing 'error' and rank by severity"	Slower due to scoring, less cacheable

Example: Filter Context Usage

{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "status": 200
          }
        }
      ]
    }
  }
}

Example: Query Context Usage

{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "message": "error"
          }
        }
      ]
    }
  }
}

Note: Filter context is ideal for exact matches and ranges, while Query context is best for full-text search with relevance scoring.

2. Optimizing Bool Queries

Property	Description	Notes
must	Queries that must match for a document to be included	Used for filtering documents based on required conditions
should	Queries that should match for a document to be included	Increases relevance score when matched, but not required
must_not	Queries that must NOT match for a document to be included	Excludes documents matching these conditions
filter	Queries that must match, but without scoring	More efficient than `must` for filtering

Example: Bool Query with must and should

{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "status": "published"
          }
        }
      ],
      "should": [
        {
          "match": {
            "title": "Elasticsearch"
          }
        }
      ]
    }
  }
}

Example: Bool Query with filter clause

{
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "publish_date": {
              "gte": "2023-01-01"
            }
          }
        }
      ]
    }
  }
}

Note: Using filter instead of must improves performance because it does not calculate scores for matching documents.

Warning: Avoid deeply nested bool queries as they can impact query performance.

3. Using Constant Score Query

Property	Description	Example
constant_score	Wraps a query and returns a constant score equal to the boost for all documents matching the query	`{ "constant_score": { "filter": { "term": { "status": "published" } }, "boost": 2.0 } }`

Example: Constant Score Query Usage

{
  "query": {
    "constant_score": {
      "filter": {
        "range": {
          "age": {
            "gte": 25,
            "lte": 40
          }
        }
      },
      "boost": 1.5
    }
  }
}

Note: The constant_score query is useful when you want to apply a boost to documents matching a filter without affecting their relevance scores.

4. Configuring Index Sorting

Setting	Description	Values	Notes
index.sort.field	Field to sort by	`string`	Must be a numeric or date field
index.sort.order	Sort order	`asc` or `desc`	Default is `asc`
index.sort.mode	Sorting mode	`min`, `max`	Applicable to multi-value fields
index.sort.nested	Nested field sorting	`object_path`	For nested object fields

Example: Sorting by timestamp

{
  "settings": {
    "index.sort.field": "timestamp",
    "index.sort.order": "desc"
  }
}

Note: Sorting is applied at index time and affects search performance for range queries.

5. Using Preference Parameter

Parameter	Description	Values	Notes
preference	Controls how search requests are distributed across shards	`_shards:1`, `_shards:2`, `_local`, `_only_nodes:`	Can be used to optimize performance by reducing shard distribution

Example: Setting preference for search

{
  "query": {
    "match": {
      "message": "error"
    }
  },
  "preference": "_shards:1"
}

Note: The preference parameter can help ensure consistent results when searching across multiple shards by specifying a shard or node to prioritize.

6. Implementing Query Timeout

Configuration	Value	Description
search.max_request_duration	`10s`	Maximum time a search request can take before timing out
search.max_query_time	`5s`	Maximum time a single query can run before timing out
search.max_scroll_time	`30s`	Maximum time a scroll operation can run before timing out

Example: Setting Query Timeout in Elasticsearch

{
  "query": {
    "match_all": {}
  },
  "timeout": "5s"
}

Note: Setting timeout in the query body overrides the default search.max_query_time setting.

7. Using Track Total Hits

Option	Values	Description
track_total_hits	`true` \| `false` \| `number`	Controls whether to track the total number of hits for each search request. Setting to `false` improves performance by not calculating the total hit count, which is useful when only a subset of results is needed.

Example: Setting track_total_hits to false

{
  "query": {
    "match_all": {}
  },
  "track_total_hits": false
}

Note: When track_total_hits is set to false, the total hit count will be inaccurate, and the response will not include the hits.total field. This can significantly improve query performance for large datasets.

8. Optimizing Wildcard Queries

Option	Values	Notes
wildcard.query	`user.*`	Pattern to match against field values
wildcard.case_insensitive	`false`	Whether to perform case-insensitive matching
wildcard.rewrite	`constant_score_boolean`	How to rewrite the query for performance

Example: Basic wildcard query

{
  "query": {
    "wildcard": {
      "user.name": "*john*"
    }
  }
}

Example: Case insensitive wildcard query

{
  "query": {
    "wildcard": {
      "user.name": {
        "value": "*JOHN*",
        "case_insensitive": true
      }
    }
  }
}

Note: Wildcard queries can be slow if used with leading wildcards like *john*. Consider using prefix or term queries for better performance.

Warning: Using wildcards at the beginning of a pattern (e.g., *john) can cause performance issues due to the need to scan all terms in the index.

9. Using Search Templates

Property	Description	Example
templateId	Unique identifier for the search template	`"my-search-template"`
source	Search template content as a string or file path	`{ "query": { "match": { "field": "{{query_string}}" } } }`
lang	Template language (e.g., "mustache")	`"mustache"`

Example: Registering a Search Template

{
  "id": "user-search-template",
  "template": {
    "query": {
      "match": {
        "username": "{{user_query}}"
      }
    }
  }
}

Example: Using a Search Template in a Query

GET /_search/template
{
  "id": "user-search-template",
  "params": {
    "user_query": "john_doe"
  }
}

Note: Search templates help reduce query parsing overhead and improve performance by reusing predefined structures.

10. Caching Aggregations

Property	Description	Default	Notes
cache.max_size	Maximum number of cached aggregation results	1000	Controls memory usage for cached aggregations
cache.ttl	Time-to-live for cached results in milliseconds	300000	Defaults to 5 minutes
cache.enabled	Whether to enable aggregation caching	true	Set to false to disable caching entirely

Example: Enabling Aggregation Caching

{
  "cache": {
    "enabled": true,
    "max_size": 2000,
    "ttl": 600000
  }
}

Note: Caching is most effective for expensive aggregations that are frequently reused.

Warning: Large cache sizes can lead to high memory consumption. Monitor heap usage carefully.

11. Using Composite Aggregation

Property	Description	Values	Notes
composite	Defines the composite aggregation	`{"sizes": 100, "sources": [...]}`	Used for paginating large result sets
sources	Defines the fields to group by	`[{"field": "user.id", "order": "asc"}]`	Supports multiple fields
size	Number of buckets to return per batch	`100`	Default is 100, max is 10000

Example: Composite Aggregation

{
  "aggs": {
    "composite_agg": {
      "composite": {
        "sizes": 100,
        "sources": [
          {
            "user_id": {
              "terms": {
                "field": "user.id"
              }
            }
          }
        ]
      },
      "aggs": {
        "total_requests": {
          "sum": {
            "field": "requests"
          }
        }
      }
    }
  }
}

Note: Composite aggregation is ideal for paginating large result sets without using scroll or search_after.

12. Implementing Shard Filtering

Filter Type	Description	Usage	Notes
query_string	Allows filtering using query string syntax	`"query_string": { "query": "field:value" }`	Supports wildcard and range queries
term	Matches exact terms in indexed fields	`"term": { "status.keyword": "active" }`	Fastest for exact matches
terms	Matches any of multiple values in a field	`"terms": { "category.keyword": ["tech", "dev"] }`	Efficient for multiple value lookups
range	Filters based on numeric or date ranges	`"range": { "timestamp": { "gte": "2023-01-01" } }`	Supports `gte`, `lte`, `gt`, `lt`
bool	Combines multiple filters with logical operators	`"bool": { "must": [...], "should": [...] }`	Allows complex filter combinations

Example: Using query_string filter

{
  "query": {
    "bool": {
      "filter": [
        {
          "query_string": {
            "query": "status:active AND (env:prod OR env:staging)"
          }
        }
      ]
    }
  }
}

Example: Using term filter

{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "status.keyword": "active"
          }
        }
      ]
    }
  }
}

Example: Using range filter

{
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "timestamp": {
              "gte": "2023-01-01T00:00:00Z",
              "lt": "2023-12-31T23:59:59Z"
            }
          }
        }
      ]
    }
  }
}

Note: Shard filtering is most effective when used with fields that are frequently filtered and have good cardinality.