Configuring Analyzers and Tokenizers

1. Understanding Analysis Process

The analysis process in the ELK Stack involves breaking down text into tokens and applying various analyzers to transform and index content. This process is crucial for effective search and retrieval of data.

Component	Description	Usage
Analyzer	Processes text to generate tokens for indexing and searching.	Used in mapping definitions to define how text fields should be analyzed.
Tokenizer	Breaks text into individual tokens (words, phrases, etc.).	Part of the analyzer pipeline; determines how text is segmented.
Token Filter	Modifies or transforms tokens (e.g., lowercase, remove stop words).	Applied after tokenization to refine tokens in the analysis pipeline.

Example: Defining a Custom Analyzer

{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_custom_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": [
            "lowercase",
            "stop"
          ]
        }
      }
    }
  }
}

Note: The standard tokenizer splits text on word boundaries, and filters like lowercase and stop help normalize and clean the tokens for better search performance.

2. Using Standard Analyzer

Property	Value	Description
analyzer	`standard`	The standard analyzer is the default analyzer in Elasticsearch. It's designed to work well with most languages and provides a good balance between performance and functionality.
max_token_length	`255`	Sets the maximum token length. Tokens longer than this will be split into smaller tokens.
stopwords	`_default_`	Uses the default list of stop words for English. Stop words are filtered out during tokenization.

Example: Configure Standard Analyzer in Index Settings

{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_standard_analyzer": {
          "type": "standard",
          "max_token_length": 255,
          "stopwords": "_default_"
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "content": {
        "type": "text",
        "analyzer": "my_standard_analyzer"
      }
    }
  }
}

Note: The standard analyzer is suitable for general-purpose text analysis and does not require any additional configuration in most cases.

3. Using Keyword Analyzer

Property	Value	Description
type	`keyword`	Analyzer type for exact match searches
buffer	`256`	Maximum token length in characters

Example: Configure Keyword Analyzer

{
  "analysis": {
    "analyzer": {
      "my_keyword_analyzer": {
        "type": "keyword",
        "buffer": 256
      }
    }
  }
}

Note: The keyword analyzer treats the entire input as a single token, making it ideal for exact matches and filtering.

4. Using Simple Analyzer

Property	Value	Description
type	`simple`	Analyzer type for simple tokenization
max_token_length	`255`	Maximum length of a token in characters

Example: Configuring Simple Analyzer

{
  "analysis": {
    "analyzer": {
      "my_simple_analyzer": {
        "type": "simple",
        "max_token_length": 255
      }
    }
  }
}

Note: The simple analyzer splits text on whitespace and converts tokens to lowercase.

5. Using Whitespace Analyzer

Property	Value	Description
analyzer	`whitespace`	Analyzer that splits text on whitespace characters
type	`whitespace`	Tokenizes text on whitespace and removes punctuation

Example: Configuring Whitespace Analyzer

{
  "analysis": {
    "analyzer": {
      "my_whitespace_analyzer": {
        "type": "whitespace"
      }
    }
  }
}

Note: The whitespace analyzer is useful for indexing text where tokenization based on whitespace is sufficient and punctuation should not be treated as delimiters.

6. Creating Custom Analyzers

Property	Description	Example
analyzer.name	Name of the custom analyzer	`"my_custom_analyzer"`
analyzer.tokenizer	Tokenizer to use for tokenization	`"standard"`
analyzer.filter	Array of filters to apply	`["lowercase", "stop"]`

Example: Define a Custom Analyzer

{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_custom_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": [
            "lowercase",
            "stop"
          ]
        }
      }
    }
  }
}

Note: Custom analyzers must be defined in the index settings before any documents are indexed.

7. Configuring Tokenizers

Property	Description	Default	Notes
type	Tokenizer type to use	`standard`	Must be one of `standard`, `keyword`, `whitespace`, or custom
max_token_length	Maximum length of a token in characters	`255`	Prevents excessively long tokens from being indexed
max_buffer_size	Maximum buffer size for tokenization in bytes	`1024`	Controls memory usage during tokenization

Example: Custom Tokenizer Configuration

{
  "tokenizer": {
    "type": "custom",
    "max_token_length": 500,
    "max_buffer_size": 2048
  }
}

Note: Custom tokenizers require additional configuration in the Elasticsearch index settings.

Warning: Setting max_token_length too high can lead to performance issues.

8. Using Token Filters

Filter	Description	Configuration Example	Notes
lowercase	Converts tokens to lowercase	`{ "type": "lowercase" }`	Commonly used as a first filter to normalize text
stop	Removes stop words from tokens	`{ "type": "stop", "stopwords": ["the", "a", "an"] }`	Can be customized with a list of stop words or use built-in lists
stemmer	Applies stemming algorithm to tokens	`{ "type": "stemmer", "language": "english" }`	Reduces words to their root form (e.g., "running" → "run")
synonym	Replaces tokens with synonyms	`{ "type": "synonym", "synonyms_path": "analysis/synonyms.txt" }`	Supports synonym files or inline synonym definitions
edge_ngram	Generates n-grams from the beginning of tokens	`{ "type": "edge_ngram", "min_gram": 2, "max_gram": 5 }`	Useful for autocomplete and prefix matching

Example: Configuring a Custom Token Filter

{
  "analysis": {
    "analyzer": {
      "custom_analyzer": {
        "tokenizer": "standard",
        "filter": [
          "lowercase",
          "stop",
          "stemmer"
        ]
      }
    }
  }
}

Note: Token filters are applied in the order they appear in the filter array. Filters can be reused across multiple analyzers.

9. Using Character Filters

Filter	Description	Configuration	Notes
html_strip	Removes HTML tags and entities from text	`{ "type": "html_strip" }`	Useful for cleaning content before analysis
mapping	Replaces characters or character sequences with specified replacements	`{ "type": "mapping", "mappings": [ "foo => bar" ] }`	Can be used to normalize text or handle special characters
pattern_replace	Replaces text matching a regex pattern with a replacement string	`{ "type": "pattern_replace", "pattern": "foo", "replacement": "bar" }`	Allows for complex text transformations

Example: Using html_strip filter

{
  "analysis": {
    "analyzer": {
      "my_analyzer": {
        "type": "custom",
        "char_filter": [
          {
            "type": "html_strip"
          }
        ],
        "tokenizer": "standard",
        "filter": [
          "lowercase"
        ]
      }
    }
  }
}

Example: Using mapping filter

{
  "analysis": {
    "analyzer": {
      "my_analyzer": {
        "type": "custom",
        "char_filter": [
          {
            "type": "mapping",
            "mappings": [
              "£ => GBP",
              "€ => EUR"
            ]
          }
        ],
        "tokenizer": "standard",
        "filter": [
          "lowercase"
        ]
      }
    }
  }
}

10. Testing Analyzers

Example: Test Analyzer with Sample Text

{
  "analyzer": "my_custom_analyzer",
  "text": "The quick brown fox jumps over the lazy dog"
}

Property	Description	Values
analyzer	Name of the analyzer to test	`my_custom_analyzer`
text	Sample text to tokenize	`The quick brown fox jumps over the lazy dog`

Note: Use the analyze API endpoint to test analyzers without indexing data.

Example: Analyze Response

{
  "tokens": [
    {
      "token": "the",
      "start_offset": 0,
      "end_offset": 3,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "quick",
      "start_offset": 4,
      "end_offset": 9,
      "type": "<ALPHANUM>",
      "position": 1
    }
  ]
}

11. Configuring Language Analyzers

Analyzer	Language	Description	Configuration Example
standard	Generic	Default analyzer; suitable for most use cases. Tokenizes text into terms based on Unicode text segmentation rules.	`{ "analyzer": { "my_standard_analyzer": { "type": "standard" } } }`
english	English	Optimized for English text. Removes common English stop words and applies Porter stemmer.	`{ "analyzer": { "my_english_analyzer": { "type": "english" } } }`
german	German	Designed for German language text. Includes stop words and stemmer specific to German.	`{ "analyzer": { "my_german_analyzer": { "type": "german" } } }`
french	French	Optimized for French text. Removes stop words and applies French stemmer.	`{ "analyzer": { "my_french_analyzer": { "type": "french" } } }`
spanish	Spanish	Designed for Spanish text. Includes stop words and stemmer specific to Spanish.	`{ "analyzer": { "my_spanish_analyzer": { "type": "spanish" } } }`
chinese	Chinese	Handles Chinese text by segmenting into characters or words using ICU tokenizer.	`{ "analyzer": { "my_chinese_analyzer": { "type": "chinese" } } }`

Example: Using a Custom Language Analyzer

{
  "analysis": {
    "analyzer": {
      "custom_english_analyzer": {
        "type": "english",
        "stopwords": ["the", "a", "an"],
        "stemmer": "minimal"
      }
    }
  }
}

Note: Language analyzers are best suited for full-text search and indexing in Elasticsearch. They can be customized further by combining tokenizers, token filters, and character filters.

12. Setting Normalizers

Property	Description	Values	Notes
type	Defines the normalizer type	`custom`, `html_strip`, `lowercase`, `uppercase`	Default is `lowercase`
filter	Array of token filters to apply	`asciifolding`, `stop`, `trim`	Used for additional processing

Example: Custom Normalizer Configuration

{
  "analysis": {
    "normalizer": {
      "my_custom_normalizer": {
        "type": "custom",
        "filter": ["asciifolding", "lowercase"]
      }
    }
  }
}

Note: Normalizers are applied after tokenization and before indexing. They are useful for standardizing text format without affecting token structure.