Configuring ML Advanced Features

1. Using Model Snapshots

Property	Description	Values	Notes
model_id	Unique identifier for the model	`string`	Required when restoring a snapshot
snapshot_id	Identifier for the model snapshot	`string`	Used to restore a specific model state
snapshot_name	Name of the snapshot	`string`	Human-readable name for the snapshot
snapshot_timestamp	Timestamp of when the snapshot was created	`date`	Used for versioning and recovery

Example: Restore a Model from Snapshot

{
  "model_id": "my-model-123",
  "snapshot_id": "snapshot-456",
  "restore_settings": {
    "overwrite": true
  }
}

Note: Model snapshots are stored in the elasticsearch cluster and can be used to restore models to a previous state.

Warning: Restoring a model from a snapshot will overwrite the current model configuration.

Snapshot Type	Description	Use Case
`full`	Complete model state including parameters and training data	Full recovery of model
`partial`	Model state with only key parameters	Quick recovery or migration

2. Configuring ML Node

Property	Values	Description
xpack.ml.node.enabled	`true`	Enables or disables ML functionality on a node.
xpack.ml.node.max_ml_threads	`4`	Maximum number of threads used by ML on this node.
xpack.ml.node.max_ml_memory_percent	`25`	Maximum percentage of heap memory used by ML on this node.
xpack.ml.node.max_ml_jvm_heap	`2g`	Maximum JVM heap size for ML processes.

Example: Enabling ML on a Node

xpack:
  ml:
    node:
      enabled: true
      max_ml_threads: 4
      max_ml_memory_percent: 25

3. Setting Custom Rules

Rule Property	Description	Example
rule_id	Unique identifier for the rule	`"custom_rule_1"`
name	Name of the rule	`"Custom anomaly detection"`
description	Explanation of the rule's purpose	`"Detects unusual patterns in log data"`
severity	Severity level (0-100)	`50`
actions	Actions to take when rule triggers	`[{ "name": "email_alert", "params": { "to": "admin@example.com" } }]`

Example: Define Custom Rule

{
  "rule_id": "custom_rule_1",
  "name": "Custom anomaly detection",
  "description": "Detects unusual patterns in log data",
  "severity": 50,
  "actions": [
    {
      "name": "email_alert",
      "params": {
        "to": "admin@example.com"
      }
    }
  ]
}

Note: Custom rules must be defined in the ml_rules index in Elasticsearch.

4. Using Calendars

Property	Description	Values
calendar_id	Unique identifier for the calendar	`string`
description	Human-readable description of the calendar	`string`
timezone	Timezone used for calendar calculations	`string`

Example: Create a Calendar

{
  "calendar_id": "work_hours",
  "description": "Working hours calendar",
  "timezone": "America/New_York"
}

Example: Update Calendar

{
  "calendar_id": "work_hours",
  "description": "Updated working hours calendar",
  "timezone": "America/Los_Angeles"
}

Note: Calendars are used in anomaly detection to define time windows for analysis.

5. Configuring Model Memory Limit

Property	Value	Description
xpack.ml.max_model_memory_limit	`4g`	Maximum memory allocated to ML models. Can be specified in MB, GB, or TB.
xpack.ml.max_model_memory_limit	`1024m`	Alternative format for specifying memory limit in MB.

Example: Setting Model Memory Limit in elasticsearch.yml

xpack:
  ml:
    max_model_memory_limit: 4g

Note: The memory limit applies per node and affects model training and inference performance.

6. Using Categorization

Property	Description	Default	Notes
category_field	Field to use for categorization	`null`	Required field for categorization
max_matched_categories	Maximum number of categories to match	`10`	Prevents excessive matches
category_pattern	Regex pattern to match categories	`null`	Optional regex for category matching

Example: Categorization Configuration

{
  "category_field": "log_level",
  "max_matched_categories": 5,
  "category_pattern": "^(DEBUG|INFO|WARN|ERROR)$"
}

Note: Categorization is useful for grouping log entries by type or severity.

7. Implementing Rare Analysis

Property	Description	Default Value	Notes
ml.anomaly_detectors	Enables anomaly detection capabilities	false	Requires license for advanced ML features
ml.max_anomaly_buckets	Maximum number of buckets for anomaly detection	10000	Higher values may impact performance
ml.max_model_memory_limit	Maximum memory allocated to ML models	1024mb	Adjust based on cluster size and resources

Example: Configure Rare Analysis Job

{
  "job_id": "rare_analysis_job",
  "description": "Detect rare events in system logs",
  "analysis_config": {
    "detectors": [
      {
        "detector_label": "Rare Event Detector",
        "field_name": "event_type",
        "by_field_name": "source"
      }
    ]
  },
  "data_description": {
    "time_field": "@timestamp",
    "time_format": "epoch_millis"
  },
  "model_plot_config": {
    "enabled": true
  }
}

Note: Rare analysis jobs require sufficient data to train models effectively. Ensure logs contain enough variety in event types for meaningful anomaly detection.

Warning: Enabling rare analysis on large datasets may lead to high resource consumption. Monitor system performance closely.

8. Using Population Analysis

Population analysis in the ELK Stack allows you to analyze and understand the distribution of data across different groups or populations within your dataset. This feature is particularly useful for identifying anomalies, trends, and patterns that may not be immediately apparent in raw data.

Parameter	Description	Default Value	Notes
population_id	Unique identifier for the population being analyzed	None	Required field for all population analysis operations
analysis_type	Type of analysis to perform (e.g., distribution, anomaly detection)	anomaly_detection	Supports multiple analysis types including distribution and anomaly detection
time_range	Time window for analysis (e.g., last_24_hours, last_7_days)	last_24_hours	Can be customized to any valid time range format
threshold	Threshold for anomaly detection (e.g., 0.95, 0.99)	0.95	Higher values indicate stricter anomaly detection

Example: Configuring Population Analysis

{
  "population_id": "web_traffic_population",
  "analysis_type": "distribution",
  "time_range": "last_7_days",
  "threshold": 0.95
}

Note: Population analysis requires sufficient data volume to ensure accurate results. For optimal performance, it's recommended to have at least 1000 data points per population.

Warning: Setting the threshold too high may result in missing important anomalies, while setting it too low may generate excessive false positives.

9. Configuring Summary Count

Property	Description	Default Value	Notes
ml.anomaly_detectors.max_summary_count	Maximum number of summary count results to store per detector	`1000`	Controls memory usage and result retention for anomaly detectors
ml.anomaly_detectors.max_summary_count.enabled	Enables or disables the summary count feature	`false`	Set to `true` to enable summary count functionality

Example: Enabling Summary Count in ML Job Configuration

{
  "job_id": "anomaly_detector_job",
  "description": "ML job with summary count enabled",
  "analysis_config": {
    "detectors": [
      {
        "detector_label": "count",
        "function": "count"
      }
    ],
    "summary_count_field_name": "count"
  },
  "analysis_limits": {
    "max_summary_count": 2000
  }
}

Note: The max_summary_count setting is applied per detector and affects how many summary count results are retained for anomaly detection.

10. Using Time Shift

Property	Description	Default Value	Notes
time_shift	Specifies the time shift to apply to the data before analysis	None	Used for forecasting and anomaly detection
time_shift.unit	Unit of time shift (e.g., minutes, hours, days)	hours	Valid values: minutes, hours, days, weeks, months
time_shift.value	Number of units to shift the data	1	Positive values shift data forward, negative backward

Example: Configuring Time Shift in ML Job

{
  "job_id": "ml_job_with_time_shift",
  "time_shift": {
    "unit": "hours",
    "value": 24
  }
}

Note: Time shift is particularly useful for forecasting models to predict future trends based on historical data shifted by a specific interval.

11. Exporting ML Jobs

Property	Description	Values	Notes
job_id	Identifier for the ML job	`string`	Required field for export
output_format	Format of the exported job	`json`, `yaml`	Default is `json`
include_model	Include model definition in export	`true`, `false`	Default is `false`

Example: Exporting ML Job to JSON

POST /_ml/jobs/<job_id>/_export
{
  "output_format": "json",
  "include_model": true
}

Note: The include_model parameter will increase the size of the export and may take longer to process.

12. Monitoring ML Resource Usage

Resource	Metric	Description	Unit
CPU Usage	`ml.node.cpu.percent`	Percentage of CPU used by ML processes	%
Memory Usage	`ml.node.memory.used.bytes`	Total memory used by ML processes	bytes
Heap Usage	`ml.node.heap.used.bytes`	Heap memory used by ML processes	bytes
JVM GC Time	`ml.node.gc.time.ms`	Total garbage collection time in milliseconds	ms
Model Inference Time	`ml.node.inference.time.ms`	Average time taken to perform inference	ms

Example: Monitoring ML Resource Usage via Kibana

{
  "query": {
    "match": {
      "event.category": "ml"
    }
  },
  "aggs": {
    "cpu_usage": {
      "avg": {
        "field": "ml.node.cpu.percent"
      }
    },
    "memory_usage": {
      "avg": {
        "field": "ml.node.memory.used.bytes"
      }
    }
  }
}

Note: Ensure that the ml module is enabled in your Elasticsearch configuration to monitor ML resource usage effectively.