Implementing Machine Learning Features
1. Understanding ML Capabilities
| Capability | Description | Use Case |
|---|---|---|
| Anomaly Detection | Detects unusual patterns in time-series data | Monitoring system logs for unexpected behavior |
| Data Frame Analytics | Performs statistical analysis on large datasets | Identifying trends in user behavior or system metrics |
| Forecasting | Predicts future values based on historical data | Anticipating system load or resource usage |
2. Creating Anomaly Detection Jobs
Example: Create Anomaly Detection Job
{
"job_id": "anomaly-detection-job",
"description": "Detect anomalies in system logs",
"analysis_config": {
"detectors": [
{
"detector_label": "Log error rate",
"field_name": "error_count"
}
]
},
"data_description": {
"time_field": "@timestamp"
}
}3. Configuring Detectors
| Detector Type | Parameters | Description |
|---|---|---|
| Metric | field_name, partition_field |
Monitors numeric fields for anomalies |
| Categorization | category_field, max_categoricals |
Identifies unusual categories in data |
| Rare | field_name, time_zone |
Detects rare occurrences of values |
4. Setting Bucket Span
| Bucket Span Value | Time Range | Use Case |
|---|---|---|
| 1m | 1 minute | Real-time monitoring of logs |
| 5m | 5 minutes | Balanced performance and accuracy |
| 1h | 1 hour | High-level trend analysis |
5. Using Influencers
Example: Define Influencer Field
{
"job_id": "influencer-job",
"analysis_config": {
"detectors": [
{
"detector_label": "Error count",
"field_name": "error_count"
}
],
"influencers": [
"host",
"user_agent"
]
}
}6. Viewing Anomaly Results
Workflow: Analyze Anomalies
- Open Kibana ML section
- Navigate to Anomaly Detection Jobs
- Select job and view results in timeline
- Filter by anomaly score or influencer
7. Creating Forecasts
8. Using Data Frame Analytics
| Analytics Type | Input Data | Output |
|---|---|---|
| Statistical Analysis | Time-series logs or metrics | Summary statistics and trends |
| Clustering | Unlabeled datasets | Groupings of similar data points |
9. Training Models
Example: Train Model with Data Frame
{
"job_id": "training-job",
"data_frame": {
"source": {
"index": "logs"
},
"analysis": {
"regression": {
"dependent_variable": "response_time"
}
}
}
}10. Evaluating Model Performance
| Metric | Definition | Threshold |
|---|---|---|
| Precision | True positives / (True positives + False positives) | > 0.8 |
| Recall | True positives / (True positives + False negatives) | > 0.8 |
| F1 Score | 2 * (Precision * Recall) / (Precision + Recall) | > 0.8 |
11. Deploying Trained Models
Workflow: Deploy ML Model
- Create model definition in Kibana
- Upload trained model to Elasticsearch
- Assign model to job or analytics
- Start job and monitor performance