Skip to main content

Managing Production Operations

1. Planning Cluster Capacity

Note: Proper cluster capacity planning ensures optimal performance and scalability for your ELK Stack deployment.
Resource Description Recommendation
Elasticsearch Nodes Primary units processing data and serving queries Start with 3 nodes for high availability; scale based on data volume and query load
Memory (RAM) Allocated to Elasticsearch JVM heap Set to 50% of total system RAM, up to 32GB; avoid exceeding 50% to prevent GC overhead
Storage Used for indices and snapshots Use SSDs with sufficient IOPS; plan for 2x data size to accommodate index growth
CPU Cores Processing power for indexing and search operations Minimum 4 cores per node; more cores improve parallel processing capability

Example: Sample Capacity Planning Configuration

# elasticsearch.yml
cluster.name: production-cluster
node.name: es-node-1
network.host: 0.0.0.0
http.port: 9200

# Memory settings (50% of system RAM)
bootstrap.memory_lock: true
indices.memory.index_buffer_size: 25%

Key Considerations for Cluster Scaling

  • Start with 3 nodes for redundancy and fault tolerance
  • Monitor GC overhead to ensure JVM heap is properly sized
  • Plan for index lifecycle management (ILM) to handle data retention
  • Use shard allocation strategies to balance load across nodes

Capacity Planning Workflow

  1. Determine current and projected data volume (GB/day)
  2. Estimate query patterns and concurrent user load
  3. Select appropriate node types (e.g., io, memory, master)
  4. Configure memory and storage for optimal performance
  5. Run load tests to validate capacity assumptions

2. Implementing Backup Strategy

Example: Elasticsearch Snapshot Configuration

{
  "type": "fs",
  "location": "/backups/elasticsearch"
}
Configuration Description Values Notes
repository.type Repository type for snapshots fs, s3, gcs, azure File system, S3, Google Cloud Storage, Azure Blob Storage
repository.location Path or bucket for snapshots /backups/elasticsearch Must be accessible to all nodes
snapshot.ignore_unavailable Ignore unavailable indices true, false Default: false
Note: Ensure the snapshot repository is configured before creating snapshots.

Example: Create Snapshot

PUT /_snapshot/my_backup/snapshot_1
{
  "indices": "index-2024-*",
  "ignore_unavailable": true,
  "include_global_state": false
}

Example: Restore Snapshot

POST /_snapshot/my_backup/snapshot_1/_restore
{
  "indices": "index-2024-*",
  "rename_pattern": "index-(.*)",
  "rename_replacement": "restored-index-$1"
}

Backup Strategy Best Practices

  • Use dedicated backup repositories (e.g., S3 or GCS)
  • Schedule regular snapshots with elasticsearch-snapshot or cron jobs
  • Test restore procedures regularly
  • Ensure sufficient disk space in backup repositories

3. Performing Rolling Upgrades

Rolling Upgrade Process

  1. Prepare the new version of Elasticsearch, Logstash, and Kibana
  2. Start the upgrade process on one node at a time
  3. Ensure cluster health remains green during the upgrade
  4. Verify that all indices are properly migrated
  5. Continue until all nodes are upgraded
Component Upgrade Method Notes
Elasticsearch rolling_upgrade Must be performed one node at a time
Logstash rolling_restart Reconfigure pipelines after restart
Kibana rolling_restart Ensure compatibility with new Elasticsearch version
Note: Always backup indices before starting the upgrade process.

Example: Upgrade Elasticsearch Node

sudo systemctl stop elasticsearch
sudo yum update elasticsearch
sudo systemctl start elasticsearch

4. Managing Index Growth

Strategy Description Implementation Notes
Index Lifecycle Management (ILM) Automatically manage index lifecycle based on age, size, or document count
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "30d"
          }
        }
      },
      "warm": {
        "actions": {
          "allocate": {
            "include": {
              "box_type": "warm"
            }
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
ILM policies can be applied to index templates
Index Rollover Split large indices into smaller ones to manage performance and storage
{
  "conditions": {
    "max_size": "50GB",
    "max_age": "30d"
  }
}
Use max_size or max_age to trigger rollover
Index Templates Define index settings and mappings to ensure consistency across indices
{
  "index_patterns": ["logs-*"],
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "timestamp": {
        "type": "date"
      }
    }
  }
}
Templates apply to new indices matching the pattern

Example: Setting up ILM Policy

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "set_priority": {
            "priority": 50
          },
          "allocate": {
            "number_of_replicas": 1,
            "include": {
              "box_type": "warm"
            }
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
Note: ILM policies must be attached to index templates or existing indices to take effect.
Warning: Deleting old indices can lead to data loss if not properly backed up or archived.

Best Practices for Index Growth Management

  • Use ILM policies to automate index lifecycle transitions
  • Set appropriate size and age thresholds for rollover
  • Archive or delete old indices to prevent storage overflow
  • Monitor index size and performance regularly

5. Monitoring Cluster Health

Example: Check Cluster Health Status

GET /_cluster/health

{
  "cluster_name": "elasticsearch",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 3,
  "number_of_data_nodes": 3,
  "active_primary_shards": 10,
  "active_shards": 20,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0
}
Field Description Values
cluster_name Name of the Elasticsearch cluster elasticsearch
status Overall health status of the cluster green, yellow, red
timed_out Indicates if the request timed out true, false
number_of_nodes Total number of nodes in the cluster Integer
number_of_data_nodes Number of nodes that hold data Integer
active_primary_shards Number of active primary shards Integer
active_shards Total number of active shards Integer
relocating_shards Number of shards currently relocating Integer
initializing_shards Number of shards initializing Integer
unassigned_shards Number of shards not assigned to any node Integer

Cluster Health Statuses

  • Green: All primary and replica shards are allocated
  • Yellow: All primary shards are allocated, but some replica shards are unassigned
  • Red: Some primary shards are unassigned (data is potentially lost)
Note: The status field is the most important indicator of cluster health.

Example: Monitor Cluster Health with Watcher

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "http": {
      "request": {
        "host": "localhost",
        "port": 9200,
        "path": "/_cluster/health"
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.status": {
        "equal": "red"
      }
    }
  },
  "actions": {
    "send_alert": {
      "email": {
        "to": ["admin@example.com"],
        "subject": "Elasticsearch Cluster is Red",
        "text": "Cluster health status is red. Immediate action required."
      }
    }
  }
}

6. Implementing Disaster Recovery

Example: Basic Elasticsearch Cluster Recovery Configuration

cluster:
  routing:
    allocation:
      enable: all
      awareness:
        attributes: rack_id, zone
      disk:
        threshold_enabled: true
        watermark:
          low: 85%
          high: 90%
          flood_stage: 95%
Setting Description Default Value Notes
cluster.routing.allocation.enable Controls allocation of shards all Can be none, primaries, or replicas
cluster.routing.allocation.awareness.attributes Attributes used for allocation awareness null Prevents allocation to nodes with same attribute values
cluster.routing.allocation.disk.threshold_enabled Enables disk-based allocation thresholds false Must be enabled to use disk watermarks
cluster.routing.allocation.disk.watermark.low Low disk watermark percentage 85% Shards will not be allocated to nodes with less than this free space
cluster.routing.allocation.disk.watermark.high High disk watermark percentage 90% Shards will not be allocated to nodes with less than this free space
cluster.routing.allocation.disk.watermark.flood_stage Flood stage disk watermark percentage 95% Shards will not be allocated to nodes with less than this free space
Note: Disk watermarks are critical for preventing data loss in case of disk failure.

Example: Setting Up Snapshot and Restore

PUT /_snapshot/my_backup
{
  "type": "fs",
  "settings": {
    "location": "/mnt/backups/elasticsearch",
    "compress": true,
    "max_snapshot_bytes_per_second": "20mb"
  }
}
Setting Description Default Value Notes
type Snapshot repository type null Can be fs, s3, gcs, or azure
location Path or bucket name for snapshots null Required for fs type
compress Compresses snapshot data false Improves performance at cost of CPU usage
max_snapshot_bytes_per_second Rate limit for snapshot creation 20mb Prevents overwhelming system resources

Disaster Recovery Workflow

  1. Create a snapshot repository (e.g., filesystem or cloud storage)
  2. Take regular snapshots of the cluster
  3. Monitor snapshot status and performance
  4. In case of failure, restore from the latest snapshot
  5. Rebuild cluster using restored data

Snapshot Repository Types

  • Filesystem: Simple, local or networked storage
  • S3: Amazon S3-compatible object storage
  • GCS: Google Cloud Storage
  • Azure: Microsoft Azure Blob Storage
Elasticsearch Cluster Recovery Process
======================================

[Healthy Cluster] --[Snapshot]--> [Backup Repository]
         |                              |
         +---[Disaster]---> [Restore]--+

7. Scaling Horizontally

Example: Horizontal Scaling Configuration

# elasticsearch.yml
cluster.name: production-cluster
node.name: node-1
discovery.seed_hosts: ["host1:9300", "host2:9300", "host3:9300"]
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]
gateway.expected_nodes: 3
Setting Description Value
discovery.seed_hosts Initial list of master-eligible nodes to join the cluster ["host1:9300", "host2:9300", "host3:9300"]
cluster.initial_master_nodes Nodes that can form the initial master node set ["node-1", "node-2", "node-3"]
gateway.expected_nodes Number of nodes expected before starting the cluster 3
Note: Ensure all nodes in the cluster have identical cluster.name and discovery.seed_hosts configurations.

Horizontal Scaling Steps

  1. Add new data nodes to the cluster
  2. Verify node joins using GET /_cat/nodes?v
  3. Adjust shard allocation settings if needed
  4. Monitor cluster health with GET /_cluster/health

Horizontal vs Vertical Scaling

  • Horizontal: Add more nodes to distribute load
  • Vertical: Increase resources on existing nodes

8. Scaling Vertically

Component Vertical Scaling Options Notes
Elasticsearch increase RAM, add CPU cores Requires restart; increases heap size up to 32GB limit
Logstash increase heap size, add more workers Improves throughput; may require more memory
Kibana increase node memory, upgrade instance type Improves UI performance and query handling

Example: Increasing Elasticsearch Heap Size

# Edit elasticsearch.yml
cluster.name: my-cluster
node.name: node-1
bootstrap.memory_lock: true

# Set heap size in jvm.options
-Xms4g
-Xmx4g
Note: Vertical scaling has limits due to hardware constraints and JVM heap size restrictions.

9. Managing Configuration Drift

Configuration Source Type Description Drift Detection
Elasticsearch YAML Cluster settings and index templates NEW in 8.12: elasticsearch.yml drift detection
Kibana JSON UI settings and saved objects Manual comparison via kibana.yml
Logstash Configuration Pipeline and filter rules File hash comparison using logstash.yml

Example: Detecting Configuration Drift in Elasticsearch

curl -X GET "localhost:9200/_cluster/health?pretty"
Note: Use elasticsearch.yml to define cluster settings and ensure consistency across nodes.
Warning: Configuration drift can lead to inconsistent behavior and performance issues in production environments.
Tool Drift Detection Capability Support Status Notes
Elastic Stack Config Manager Automated drift detection BETA Available in 8.12+
Kibana UI Manual diff view DEPRECATED Removed in 8.13+
File-based comparison Hash-based drift detection NEW Requires external tooling

10. Implementing Blue-Green Deployments

Example: Blue-Green Deployment Configuration

# elasticsearch-blue-green.yml
cluster.name: "elk-blue-green"
node.name: "elk-node-blue"

# Blue environment
network.host: 192.168.1.10
http.port: 9200

# For green environment, change node.name and http.port
Deployment Configuration Description
Environment blue or green Active deployment environment
Port 9200 HTTP port for Elasticsearch
Host 192.168.1.10 Network interface binding

Deployment Workflow

  1. Deploy new version to green environment
  2. Run integration tests on green
  3. Switch traffic from blue to green
  4. Shut down blue environment
Note: Ensure both environments have identical data and configuration for seamless switching.
Warning: Blue-green deployments require double the infrastructure resources.

11. Using Canary Deployments

Example: Canary Deployment Configuration

# Example canary deployment config for ELK Stack
deployment:
  name: "canary-deployment"
  version: "8.10.0"
  strategy:
    type: "canary"
    steps:
      - percentage: 10
        duration: "5m"
      - percentage: 30
        duration: "10m"
      - percentage: 100
        duration: "15m"
Field Description Values Notes
deployment.strategy.type Deployment strategy type canary Must be set to canary for canary deployments
deployment.strategy.steps Deployment steps configuration Array of step objects Each step defines percentage and duration
steps.percentage Percentage of traffic to route to new version 0-100 Starts at 10%, increases to 100%
steps.duration Duration to keep this traffic split time duration Format: 5m, 10s, etc.
Note: Canary deployments help reduce risk by gradually shifting traffic to a new version while monitoring performance and errors.
Warning: Ensure that all services in the ELK stack are compatible with the new version before starting a canary deployment.

Canary Deployment Benefits

  • Gradual rollout reduces impact of bugs or performance issues
  • Real-time monitoring allows quick rollback if problems occur
  • Enables A/B testing of new features or configurations

12. Documenting Runbooks

Example: Runbook Structure

# Runbook: Application Deployment

## Overview
This runbook outlines the steps to deploy the application to production.

## Prerequisites
- Access to production environment
- Approved deployment branch

## Steps
1. Validate deployment branch
2. Run integration tests
3. Deploy to production

## Rollback Procedure
In case of failure, rollback to previous version.
Section Description Best Practice
Overview High-level description of the runbook's purpose Include scope and impact
Prerequisites Requirements before execution List all necessary permissions and tools
Steps Sequential actions to complete the task Use clear, actionable language
Rollback Procedure Recovery steps if deployment fails Document before execution

Runbook Components

  • Overview: Context and scope of the procedure
  • Prerequisites: Required permissions, tools, and conditions
  • Steps: Detailed actions to execute the task
  • Rollback: Recovery plan in case of failure
Note: Runbooks should be updated whenever procedures change to ensure accuracy.

Was this article helpful?