Managing Production Operations
1. Planning Cluster Capacity
Note: Proper cluster capacity planning ensures optimal performance and scalability for your ELK Stack deployment.
| Resource | Description | Recommendation |
|---|---|---|
| Elasticsearch Nodes | Primary units processing data and serving queries | Start with 3 nodes for high availability; scale based on data volume and query load |
| Memory (RAM) | Allocated to Elasticsearch JVM heap | Set to 50% of total system RAM, up to 32GB; avoid exceeding 50% to prevent GC overhead |
| Storage | Used for indices and snapshots | Use SSDs with sufficient IOPS; plan for 2x data size to accommodate index growth |
| CPU Cores | Processing power for indexing and search operations | Minimum 4 cores per node; more cores improve parallel processing capability |
Example: Sample Capacity Planning Configuration
# elasticsearch.yml
cluster.name: production-cluster
node.name: es-node-1
network.host: 0.0.0.0
http.port: 9200
# Memory settings (50% of system RAM)
bootstrap.memory_lock: true
indices.memory.index_buffer_size: 25%
Key Considerations for Cluster Scaling
- Start with
3 nodesfor redundancy and fault tolerance - Monitor
GC overheadto ensure JVM heap is properly sized - Plan for
index lifecycle management (ILM)to handle data retention - Use
shard allocationstrategies to balance load across nodes
Capacity Planning Workflow
- Determine current and projected data volume (GB/day)
- Estimate query patterns and concurrent user load
- Select appropriate node types (e.g.,
io,memory,master) - Configure memory and storage for optimal performance
- Run load tests to validate capacity assumptions
2. Implementing Backup Strategy
Example: Elasticsearch Snapshot Configuration
{
"type": "fs",
"location": "/backups/elasticsearch"
}
| Configuration | Description | Values | Notes |
|---|---|---|---|
| repository.type | Repository type for snapshots | fs, s3, gcs, azure |
File system, S3, Google Cloud Storage, Azure Blob Storage |
| repository.location | Path or bucket for snapshots | /backups/elasticsearch |
Must be accessible to all nodes |
| snapshot.ignore_unavailable | Ignore unavailable indices | true, false |
Default: false |
Note: Ensure the snapshot repository is configured before creating snapshots.
Example: Create Snapshot
PUT /_snapshot/my_backup/snapshot_1
{
"indices": "index-2024-*",
"ignore_unavailable": true,
"include_global_state": false
}
Example: Restore Snapshot
POST /_snapshot/my_backup/snapshot_1/_restore
{
"indices": "index-2024-*",
"rename_pattern": "index-(.*)",
"rename_replacement": "restored-index-$1"
}
Backup Strategy Best Practices
- Use dedicated backup repositories (e.g., S3 or GCS)
- Schedule regular snapshots with
elasticsearch-snapshotor cron jobs - Test restore procedures regularly
- Ensure sufficient disk space in backup repositories
3. Performing Rolling Upgrades
Rolling Upgrade Process
- Prepare the new version of Elasticsearch, Logstash, and Kibana
- Start the upgrade process on one node at a time
- Ensure cluster health remains green during the upgrade
- Verify that all indices are properly migrated
- Continue until all nodes are upgraded
| Component | Upgrade Method | Notes |
|---|---|---|
| Elasticsearch | rolling_upgrade |
Must be performed one node at a time |
| Logstash | rolling_restart |
Reconfigure pipelines after restart |
| Kibana | rolling_restart |
Ensure compatibility with new Elasticsearch version |
Note: Always backup indices before starting the upgrade process.
Example: Upgrade Elasticsearch Node
sudo systemctl stop elasticsearch
sudo yum update elasticsearch
sudo systemctl start elasticsearch
4. Managing Index Growth
| Strategy | Description | Implementation | Notes |
|---|---|---|---|
| Index Lifecycle Management (ILM) | Automatically manage index lifecycle based on age, size, or document count | |
ILM policies can be applied to index templates |
| Index Rollover | Split large indices into smaller ones to manage performance and storage | |
Use max_size or max_age to trigger rollover |
| Index Templates | Define index settings and mappings to ensure consistency across indices | |
Templates apply to new indices matching the pattern |
Example: Setting up ILM Policy
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_size": "50gb",
"max_age": "30d"
},
"set_priority": {
"priority": 100
}
}
},
"warm": {
"min_age": "30d",
"actions": {
"set_priority": {
"priority": 50
},
"allocate": {
"number_of_replicas": 1,
"include": {
"box_type": "warm"
}
}
}
},
"delete": {
"min_age": "90d",
"actions": {
"delete": {}
}
}
}
}
}
Note: ILM policies must be attached to index templates or existing indices to take effect.
Warning: Deleting old indices can lead to data loss if not properly backed up or archived.
Best Practices for Index Growth Management
- Use ILM policies to automate index lifecycle transitions
- Set appropriate size and age thresholds for rollover
- Archive or delete old indices to prevent storage overflow
- Monitor index size and performance regularly
5. Monitoring Cluster Health
Example: Check Cluster Health Status
GET /_cluster/health
{
"cluster_name": "elasticsearch",
"status": "green",
"timed_out": false,
"number_of_nodes": 3,
"number_of_data_nodes": 3,
"active_primary_shards": 10,
"active_shards": 20,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0
}
| Field | Description | Values |
|---|---|---|
| cluster_name | Name of the Elasticsearch cluster | elasticsearch |
| status | Overall health status of the cluster | green, yellow, red |
| timed_out | Indicates if the request timed out | true, false |
| number_of_nodes | Total number of nodes in the cluster | Integer |
| number_of_data_nodes | Number of nodes that hold data | Integer |
| active_primary_shards | Number of active primary shards | Integer |
| active_shards | Total number of active shards | Integer |
| relocating_shards | Number of shards currently relocating | Integer |
| initializing_shards | Number of shards initializing | Integer |
| unassigned_shards | Number of shards not assigned to any node | Integer |
Cluster Health Statuses
- Green: All primary and replica shards are allocated
- Yellow: All primary shards are allocated, but some replica shards are unassigned
- Red: Some primary shards are unassigned (data is potentially lost)
Note: The
status field is the most important indicator of cluster health.
Example: Monitor Cluster Health with Watcher
{
"trigger": {
"schedule": {
"interval": "1m"
}
},
"input": {
"http": {
"request": {
"host": "localhost",
"port": 9200,
"path": "/_cluster/health"
}
}
},
"condition": {
"compare": {
"ctx.payload.status": {
"equal": "red"
}
}
},
"actions": {
"send_alert": {
"email": {
"to": ["admin@example.com"],
"subject": "Elasticsearch Cluster is Red",
"text": "Cluster health status is red. Immediate action required."
}
}
}
}
6. Implementing Disaster Recovery
Example: Basic Elasticsearch Cluster Recovery Configuration
cluster:
routing:
allocation:
enable: all
awareness:
attributes: rack_id, zone
disk:
threshold_enabled: true
watermark:
low: 85%
high: 90%
flood_stage: 95%
| Setting | Description | Default Value | Notes |
|---|---|---|---|
| cluster.routing.allocation.enable | Controls allocation of shards | all |
Can be none, primaries, or replicas |
| cluster.routing.allocation.awareness.attributes | Attributes used for allocation awareness | null |
Prevents allocation to nodes with same attribute values |
| cluster.routing.allocation.disk.threshold_enabled | Enables disk-based allocation thresholds | false |
Must be enabled to use disk watermarks |
| cluster.routing.allocation.disk.watermark.low | Low disk watermark percentage | 85% |
Shards will not be allocated to nodes with less than this free space |
| cluster.routing.allocation.disk.watermark.high | High disk watermark percentage | 90% |
Shards will not be allocated to nodes with less than this free space |
| cluster.routing.allocation.disk.watermark.flood_stage | Flood stage disk watermark percentage | 95% |
Shards will not be allocated to nodes with less than this free space |
Note: Disk watermarks are critical for preventing data loss in case of disk failure.
Example: Setting Up Snapshot and Restore
PUT /_snapshot/my_backup
{
"type": "fs",
"settings": {
"location": "/mnt/backups/elasticsearch",
"compress": true,
"max_snapshot_bytes_per_second": "20mb"
}
}
| Setting | Description | Default Value | Notes |
|---|---|---|---|
| type | Snapshot repository type | null |
Can be fs, s3, gcs, or azure |
| location | Path or bucket name for snapshots | null |
Required for fs type |
| compress | Compresses snapshot data | false |
Improves performance at cost of CPU usage |
| max_snapshot_bytes_per_second | Rate limit for snapshot creation | 20mb |
Prevents overwhelming system resources |
Disaster Recovery Workflow
- Create a snapshot repository (e.g., filesystem or cloud storage)
- Take regular snapshots of the cluster
- Monitor snapshot status and performance
- In case of failure, restore from the latest snapshot
- Rebuild cluster using restored data
Snapshot Repository Types
- Filesystem: Simple, local or networked storage
- S3: Amazon S3-compatible object storage
- GCS: Google Cloud Storage
- Azure: Microsoft Azure Blob Storage
Elasticsearch Cluster Recovery Process
======================================
[Healthy Cluster] --[Snapshot]--> [Backup Repository]
| |
+---[Disaster]---> [Restore]--+
7. Scaling Horizontally
Example: Horizontal Scaling Configuration
# elasticsearch.yml
cluster.name: production-cluster
node.name: node-1
discovery.seed_hosts: ["host1:9300", "host2:9300", "host3:9300"]
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]
gateway.expected_nodes: 3
| Setting | Description | Value |
|---|---|---|
| discovery.seed_hosts | Initial list of master-eligible nodes to join the cluster | ["host1:9300", "host2:9300", "host3:9300"] |
| cluster.initial_master_nodes | Nodes that can form the initial master node set | ["node-1", "node-2", "node-3"] |
| gateway.expected_nodes | Number of nodes expected before starting the cluster | 3 |
Note: Ensure all nodes in the cluster have identical
cluster.name and discovery.seed_hosts configurations.
Horizontal Scaling Steps
- Add new data nodes to the cluster
- Verify node joins using
GET /_cat/nodes?v - Adjust shard allocation settings if needed
- Monitor cluster health with
GET /_cluster/health
Horizontal vs Vertical Scaling
- Horizontal: Add more nodes to distribute load
- Vertical: Increase resources on existing nodes
8. Scaling Vertically
| Component | Vertical Scaling Options | Notes |
|---|---|---|
| Elasticsearch | increase RAM, add CPU cores |
Requires restart; increases heap size up to 32GB limit |
| Logstash | increase heap size, add more workers |
Improves throughput; may require more memory |
| Kibana | increase node memory, upgrade instance type |
Improves UI performance and query handling |
Example: Increasing Elasticsearch Heap Size
# Edit elasticsearch.yml
cluster.name: my-cluster
node.name: node-1
bootstrap.memory_lock: true
# Set heap size in jvm.options
-Xms4g
-Xmx4g
Note: Vertical scaling has limits due to hardware constraints and JVM heap size restrictions.
9. Managing Configuration Drift
| Configuration Source | Type | Description | Drift Detection |
|---|---|---|---|
| Elasticsearch | YAML | Cluster settings and index templates | NEW in 8.12: elasticsearch.yml drift detection |
| Kibana | JSON | UI settings and saved objects | Manual comparison via kibana.yml |
| Logstash | Configuration | Pipeline and filter rules | File hash comparison using logstash.yml |
Example: Detecting Configuration Drift in Elasticsearch
curl -X GET "localhost:9200/_cluster/health?pretty"
Note: Use
elasticsearch.yml to define cluster settings and ensure consistency across nodes.
Warning: Configuration drift can lead to inconsistent behavior and performance issues in production environments.
| Tool | Drift Detection Capability | Support Status | Notes |
|---|---|---|---|
| Elastic Stack Config Manager | Automated drift detection | BETA | Available in 8.12+ |
| Kibana UI | Manual diff view | DEPRECATED | Removed in 8.13+ |
| File-based comparison | Hash-based drift detection | NEW | Requires external tooling |
10. Implementing Blue-Green Deployments
Example: Blue-Green Deployment Configuration
# elasticsearch-blue-green.yml
cluster.name: "elk-blue-green"
node.name: "elk-node-blue"
# Blue environment
network.host: 192.168.1.10
http.port: 9200
# For green environment, change node.name and http.port
| Deployment | Configuration | Description |
|---|---|---|
| Environment | blue or green |
Active deployment environment |
| Port | 9200 |
HTTP port for Elasticsearch |
| Host | 192.168.1.10 |
Network interface binding |
Deployment Workflow
- Deploy new version to
greenenvironment - Run integration tests on green
- Switch traffic from
bluetogreen - Shut down
blueenvironment
Note: Ensure both environments have identical data and configuration for seamless switching.
Warning: Blue-green deployments require double the infrastructure resources.
11. Using Canary Deployments
Example: Canary Deployment Configuration
# Example canary deployment config for ELK Stack
deployment:
name: "canary-deployment"
version: "8.10.0"
strategy:
type: "canary"
steps:
- percentage: 10
duration: "5m"
- percentage: 30
duration: "10m"
- percentage: 100
duration: "15m"
| Field | Description | Values | Notes |
|---|---|---|---|
| deployment.strategy.type | Deployment strategy type | canary |
Must be set to canary for canary deployments |
| deployment.strategy.steps | Deployment steps configuration | Array of step objects | Each step defines percentage and duration |
| steps.percentage | Percentage of traffic to route to new version | 0-100 |
Starts at 10%, increases to 100% |
| steps.duration | Duration to keep this traffic split | time duration |
Format: 5m, 10s, etc. |
Note: Canary deployments help reduce risk by gradually shifting traffic to a new version while monitoring performance and errors.
Warning: Ensure that all services in the ELK stack are compatible with the new version before starting a canary deployment.
Canary Deployment Benefits
- Gradual rollout reduces impact of bugs or performance issues
- Real-time monitoring allows quick rollback if problems occur
- Enables A/B testing of new features or configurations
12. Documenting Runbooks
Example: Runbook Structure
# Runbook: Application Deployment
## Overview
This runbook outlines the steps to deploy the application to production.
## Prerequisites
- Access to production environment
- Approved deployment branch
## Steps
1. Validate deployment branch
2. Run integration tests
3. Deploy to production
## Rollback Procedure
In case of failure, rollback to previous version.
| Section | Description | Best Practice |
|---|---|---|
| Overview | High-level description of the runbook's purpose | Include scope and impact |
| Prerequisites | Requirements before execution | List all necessary permissions and tools |
| Steps | Sequential actions to complete the task | Use clear, actionable language |
| Rollback Procedure | Recovery steps if deployment fails | Document before execution |
Runbook Components
- Overview: Context and scope of the procedure
- Prerequisites: Required permissions, tools, and conditions
- Steps: Detailed actions to execute the task
- Rollback: Recovery plan in case of failure
Note: Runbooks should be updated whenever procedures change to ensure accuracy.