Skip to main content
Mapagam

Working with Snapshots and Restore

1. Understanding Snapshot Repositories

Repository Type Description Configuration Example Notes
fs File system-based repository "type": "fs" Stores snapshots on the file system of each node
s3 Amazon S3 repository "type": "s3" Requires AWS credentials and bucket configuration
gcs Google Cloud Storage repository "type": "gcs" Requires GCP service account credentials
azure Azure Blob Storage repository "type": "azure" Requires Azure storage account credentials
hdfs HDFS repository "type": "hdfs" Stores snapshots in Hadoop Distributed File System

Example: Registering an S3 Snapshot Repository

{
  "type": "s3",
  "settings": {
    "bucket": "elk-snapshots-bucket",
    "region": "us-west-2",
    "access_key": "AKIAIOSFODNN7EXAMPLE",
    "secret_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
  }
}
Note: The fs repository is useful for testing and development environments where snapshots are stored locally on each node.
Warning: Ensure that the snapshot repository is properly configured with appropriate permissions to avoid restore failures.

2. Configuring Snapshot Repositories

Repository Type Configuration Parameters Description Notes
fs type: fs
location: /path/to/snapshot/dir		 File system-based repository Supports local and network file systems. Requires directory to be accessible by all nodes.
s3 type: s3
bucket: my-snapshot-bucket
region: us-east-1		 S3-compatible repository Supports AWS S3, MinIO, and other S3-compatible storage services.
gcs type: gcs
bucket: my-snapshot-bucket		 Google Cloud Storage repository Requires Google Cloud credentials and permissions.
azure type: azure
container: my-snapshot-container		 Azure Blob Storage repository Requires Azure Storage account and permissions.

Example: Creating an S3 Snapshot Repository

PUT /_snapshot/my_s3_repository
{
  "type": "s3",
  "settings": {
    "bucket": "my-elk-snapshots",
    "region": "us-west-2",
    "compress": true
  }
}
Note: Ensure that the Elasticsearch nodes have appropriate permissions to access the snapshot repository. For S3, this includes proper IAM roles or credentials.

3. Creating Snapshots

Snapshots in the ELK Stack are used to back up and restore indices and cluster state. They are created using the Snapshot API.

Parameter Description Values
repository Name of the snapshot repository my_backup
snapshot Name of the snapshot to create snapshot_2026_01_01
indices Indices to include in the snapshot index1,index2
ignore_unavailable Whether to ignore unavailable indices true or false
include_global_state Whether to include cluster state true or false

Example: Creating a Snapshot

PUT /_snapshot/my_backup/snapshot_2026_01_01
{
  "indices": "index1,index2",
  "ignore_unavailable": true,
  "include_global_state": false
}
Note: Ensure that the repository is properly configured before creating snapshots.
Warning: Snapshots can be large and may take considerable time to complete depending on the size of indices being backed up.

4. Listing Snapshots

Snapshot ID Repository Status Timestamp
snapshot-2025.04.01 my-repo Successful 2025-04-01T10:30:00Z
snapshot-2025.04.02 my-repo Failed 2025-04-02T14:15:00Z
snapshot-2025.04.03 backup-repo Successful 2025-04-03T09:45:00Z

Example: List all snapshots in a repository

GET /_snapshot/my-repo/_all

Example: List snapshots with specific pattern

GET /_snapshot/my-repo/snapshot-2025.04.*

5. Viewing Snapshot Status

Field Description Example Value
snapshot_id Unique identifier for the snapshot snapshot-1234567890
repository Name of the repository where the snapshot is stored my-repo
state Current state of the snapshot (e.g., success, failed, in_progress) success
start_time Timestamp when the snapshot process started 2026-04-05T10:00:00Z
end_time Timestamp when the snapshot process completed 2026-04-05T10:05:00Z
indices List of indices included in the snapshot [logs-2026.04.05, metrics-2026.04.05]
shards Information about shards in the snapshot { successful: 5, total: 5 }

Example: Fetching Snapshot Status

GET /_snapshot/my-repo/snapshot-1234567890

Example: Listing All Snapshots in a Repository

GET /_snapshot/my-repo/_all
Note: The state field can be one of success, failed, in_progress, or partially_failed.

6. Restoring Snapshots

Step Description Action
1 Identify the snapshot to restore Use GET /_snapshot/<repository_name>/<snapshot_name>
2 Check snapshot status Use GET /_snapshot/<repository_name>/<snapshot_name>
3 Prepare for restore Ensure indices are closed or compatible with restore
4 Initiate restore Use POST /_snapshot/<repository_name>/<snapshot_name>/_restore
5 Monitor restore progress Use GET /_cat/indices?v or GET /_recovery

Example: Restore Snapshot

PUT /_snapshot/my_backup/snapshot_1/_restore
{
  "indices": "index_1,index_2",
  "ignore_unavailable": true,
  "include_global_state": false
}
Note: The restore operation will fail if the indices are already present in the cluster.
Warning: Restoring to a different cluster or version may cause compatibility issues.
Parameter Description Default
indices List of indices to restore All indices
ignore_unavailable Ignore unavailable indices false
include_global_state Restore cluster state false
rename_pattern Pattern to rename indices None
rename_replacement Replacement pattern for index names None

Restore Snapshot Summary

  • Snapshot must exist in repository
  • Indices must be closed or compatible
  • Restore can target specific indices
  • Global state can be included or excluded
  • Restore is asynchronous by default

7. Deleting Snapshots

Option Values Notes
snapshot.id snapshot-2024-01-01 ID of the snapshot to delete
repository.name my-repo Name of the snapshot repository

Example: Delete a Snapshot

DELETE /_snapshot/my-repo/snapshot-2024-01-01
Note: The snapshot must be in a completed state to be deleted.
Warning: Deleting a snapshot is irreversible and will remove all associated data.

8. Using Searchable Snapshots

Property Description Default Notes
index.store.type Storage type for the index fs Can be set to searchable_snapshot for searchable snapshots
index.blocks.read_only_allow_delete Allows deletion of indices in read-only mode false Set to true for searchable snapshots
index.blocks.read_only Makes the index read-only false Set to true for searchable snapshots

Example: Enabling Searchable Snapshots

{
  "index": {
    "store": {
      "type": "searchable_snapshot"
    },
    "blocks": {
      "read_only_allow_delete": "true",
      "read_only": "true"
    }
  }
}
Note: Searchable snapshots are available in Elasticsearch 7.10+ and require a snapshot repository to be configured.
Warning: Enabling searchable snapshots will make the index read-only and prevent any writes to it.

Key Features of Searchable Snapshots

  • Reduce storage costs by storing index data in a compressed format
  • Enable faster search performance on large datasets
  • Supports read-only access to index data
  • Requires a snapshot repository for storage

9. Automating Snapshots

Example: Snapshot Automation with Cron Job

# Create a cron job to run daily at 2 AM
0 2 * * * /usr/bin/curl -X POST "http://localhost:9200/_snapshot/my_backup/snapshot_$(date +\%Y-\%m-\%d)" -H "Content-Type: application/json" -d &#x27;{"indices":"logs-*"}&#x27;
Property Description Default
schedule Cron expression for snapshot timing None
repository Name of snapshot repository None
indices Indices to include in snapshot All indices
Note: Ensure the Elasticsearch user has necessary permissions to create snapshots and access the repository.
Warning: Cron jobs should be monitored for failures and log retention policies should be enforced.

10. Performing Partial Restore

Option Values Description
indices index1,index2 Comma-separated list of indices to restore
ignore_unavailable true | false If true, ignores unavailable indices during restore
include_global_state true | false If true, restores global cluster state (mappings, settings)
rename_pattern .* Regex pattern to rename indices during restore
rename_replacement new_$1 Replacement string for renamed indices

Example: Restore specific indices

PUT /_snapshot/my_backup/snapshot_1/_restore
{
  "indices": "index1,index2",
  "ignore_unavailable": true,
  "include_global_state": false
}
Note: The restore operation will fail if the indices are already present in the cluster.
Warning: Restoring indices with different names requires careful handling of index aliases and mappings.

11. Monitoring Snapshot Progress

Property Description Values
snapshot_id Unique identifier for the snapshot snapshot-2025-04-05-14-30-00
status Current state of the snapshot process in_progress, failed, success
start_time Timestamp when the snapshot started 2025-04-05T14:30:00Z
end_time Timestamp when the snapshot completed (if finished) 2025-04-05T14:35:00Z
indices List of indices included in the snapshot ["logs-2025.04.05", "metrics-2025.04.05"]
total_shards Total number of shards in the snapshot 24
successful_shards Number of shards successfully snapshotted 20
failed_shards Number of shards that failed to snapshot 4

Example: Fetching Snapshot Status

GET /_snapshot/my_backup/snapshot-2025-04-05-14-30-00

Example: Listing All Snapshots

GET /_snapshot/my_backup/_all
Note: The status field can be used to determine if a snapshot is still in progress or has completed.
Warning: If failed_shards is greater than zero, the snapshot may not be fully recoverable.

12. Configuring Snapshot Threads

Property Default Value Description
thread_pool.snapshot.queue_size 20 Number of snapshot operations that can be queued before blocking.
thread_pool.snapshot.max_threads 4 Maximum number of threads used for snapshot operations.
thread_pool.snapshot.affinity false Whether to use thread affinity for snapshot operations.

Example: Snapshot Thread Configuration

thread_pool:
  snapshot:
    queue_size: 30
    max_threads: 6
    affinity: true

Was this article helpful?

Related Cheat Sheets