Implementing Cross-Cluster Replication

1. Understanding CCR

Feature	Description	Notes
Cross-Cluster Replication (CCR)	Replicates indices from one Elasticsearch cluster to another	Available in Elasticsearch 7.12+
Replication Relationship	Defines which indices are replicated from a source cluster to a target cluster	Managed via CCR APIs or Kibana UI
Follower Index	Index in the target cluster that replicates data from a leader index	Created automatically when a follower cluster is configured
Leader Index	Index in the source cluster that is replicated to a follower cluster	Must be writable and accessible from the follower cluster

Example: Enabling CCR on a follower cluster

PUT /_plugins/replication/follower_index/_settings
{
  "index.plugins.replication.follower": {
    "leader_alias": "leader_cluster"
  }
}

Note: CCR requires the replication plugin to be installed on both clusters.

Warning: The leader index must not be deleted or modified while replication is active.

Key Concepts in CCR

Leader Cluster: Source of the replicated indices
Follower Cluster: Target cluster that receives replicated indices
Replication Policy: Configuration that defines how replication is performed
Index State: Follower indices are read-only and cannot be modified directly

2. Configuring Leader Clusters

Property	Description	Default Value	Notes
cluster.name	Name of the cluster	elasticsearch	Must be unique across clusters in the replication setup
node.name	Name of the node	auto-generated	Must be unique within the cluster
discovery.seed_hosts	Addresses of other nodes in the cluster	empty	Required for multi-node clusters
cluster.initial_master_nodes	Nodes eligible to become master	empty	Required in v7+ for new clusters
xpack.security.enabled	Enables security features	false	Recommended for production environments

Example: Basic Leader Cluster Configuration

cluster.name: elk-replication-leader
node.name: leader-node-1
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["10.0.1.10:9300", "10.0.1.11:9300"]
cluster.initial_master_nodes: ["leader-node-1", "leader-node-2"]
xpack.security.enabled: true

Note: The leader cluster must be configured to allow cross-cluster communication and replication settings.

Warning: Ensure that all nodes in the leader cluster are part of the same network and can communicate with each other.

3. Creating Follower Indices

Property	Description	Value
index	Name of the follower index to be created	`my-follower-index`
remote.cluster	Alias of the remote cluster to replicate from	`upstream-cluster`
remote.index	Name of the leader index in the remote cluster	`my-leader-index`
remote.alias	Alias of the leader index in the remote cluster (optional)	`my-leader-alias`

Example: Creating a Follower Index

{
  "settings": {
    "index": {
      "remote": {
        "cluster": "upstream-cluster",
        "index": "my-leader-index"
      }
    }
  }
}

Note: The follower index will be created with the same settings and mappings as the leader index.

Warning: Ensure that the remote cluster is accessible and that the user has appropriate permissions to read from the leader index.

4. Managing Auto-Follow Patterns

Property	Description	Default Value	Notes
auto_follow_patterns	List of index patterns to auto-follow	Empty list	Each pattern is a regular expression matching index names
max_connections	Maximum number of connections to remote clusters	3	Controls parallel replication
max_read_request_size	Maximum size of read requests in bytes	1048576	Prevents large reads from blocking

Example: Configure auto-follow patterns

{
  "auto_follow_patterns": [
    {
      "pattern": "logs-.*",
      "max_connections": 3,
      "max_read_request_size": 1048576
    }
  ]
}

Note: Auto-follow patterns are managed via the elasticsearch.yml configuration or through the Cluster Update Settings API.

Warning: Misconfigured auto-follow patterns can lead to excessive resource consumption or index creation failures.

Pattern	Description	Use Case
`logs-.*`	Matches all indices starting with `logs-`	Replicating log data across clusters
`metrics-.*`	Matches all indices starting with `metrics-`	Replicating metrics data

5. Pausing and Resuming CCR

Operation	Description	Command	Notes
Pause CCR	Temporarily stops replication of a follower index	`PUT /_ccr/pause_follow<follower_index>`	Can be resumed later using `resume_follow`
Resume CCR	Resumes replication of a previously paused follower index	`PUT /_ccr/resume_follow<follower_index>`	Must be called on the follower cluster

Example: Pausing CCR for a follower index

PUT /_ccr/pause_follow/my_follower_index

Example: Resuming CCR for a follower index

PUT /_ccr/resume_follow/my_follower_index

Note: Pausing CCR does not affect the leader index. The follower index will remain in a paused state until resumed or deleted.

6. Monitoring Replication Stats

Stat	Description	Unit	Notes
cluster.replication.bytes	Bytes replicated across clusters	bytes	Monitored via Cluster Health API
cluster.replication.lag	Replication lag in milliseconds	ms	Indicates delay between primary and replica
cluster.replication.status	Replication status (healthy, degraded)	string	Can be used for alerting and dashboarding

Example: Fetching Replication Stats

GET /_cluster/health?level=replication

Note: Replication stats are updated every 5 seconds by default.

7. Handling Replication Errors

Error Type	Description	Resolution Steps
ClusterBlockException	Occurs when the cluster is blocked from changes due to maintenance or configuration issues.	Check cluster status using `GET /_cluster/health` Resolve blocking operations or wait for maintenance window Retry replication after cluster is unblocked
ReplicationFailedException	Indicates that a document failed to replicate across nodes in the cluster.	Review logs for specific error details Ensure target node has sufficient resources (memory, disk) Adjust replication settings or retry operation
TimeoutException	Replication process timed out due to network latency or node unavailability.	Check network connectivity between nodes Increase `replication.timeout` setting if appropriate Restart the affected nodes if necessary

Example: Error Log Analysis

{
  "error": {
    "type": "ReplicationFailedException",
    "reason": "Failed to replicate document to node [node-2]"
  },
  "status": 500
}

Note: Use GET /_cluster/allocation/explain to analyze allocation issues that may cause replication errors.

Warning: Repeated replication failures can lead to data inconsistency. Monitor logs regularly and address root causes promptly.

8. Configuring Read from Follower

Property	Description	Default	Notes
cluster.routing.allocation.read_from_follower	Enables or disables read operations from follower shards in cross-cluster replication.	`false`	Requires `cluster.routing.allocation.enable` to be set to `true`.

Example: Enabling Read from Follower

{
  "cluster": {
    "routing": {
      "allocation": {
        "read_from_follower": true
      }
    }
  }
}

Note: Enabling this setting can improve read performance in cross-cluster replication setups, but it may increase latency for writes due to follower shard synchronization.

9. Using Bi-Directional Replication

Bi-directional replication in the ELK Stack allows data to be synchronized between two or more clusters in both directions. This setup is useful for high availability, disaster recovery, and load distribution.

Configuration	Description	Notes
cluster.routing.allocation.cluster_settings	`"cluster.routing.allocation.enable"`	Enables allocation of shards across clusters
cluster.remote.connections	`"remote_cluster_name:connection_string"`	Defines remote cluster connections
cluster.remote.transport.hosts	`"host:port"`	Specifies transport hosts for remote clusters

Example: Configuring Bi-Directional Replication

cluster:
  remote:
    cluster_one:
      transport_host: "192.168.1.10:9300"
      proxy_host: "192.168.1.10:9300"
    cluster_two:
      transport_host: "192.168.1.20:9300"
      proxy_host: "192.168.1.20:9300"

Note: Bi-directional replication requires careful configuration to prevent data conflicts and ensure consistency across clusters.

Warning: Misconfigured bi-directional replication can lead to data loss or cluster instability. Always test in a staging environment.

Replication Mode	Description	Use Case
sync	Replicates data synchronously across clusters	High availability, real-time consistency
async	Replicates data asynchronously	Load distribution, backup

10. Unfollowing Indices

Property	Description	Values
indices	Name of the index to unfollow	`my-index`
cluster	Target cluster to unfollow from	`remote-cluster`

Example: Unfollow Index

PUT /my-index/_settings
{
  "index": {
    "remote": {
      "cluster": ""
    }
  }
}

Note: Unfollowing an index disables cross-cluster replication for that index.

11. Configuring Replication Settings

Setting	Description	Default Value	Notes
cluster.routing.allocation.cluster_concurrent_rebalance	Controls the number of concurrent rebalance operations	`2`	Higher values can speed up rebalancing but may impact performance
cluster.routing.allocation.node_concurrent_recoveries	Number of concurrent recovery operations per node	`2`	Adjust based on hardware capabilities and cluster size
cluster.routing.allocation.disk.watermark.low	Low watermark for disk usage (percentage or bytes)	`85%`	Prevents allocation to nodes with disk usage above this threshold
cluster.routing.allocation.disk.watermark.high	High watermark for disk usage (percentage or bytes)	`90%`	Triggers relocation of shards to other nodes
cluster.routing.allocation.disk.watermark.flood_stage	Flood stage watermark (percentage or bytes)	`95%`	Prevents allocation and triggers shard allocation cancellation

Example: Setting Disk Watermarks

{
  "cluster": {
    "routing": {
      "allocation": {
        "disk": {
          "watermark": {
            "low": "85%",
            "high": "90%",
            "flood_stage": "95%"
          }
        }
      }
    }
  }
}

Note: These settings can be updated dynamically without restarting nodes.