Working with Classic Queue Mirroring

1. Understanding Classic Mirroring

Classic mirroring is a RabbitMQ feature that provides high availability and fault tolerance for queues by maintaining multiple copies (mirrors) of a queue across different nodes in a cluster.

Property	Description	Default Value
queue mirroring	Enables or disables queue mirroring for a specific queue.	false
mirroring policy	Defines how mirrors are created and managed (e.g., all nodes, specific nodes).	all
mirroring sync mode	Determines how messages are synchronized between mirrors.	automatic

Example: Enabling Mirroring on a Queue

const amqp = require('amqplib');

async function setupMirroredQueue() {
  const connection = await amqp.connect('amqp://localhost');
  const channel = await connection.createChannel();

  const queueName = 'mirrored-queue';
  const options = {
    durable: true,
    arguments: {
      'x-queue-type': 'classic',
      'x-queue-mirror': true
    }
  };

  await channel.assertQueue(queueName, options);
  console.log(`Mirrored queue ${queueName} created.`);
}

Note: Mirroring is only available for classic queues. The x-queue-type must be set to classic for mirroring to function.

Key Concepts of Classic Mirroring

Master Queue: The primary queue that accepts all messages.
Mirror Queues: Copies of the master queue distributed across cluster nodes.
Synchronization: Messages are synchronized between the master and mirrors to maintain consistency.
Failover: If the master node fails, one of the mirrors is promoted to become the new master.

2. Setting ha-mode

Mode	Description	Configuration
ha-mode	Defines how queue replicas are distributed across nodes	`all`, `exactly`, `nodes`
ha-params	Parameters for `ha-mode`	Depends on mode (e.g., number of replicas or node list)
ha-sync-mode	Defines how replicas are synchronized	`manual`, `automatic`

Example: Setting ha-mode to 'all'

rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all"}'

Example: Setting ha-mode to 'exactly'

rabbitmqctl set_policy ha-exactly "^" '{"ha-mode":"exactly","ha-params":2}'

Example: Setting ha-mode to 'nodes'

rabbitmqctl set_policy ha-nodes "^" '{"ha-mode":"nodes","ha-params":["node1","node2"]}'

Note: Policies apply to queues matching the pattern. Use ^ to match all queues.

Warning: Changing ha-mode on existing queues does not affect already created replicas.

3. Configuring ha-params

Parameter	Description	Values	Default
ha-mode	Defines how queues are mirrored	`all`, `exactly`, `nodes`	`all`
ha-params	Additional parameters for mirroring	Object with `nodes`, `qos`, etc.	None
ha-promote-on-failure	Whether to promote a queue when a node fails	`true`, `false`	`false`

Example: Setting ha-params for queue mirroring

{
  "ha-mode": "exactly",
  "ha-params": {
    "nodes": ["rabbit@node1", "rabbit@node2"],
    "qos": 1000
  },
  "ha-promote-on-failure": true
}

Note: The ha-params object can include additional settings like qos to control queue behavior.

4. Setting ha-sync-mode

The ha-sync-mode parameter controls how queue mirrors are synchronized in a RabbitMQ cluster. It determines whether mirrors are synchronized asynchronously or synchronously.

Mode	Description	Default	Notes
async	Mirrors are synchronized asynchronously. The master queue processes messages and sends them to mirrors without waiting for acknowledgment.	`async`	Lower latency but higher risk of message loss if a mirror fails.
sync	Mirrors are synchronized synchronously. The master queue waits for acknowledgment from mirrors before processing the next message.	`sync`	Higher consistency but higher latency due to synchronization overhead.

Example: Setting ha-sync-mode via RabbitMQ Management UI

curl -u guest:guest -X PUT http://localhost:15672/api/queues/%2f/my-queue -d '{
  "ha-mode": "all",
  "ha-sync-mode": "sync"
}'

Example: Setting ha-sync-mode via RabbitMQ Configuration File

queue&#45;mirrors&#45;min&#45;sync&#45;nodes: 2
queue&#45;mirrors&#45;sync&#45;mode: sync

5. Promoting Queue Master

When a queue's master node becomes unavailable, RabbitMQ can promote a replica to become the new master. This process ensures high availability and continues message delivery.

Command	Description	Notes
rabbitmqctl promote_master	Promotes a queue's replica to master	Must be run on the node where the queue is located

Example: Promoting a Queue Master

rabbitmqctl promote_master rabbit@node1 my_queue

Note: The queue must be in a mirrored configuration for this command to work. Ensure that the queue has replicas before attempting promotion.

6. Using ha-promote-on-shutdown

The ha-promote-on-shutdown policy is used to control the behavior of mirrored queues when a master node shuts down. When enabled, it allows the queue to promote one of its replicas to become the new master node.

Policy	Values	Description
ha-promote-on-shutdown	`true` \| `false`	When set to `true`, promotes a replica to master on shutdown. Defaults to `false`.

Example: Setting ha-promote-on-shutdown

{"ha-promote-on-shutdown": true}

Note: This policy is useful in environments where queue availability is critical and manual intervention is not desired during shutdowns.

7. Setting ha-promote-on-failure

Property	Values	Description
ha-promote-on-failure	`auto`, `never`	Determines whether a replica queue is promoted to master when the master queue fails. `auto` promotes the queue if it is the only replica and has been synchronized. `never` prevents promotion.

Example: Configuring ha-promote-on-failure

{
  "queue": "my-queue",
  "ha-promote-on-failure": "auto"
}

Note: The ha-promote-on-failure setting is part of the ha-policy and affects queue behavior during failover scenarios.

8. Monitoring Mirror Status

Example: Check mirror status of a queue

rabbitmqctl list_queues name mirrors local_mirror_status

Field	Description	Values
name	Queue name	`my-queue`
mirrors	Total number of mirrors	`3`
local_mirror_status	Local mirror status	`sync`, `async`, `down`

Note: The local_mirror_status field indicates whether the local mirror is in sync, async, or down.

Example: Using RabbitMQ Management Plugin

curl -u guest:guest http://localhost:15672/api/queues/%2f/my-queue

Property	Description	Values
status	Queue status	`running`, `stopped`
mirrors	Number of mirrors	`3`
active_consumers	Active consumers count	`2`

Monitoring Mirror Status Summary

Use rabbitmqctl list_queues to check mirror status.
Use the Management Plugin API for detailed queue information.
Monitor local_mirror_status to ensure sync state.

9. Understanding Split-Brain Scenarios

Note: Split-brain scenarios occur when a cluster loses network connectivity and splits into isolated partitions, each believing it is the primary cluster.

Scenario	Description	Impact	Solution
Network Partition	Cluster nodes lose connectivity to each other, forming isolated groups.	Queues and messages may be lost or duplicated across partitions.	Use `cluster-formation.strategy` with `ratelimit` or `disc` to prevent splits.
Queue Mirroring Disruption	When a mirrored queue's master node becomes unreachable, replicas may not sync properly.	Potential data inconsistency or message loss if no quorum is maintained.	Configure `queue-master-locator` to ensure consistent master selection.
Cluster Reconnection	After network recovery, nodes attempt to rejoin the cluster.	May cause message duplication or loss if not handled carefully.	Enable `queue-rebalance-on-node-join` to maintain consistency.

Example: Configuring Split-Brain Protection

{
  "cluster-formation": {
    "strategy": "ratelimit",
    "ratelimit": {
      "interval": "30s",
      "max-partitions": 2
    }
  },
  "queue-master-locator": "min-masters"
}

Key Considerations for Split-Brain Prevention

Use cluster-formation.strategy=ratelimit to limit cluster splits.
Ensure queue-master-locator is set to maintain queue consistency.
Enable queue-rebalance-on-node-join to handle reconnection safely.

10. Migrating to Quorum Queues

Note: This section outlines the process and considerations for migrating from classic mirrored queues to quorum queues in RabbitMQ.

Quorum queues provide better consistency and durability compared to classic mirrored queues. Migrating involves careful planning to ensure minimal disruption.

Step	Description	Action Required
1	Prepare for migration by backing up queue configurations and messages.	Export queue definitions using `rabbitmqctl` or API.
2	Verify queue type and settings in the current classic mirrored queue.	Check `queue_type`, `mirrored`, and `ha-mode`.
3	Create new quorum queue with equivalent configuration.	Use `rabbitmqadmin` or API to declare the new queue.
4	Replay messages from classic queue to quorum queue.	Use `rabbitmqctl` or a custom script to replay messages.
5	Update client applications to point to the new quorum queue.	Modify connection strings or queue names in application code.
6	Verify message delivery and consumer behavior.	Confirm that all messages are processed correctly.

Example: Creating a Quorum Queue

rabbitmqadmin declare queue name=my_quorum_queue type=quorum durable=true

Classic Mirrored vs Quorum Queue

Classic Mirrored: Uses ha-mode and ha-params for mirroring.
Quorum Queue: Uses queue_type set to quorum.
Consistency: Quorum queues provide stronger consistency guarantees.

Warning: Migrating queues can lead to message loss if not handled carefully. Always test in a staging environment first.