Implementing Backup and Recovery

1. Exporting Definitions

Exporting RabbitMQ definitions allows you to back up and restore the configuration of your broker, including exchanges, queues, bindings, and users.

Property	Description	Example Value
rabbitmqctl export	Exports the current broker's definitions to a JSON file	`rabbitmqctl export /path/to/definitions.json`
rabbitmqadmin export	Exports definitions using the rabbitmqadmin command-line tool	`rabbitmqadmin export /path/to/definitions.json`

Example: Exporting Definitions Using rabbitmqctl

rabbitmqctl export /home/user/rabbitmq_definitions.json

Example: Exporting Definitions Using rabbitmqadmin

rabbitmqadmin export /home/user/rabbitmq_definitions.json

Note: The exported definitions include all exchanges, queues, bindings, users, and permissions. Ensure the file is securely stored for recovery purposes.

2. Using Definitions File

Definitions files in RabbitMQ are used to export and import the structure of a broker, including exchanges, queues, bindings, and users. These files are typically in JSON format and can be used for backup and recovery purposes.

Field	Description	Example Value
version	Version of the definitions file format	`"1.0"`
users	List of users and their permissions	`[{ "name": "user1", "password_hash": "...", "tags": ["administrator"] }]`
vhosts	List of virtual hosts and their configurations	`[{ "name": "/", "permissions": [...] }]`
exchanges	Exchange definitions	`[{ "name": "my_exchange", "type": "direct", "durable": true }]`
queues	Queue definitions	`[{ "name": "my_queue", "durable": true, "auto_delete": false }]`
bindings	Bindings between exchanges and queues	`[{ "source": "exchange1", "destination": "queue1" }]`

Example: Exporting Definitions

curl -u guest:guest http://localhost:15672/api/definitions > definitions.json

Example: Importing Definitions

curl -u guest:guest -H "Content-Type: application/json" --data-binary @definitions.json http://localhost:15672/api/definitions

Note: When importing definitions, ensure that the target RabbitMQ instance is in a clean state or that existing resources are compatible with the imported definitions to avoid conflicts.

3. Importing Definitions

Importing RabbitMQ definitions allows you to restore or replicate the configuration of exchanges, queues, bindings, and other elements from a JSON file. This is useful for backup, migration, or setting up a consistent environment.

Example: Importing Definitions

curl -X POST "http://localhost:15672/api/definitions" \
  -u guest:guest \
  -H "Content-Type: application/json" \
  --data-binary @definitions.json

Parameter	Description	Default
--data-binary @definitions.json	Path to the JSON file containing definitions to import	None

Note: The definitions file must be in the correct format as exported by RabbitMQ's API or CLI tools. Ensure all referenced entities (e.g., exchanges, queues) are compatible with the target broker's version.

Warning: Importing definitions will overwrite existing configurations with matching names. Always backup current definitions before importing.

4. Backing Up Configuration

Backing up RabbitMQ configuration is essential for disaster recovery and maintaining system consistency. Configuration includes exchanges, queues, bindings, users, permissions, and policies.

Configuration Element	Backup Method	Notes
Exchanges	`rabbitmqctl export /path/to/exchanges.json`	Exports all exchanges and their properties to a JSON file.
Queues	`rabbitmqctl export /path/to/queues.json`	Exports all queues and their properties to a JSON file.
Bindings	`rabbitmqctl export /path/to/bindings.json`	Exports all bindings between exchanges and queues to a JSON file.
Users & Permissions	`rabbitmqctl export /path/to/users.json`	Exports all users and their permissions to a JSON file.
Policies	`rabbitmqctl export /path/to/policies.json`	Exports all policies to a JSON file.
Definitions	`rabbitmqctl export /path/to/definitions.json`	Exports all definitions including exchanges, queues, bindings, users, permissions, and policies.

Example: Exporting All Definitions

rabbitmqctl export /backup/rabbitmq_definitions.json

Note: Ensure that the backup directory has appropriate permissions to prevent unauthorized access.

Warning: The exported JSON file contains sensitive information like user credentials. Handle it securely.

5. Backing Up Message Data

Backing up message data in RabbitMQ is critical for ensuring system resilience and data integrity. The following methods outline how to perform backups effectively.

Backup Method	Description	Notes
RabbitMQ Management Plugin	Uses the HTTP API to export queue and exchange definitions.	Does not include actual message content, only metadata.
RabbitMQ Backup Tool (rabbitmq-backup)	Command-line tool to backup queues, exchanges, and messages.	Requires installation and configuration; supports message content.
File System Backup	Manually copying the `mnesia` directory.	Includes all messages and metadata but is fragile and not recommended for production.
RabbitMQ Federation Plugin	Allows mirroring of queues across nodes or clusters.	Used for high availability, not direct backup.

Example: Using RabbitMQ Management Plugin to Export Definitions

curl -u guest:guest http://localhost:15672/api/definitions | jq > definitions.json

Example: Using rabbitmq-backup Tool to Backup Messages

rabbitmq-backup --backup-messages --output-dir /backup/rabbitmq

Note: Message content is not included in the management plugin export. For full backups including messages, use rabbitmq-backup or file system backups.

Warning: File system backups are not recommended in production due to potential data corruption and inconsistency.

Property	Value	Description
erlang.cookie	`~/.erlang.cookie`	File containing the Erlang cookie used for node authentication

ls -l ~/.erlang.cookie
cp ~/.erlang.cookie /backup/erlang_cookie_backup

Note: The Erlang cookie file must have strict permissions (e.g., chmod 600) to prevent unauthorized access.

7. Implementing Automated Backups

Example: RabbitMQ Backup Script

#!/bin/bash
# Backup RabbitMQ configuration and queues

BACKUP_DIR="/var/backups/rabbitmq"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/rabbitmq_backup_$DATE.tar.gz"

# Create backup directory
mkdir -p $BACKUP_DIR

# Export RabbitMQ configuration
rabbitmqctl export_definitions /tmp/rabbitmq_definitions.json

# Compress backup files
tar -czf $BACKUP_FILE /tmp/rabbitmq_definitions.json

# Clean up temporary files
rm /tmp/rabbitmq_definitions.json

echo "Backup created: $BACKUP_FILE"

Configuration Option	Description	Default Value	Notes
management.tcp.port	Port for the management plugin	`15672`	Used for monitoring and backup operations
management.tcp.ip	IP address to bind the management plugin	`0.0.0.0`	Allows remote access to management interface
default_user	Default user for RabbitMQ	`guest`	Must be changed in production environments
default_pass	Default password for RabbitMQ	`guest`	Must be changed in production environments

Note: Ensure that the backup script has appropriate permissions to access RabbitMQ data and configuration files.

Warning: The default user guest should not be used in production environments. Always change the default credentials.

Backup Strategy Summary

Export definitions using rabbitmqctl export_definitions
Compress backup files into a single archive
Schedule backups using cron jobs or external tools
Store backups in a secure, accessible location

Automated Backup Workflow

Run backup script daily or hourly based on requirements
Verify that the backup file is created successfully
Transfer backup files to a remote storage system (e.g., S3, NAS)
Set up retention policies to delete old backups
Monitor backup success and failure logs

8. Using Shovel for Data Migration

The Shovel plugin in RabbitMQ is a powerful tool for migrating messages between different RabbitMQ instances or even across different AMQP 0-9-1 brokers. It allows you to move messages from one queue to another, either within the same broker or across different brokers.

Example: Configuring a Shovel

{
  "shovels": {
    "my-shovel": {
      "src-uri": "amqp://source-user:source-password@source-host:5672",
      "src-queue": "source-queue",
      "dest-uri": "amqp://dest-user:dest-password@dest-host:5672",
      "dest-queue": "dest-queue"
    }
  }
}

Property	Description	Default Value
src-uri	URI of the source broker	None
src-queue	Name of the source queue	None
dest-uri	URI of the destination broker	None
dest-queue	Name of the destination queue	None
ack-mode	Mode for acknowledging messages (none, consumer, or transactional)	consumer
reconnect-delay	Time in seconds to wait before reconnecting if connection fails	5

Note: Shovels can be configured using the RabbitMQ Management UI or via a JSON configuration file. Ensure that the URIs provided have appropriate permissions to access the queues.

Warning: When migrating data using Shovel, ensure that the destination queue is configured to handle the expected message load. If not, messages may be lost or delayed.

Shovel Configuration Summary

src-uri: Specifies the source broker's connection details
src-queue: The queue from which messages will be consumed
dest-uri: Specifies the destination broker's connection details
dest-queue: The queue where messages will be published
ack-mode: Determines how acknowledgments are handled during migration
reconnect-delay: Controls how long the shovel waits before attempting to reconnect

9. Testing Restore Procedures

Example: Basic Restore Test Script

#!/bin/bash
# Simulate restore from backup
echo "Starting restore test..."
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
echo "Restore test completed."

Step	Description	Status
1	Stop the RabbitMQ application	NEW
2	Reset the node to clean state	NEW
3	Start the RabbitMQ application	NEW

Note: Ensure all queues and exchanges are recreated after restore.

Warning: This process may cause temporary service disruption.

Restore Test Checklist

Stop application gracefully
Reset node configuration
Start application with restored data
Verify queue and exchange integrity

10. Replicating to Secondary Cluster

Property	Description	Values
cluster_name	Name of the RabbitMQ cluster	`secondary-cluster`
queue_name	Name of the queue to replicate	`replicated-queue`
node_type	Node type for replication	`disc` or `ram`

Example: Setting up replication

rabbitmqctl stop_app
rabbitmqctl join_cluster rabbit@primary-node
rabbitmqctl start_app

Note: Ensure all nodes are running the same version of RabbitMQ for replication to work correctly.

Warning: Replication is not a substitute for backup and recovery procedures.

Replication Type	Description	Status
Synchronous	Messages are replicated before acknowledgment	NEW
Asynchronous	Messages are replicated in the background	DEPRECATED

11. Handling Schema Upgrades

Schema Type	Description	Upgrade Method	Notes
Exchange	Message routing topology	`rabbitmqadmin declare exchange`	Ensure routing keys and bindings are preserved
Queue	Message storage and delivery	`rabbitmqadmin declare queue`	Recreate with same name and durable settings
Binding	Link between exchanges and queues	`rabbitmqadmin declare binding`	Recreate with same source and destination
Policy	Configuration rules for queues/exchanges	`rabbitmqadmin set policy`	Apply same pattern and parameters to new schema

Example: Schema Upgrade Script

rabbitmqadmin declare exchange name=my_exchange type=direct durable=true
rabbitmqadmin declare queue name=my_queue durable=true
rabbitmqadmin declare binding source=my_exchange destination=my_queue routing_key=my_key
rabbitmqadmin set policy name=ha-policy pattern=&#34;^my_queue
rabbitmqadmin declare exchange name=my_exchange type=direct durable=true
rabbitmqadmin declare queue name=my_queue durable=true
rabbitmqadmin declare binding source=my_exchange destination=my_queue routing_key=my_key
rabbitmqadmin set policy name=ha-policy pattern="^my_queue$" definition="{"ha-mode":"all"}"
#x26;#34; definition=&#34;{&#34;ha-mode&#34;:&#34;all&#34;}&#34;

Note: Always backup schema before applying upgrades to prevent data loss.

Warning: Schema upgrades may cause message loss if queue durability settings change.

12. Documenting Recovery Procedures

Example: Recovery Procedure Documentation Template

# RabbitMQ Recovery Procedure

## 1. Initial Assessment
- Check cluster status: `rabbitmqctl cluster_status`
- Identify failed nodes

## 2. Backup Restoration
- Restore from latest backup: `rabbitmqctl restore_backup <backup-file>`

## 3. Node Rejoining
- Start node: `rabbitmq-server start`
- Join cluster: `rabbitmqctl join_cluster rabbit@node1`

## 4. Validation
- Confirm queue recovery: `rabbitmqctl list_queues`
- Test message flow

## 5. Monitoring
- Monitor logs for errors: `tail -f /var/log/rabbitmq/rabbit@hostname.log`

Procedure Step	Description	Tools Required
Initial Assessment	Identify failed nodes and cluster status	`rabbitmqctl cluster_status`
Backup Restoration	Restore from latest backup file	`rabbitmqctl restore_backup`
Node Rejoining	Rejoin nodes to the cluster	`rabbitmqctl join_cluster`
Validation	Confirm queue and message recovery	`rabbitmqctl list_queues`
Monitoring	Monitor logs for errors post-recovery	`tail -f`

Note: Ensure backups are stored securely and tested regularly for validity.

Warning: Recovery procedures should be documented and tested in a non-production environment before use.

Implementing Backup and Recovery

1. Exporting Definitions

Example: Exporting Definitions Using rabbitmqctl

Example: Exporting Definitions Using rabbitmqadmin

2. Using Definitions File

Example: Exporting Definitions

Example: Importing Definitions

3. Importing Definitions

Example: Importing Definitions

4. Backing Up Configuration

Example: Exporting All Definitions

5. Backing Up Message Data

Example: Using RabbitMQ Management Plugin to Export Definitions

Example: Using rabbitmq-backup Tool to Backup Messages

6. Backing Up Erlang Cookie

Example: Locate and Backup Erlang Cookie

7. Implementing Automated Backups

Example: RabbitMQ Backup Script

Backup Strategy Summary

Automated Backup Workflow

8. Using Shovel for Data Migration

Example: Configuring a Shovel

Shovel Configuration Summary

9. Testing Restore Procedures

Example: Basic Restore Test Script

Restore Test Checklist

10. Replicating to Secondary Cluster

Example: Setting up replication

11. Handling Schema Upgrades

Example: Schema Upgrade Script

12. Documenting Recovery Procedures

Example: Recovery Procedure Documentation Template

Was this article helpful?

Related Cheat Sheets

Using Schema Registry

Working with Partitions

Configuring Producer Settings

Configuring Consumer Settings