Skip to main content
Mapagam

Implementing Shovel

1. Understanding Shovel Use Cases

Shovel Use Cases

  • Transferring messages between RabbitMQ instances
  • Implementing message routing across different networks
  • Enabling data migration or archival strategies
  • Supporting legacy systems integration with modern AMQP brokers
Use Case Description Benefits
Cross-Cluster Message Transfer Shovel can move messages between RabbitMQ clusters or nodes in different networks. Enables distributed messaging and load balancing across systems.
Legacy System Integration Shovel allows integration of older systems that may not support modern AMQP features. Provides a bridge for gradual migration or coexistence of systems.
Data Archiving Shovel can be used to archive messages from a production queue to an archival system. Supports long-term storage and compliance requirements.
Message Filtering Shovel can filter messages based on criteria like routing keys or message headers. Allows selective transfer of messages to specific destinations.
Note: Shovel is a RabbitMQ plugin that enables message transfer between different RabbitMQ instances or between AMQP 0-9-1 compatible systems.

2. Enabling Shovel Plugin

Command Description Notes
rabbitmq-plugins enable rabbitmq_shovel Enables the Shovel plugin Required for using shovel features
rabbitmq-plugins enable rabbitmq_shovel_management Enables the Shovel Management plugin Provides web UI and REST API for shovel configuration

Example: Enable Shovel Plugins

rabbitmq-plugins enable rabbitmq_shovel
rabbitmq-plugins enable rabbitmq_shovel_management
Note: After enabling the plugin, restart RabbitMQ for changes to take effect.

3. Configuring Static Shovel

Property Values Description
source.uri amqp://user:pass@host:port/vhost URI of the source broker or queue
destination.uri amqp://user:pass@host:port/vhost URI of the destination broker or queue
source.queue my-source-queue Name of the source queue (optional if using URI)
destination.queue my-dest-queue Name of the destination queue (optional if using URI)
batch_size 100 Number of messages to shovel in a batch
reconnect_delay 5000 Delay in milliseconds before reconnecting (default 5s)

Example: Static Shovel Configuration

{
  "shovels": {
    "my-shovel": {
      "source": {
        "uri": "amqp://source-host:5672/source-vhost",
        "queue": "source-queue"
      },
      "destination": {
        "uri": "amqp://dest-host:5672/dest-vhost",
        "queue": "dest-queue"
      },
      "batch_size": 100,
      "reconnect_delay": 5000
    }
  }
}
Note: Shovel configuration must be applied via the RabbitMQ Management HTTP API or through a configuration file.

4. Configuring Dynamic Shovel

Example: Declaring a Dynamic Shovel

{
  "shovels": {
    "my-dynamic-shovel": {
      "src-uri": "amqp://localhost:5672/source-vhost",
      "src-queue": "source-queue",
      "dest-uri": "amqp://localhost:5672/destination-vhost",
      "dest-queue": "destination-queue",
      "reconnect-delay": 5
    }
  }
}
Property Description Default Value
src-uri Source broker URI None
src-queue Source queue name None
dest-uri Destination broker URI None
dest-queue Destination queue name None
reconnect-delay Seconds to wait before reconnecting 5
Note: Dynamic shovels are managed via the rabbitmqctl command or the HTTP API.
Warning: Ensure that both source and destination brokers are accessible and have matching queue configurations.

5. Setting Source Configuration

Property Value Description
source.uri amqp://user:pass@localhost:5672/vhost URI of the source broker
source.queue my-queue Name of the queue to shovel from
source.exchange my-exchange Name of the exchange to shovel from (optional)
source.routing_key my-routing-key Routing key to filter messages (optional)
source.prefetch_count 10 Number of messages to prefetch from source (optional)

Example: Source Configuration

{
  "source": {
    "uri": "amqp://guest:guest@source-broker:5672/",
    "queue": "incoming-queue",
    "prefetch_count": 5
  }
}

6. Setting Destination Configuration

Property Value Description
destination.address amqp://user:pass@host:port/vhost AMQP address of the destination broker
destination.exchange my_exchange Name of the exchange to publish messages to
destination.routing_key my_routing_key Routing key for message routing
destination.queue my_queue Name of the queue to publish messages to
destination.virtual_host /my_vhost Virtual host to connect to on destination broker

Example: Destination Configuration

{
  "destination": {
    "address": "amqp://guest:guest@destination-broker:5672/my_vhost",
    "exchange": "shovel-exchange",
    "routing_key": "shovel.routing.key",
    "queue": "shovel-queue",
    "virtual_host": "/my_vhost"
  }
}

7. Using Shovel Acknowledgments

Property Description Default Value Notes
ack.mode Controls how acknowledgments are handled for messages transferred by the Shovel none Valid values: none, basic, publisher
ack.timeout Timeout for acknowledgment handling in milliseconds 30000 Used when ack.mode is set to basic

Example: Configuring Shovel Acknowledgments

{
  "shovel": {
    "ack.mode": "basic",
    "ack.timeout": 50000
  }
}
Note: When ack.mode is set to basic, the Shovel will wait for an acknowledgment from the destination broker before considering a message successfully transferred.

8. Setting Reconnect Delay

Property Values Description
reconnect_delay 5 Delay in seconds before attempting to reconnect after a connection loss
reconnect_delay_max 300 Maximum delay in seconds for exponential backoff
reconnect_delay_multiplier 2.0 Multiplier for exponential backoff calculation

Example: Configuring Reconnect Delay in Shovel

{
  "shovels": {
    "my-shovel": {
      "src-uri": "amqp://source-vhost",
      "dest-uri": "amqp://destination-vhost",
      "reconnect_delay": 10,
      "reconnect_delay_max": 300,
      "reconnect_delay_multiplier": 2.0
    }
  }
}

9. Using Shovel for Migration

RabbitMQ's Shovel plugin is a powerful tool for migrating messages between brokers or queues. It supports various migration strategies, including point-to-point and broadcast scenarios.

Configuration Description Example
source.uri URI of the source broker or queue amqp://user:pass@source-broker:5672/vhost
destination.uri URI of the destination broker or queue amqp://user:pass@destination-broker:5672/vhost
source.queue Source queue name (if applicable) my-source-queue
destination.queue Destination queue name (if applicable) my-destination-queue
ack.mode Message acknowledgment mode none, client, server

Example: Basic Migration Configuration

{
  "shovel": {
    "name": "migration-shovel",
    "source": {
      "uri": "amqp://user:pass@old-broker:5672/vhost",
      "queue": "old-queue"
    },
    "destination": {
      "uri": "amqp://user:pass@new-broker:5672/vhost",
      "queue": "new-queue"
    },
    "ack.mode": "client"
  }
}
Note: Ensure that the source and destination brokers are accessible from each other. The ack.mode setting affects how messages are acknowledged during migration.
Warning: Migrating messages with ack.mode set to none can lead to message loss if the migration process is interrupted.

Migration Strategy Summary

  • Point-to-Point Migration: Migrates messages from one queue to another, preserving order and ensuring no duplicates.
  • Broadcast Migration: Distributes messages to multiple queues or brokers simultaneously.
  • Incremental Migration: Migrates messages in batches to reduce load on the system.

10. Combining Shovel with Federation

RabbitMQ's shovel and federation plugins can be used together to create powerful message routing and replication strategies. While shovel is designed for point-to-point or one-way message forwarding, federation enables more complex topologies by allowing downstream brokers to federate with upstream ones.

Feature Description Use Case
Shovel Forwards messages from one broker to another, typically in a unidirectional manner. Replicating messages across brokers or forwarding to external systems.
Federation Enables downstream brokers to federate with upstream ones, supporting bidirectional communication and message replication. Building distributed systems where messages can be mirrored or routed across clusters.

Example: Configuring Federation and Shovel Together

{
  "federation": {
    "upstream": {
      "exchange": "my.exchange",
      "queue": "my.queue"
    }
  },
  "shovel": {
    "source": "amqp://upstream-broker",
    "destination": "amqp://local-broker"
  }
}
Note: When combining shovel and federation, ensure that the upstream broker supports both plugins to avoid configuration conflicts or message loss.

11. Monitoring Shovel Status

Property Value Description
shovel.status active | inactive Current operational status of the shovel
shovel.message_count number Number of messages currently queued for transfer
shovel.last_event_time timestamp Last time a message was successfully transferred
shovel.errors number Total number of errors encountered during transfer

Example: Check Shovel Status via CLI

rabbitmqctl list_shovels

Example: Query Shovel Metrics via Management API

GET /api/shovels
Note: Shovel status can be monitored through RabbitMQ's management UI or via the rabbitmqctl command-line tool.

12. Troubleshooting Shovel Issues

Common Shovel Problems and Solutions

  • Shovel not starting: Check permissions, network connectivity, and configuration syntax.
  • Message loss: Verify durable queues and exchanges; ensure message persistence settings.
  • Performance degradation: Adjust prefetch count and concurrency settings.
  • Connection timeouts: Increase timeout values or check broker availability.
Issue Possible Cause Solution
Shovel fails to start Invalid configuration or missing permissions Review shovel configuration JSON and user permissions
Messages not delivered Non-durable queues or missing persistence flags Ensure messages are marked as persistent and queues are durable
High latency Network issues or insufficient prefetch count Increase prefetch_count and check network bandwidth
Connection drops Broker unreachable or timeout settings too low Adjust reconnect_delay and heartbeat values

Example: Diagnosing Shovel Configuration Errors

{
  "shovel": {
    "name": "my-shovel",
    "src-uri": "amqp://user:pass@source-host:5672",
    "dest-uri": "amqp://user:pass@dest-host:5672",
    "src-queue": "source-queue",
    "dest-queue": "dest-queue"
  }
}
Note: Use rabbitmqctl list_shovels to check active shovels and their status.
Warning: Ensure that both source and destination brokers support the same AMQP version.

Shovel Troubleshooting Workflow

  1. Check shovel status using rabbitmqctl list_shovels
  2. Verify connection details and credentials in configuration
  3. Confirm queue durability and message persistence settings
  4. Review RabbitMQ logs for error messages related to shovel operations
  5. Adjust prefetch_count and reconnect_delay if needed

Was this article helpful?

Related Cheat Sheets