Debugging and Troubleshooting
1. Using rabbitmqctl Commands
| Command | Description | Usage |
|---|---|---|
| rabbitmqctl status | Displays the current status of the RabbitMQ node | rabbitmqctl status |
| rabbitmqctl list_queues | Lists all queues on the node | rabbitmqctl list_queues |
| rabbitmqctl list_exchanges | Lists all exchanges on the node | rabbitmqctl list_exchanges |
| rabbitmqctl list_bindings | Lists all bindings between exchanges and queues | rabbitmqctl list_bindings |
| rabbitmqctl list_connections | Lists all active connections | rabbitmqctl list_connections |
| rabbitmqctl list_channels | Lists all active channels | rabbitmqctl list_channels |
| rabbitmqctl eval | Evaluates Erlang expressions in the RabbitMQ node | rabbitmqctl eval "rabbit_amqqueue:info(\\"queue_name\\")." |
| rabbitmqctl wait | Waits for a specific condition to be met | rabbitmqctl wait /var/lib/rabbitmq/rabbitmq.pid |
Note: The
rabbitmqctl command must be run from the RabbitMQ installation directory or with the appropriate environment variables set.2. Enabling AMQP Traces
Example: Enable AMQP Traces in RabbitMQ
# Enable AMQP traces via rabbitmqctl
rabbitmqctl eval 'amqp_trace:start().'
# Stop AMQP traces
rabbitmqctl eval 'amqp_trace:stop().'| Configuration Option | Description | Default Value |
|---|---|---|
| amqp_trace.enable | Enables AMQP trace logging | false |
| amqp_trace.file | File path to write trace logs | /var/log/rabbitmq/amqp-trace.log |
| amqp_trace.level | Log level for trace output | debug |
Note: AMQP traces capture all incoming and outgoing AMQP frames. Use with caution in production environments due to performance impact.
3. Using Firehose
The Firehose feature in RabbitMQ allows you to capture and inspect all messages flowing through the broker. It's particularly useful for debugging and monitoring.
| Configuration | Description | Notes |
|---|---|---|
| firehose.enabled | true or false |
Enables or disables the firehose |
| firehose.queue | string |
Name of the queue where firehose messages are routed |
Example: Enabling Firehose via Management UI
curl -u guest:guest -X PUT http://localhost:15672/api/parameters/firehose/%2f/enable \
-H "Content-Type: application/json" \
-d '{"value": true}'Example: Firehose Output Message Format
{
"exchange": "amq.direct",
"routing_key": "test.key",
"message": {
"payload": "Hello World!",
"properties": {
"delivery_mode": 2,
"content_type": "text/plain"
}
},
"timestamp": "2026-01-01T12:00:00Z"
}
Note: Firehose can significantly impact performance if enabled for high-volume systems. It's recommended to enable it only during debugging sessions.
4. Analyzing Queue Backlogs
| Property | Description | Value |
|---|---|---|
| queue.length | Number of messages currently in the queue | 0 to ∞ |
| queue.consumer_count | Number of active consumers for the queue | 0 to ∞ |
| queue.messages_ready | Messages ready to be delivered to consumers | 0 to ∞ |
| queue.messages_unacknowledged | Messages delivered but not yet acknowledged by consumers | 0 to ∞ |
Example: Check queue backlog using RabbitMQ Management UI
curl -u guest:guest http://localhost:15672/api/queues/%2f/my_queue
Example: Monitor queue length via CLI
rabbitmqctl list_queues name messages_ready messages_unacknowledged
Note: A growing queue length with no increase in consumers may indicate a performance bottleneck or consumer lag.
Warning: If
messages_unacknowledged is consistently high, it might suggest consumers are not processing messages efficiently or are stuck.
5. Investigating Memory Issues
| Issue | Description | Symptoms | Resolution |
|---|---|---|---|
| High Memory Usage | Broker consumes excessive RAM |
|
|
| Memory Limit Exceeded | Broker hits configured memory limit |
|
|
Note: Memory usage can be influenced by message size, queue depth, and the number of connections. Use
rabbitmqctl list_queues to inspect queue details.
6. Debugging Consumer Issues
| Issue | Possible Causes | Solutions |
|---|---|---|
| Messages not being processed |
|
|
| High latency in message processing |
|
|
| Message redeliveries |
|
|
Example: Consumer Acknowledgment
const amqp = require('amqplib');
amqp.connect('amqp://localhost').then(connection => {
return connection.createChannel();
}).then(channel => {
const queue = 'task_queue';
channel.assertQueue(queue, { durable: true });
channel.consume(queue, (msg) => {
console.log('Received:', msg.content.toString());
// Process the message
setTimeout(() => {
channel.ack(msg); // Acknowledge the message
}, 1000);
}, { noAck: false });
});
Note: Always ensure that messages are acknowledged to prevent redeliveries and improve consumer performance.
Warning: Setting
noAck: true can lead to message loss if the consumer crashes.
7. Checking Network Connectivity
| Component | Status | Notes |
|---|---|---|
| RabbitMQ Port | 5672 |
Default AMQP port |
| Management Port | 15672 |
Default HTTP port for management plugin |
| Erlang Port | 25672 |
Erlang distribution port (used for clustering) |
Note: If telnet or curl fails, check firewall settings and ensure RabbitMQ is running.
Workflow: Troubleshooting Network Issues
- Verify RabbitMQ service is running
- Check if port 5672 is accessible via telnet or nc
- Ensure management plugin is enabled and port 15672 is accessible
- Review logs in
/var/log/rabbitmq/for errors - Confirm Erlang distribution port 25672 is open if clustering is used
8. Analyzing Logs
| Log Level | Description | Use Case |
|---|---|---|
| trace | Most verbose level, includes all internal operations and message flows | Debugging complex routing issues or internal state problems |
| debug | Includes detailed information about internal operations and message processing | Investigating performance bottlenecks or specific operation details |
| info | General operational information, such as startup, shutdown, and connection events | Monitoring system health and tracking normal operations |
| warning | Indicates potential issues that do not stop the system but may require attention | Identifying configuration warnings or recoverable errors |
| error | Reports errors that affect system functionality or message processing | Diagnosing failures in message delivery or system errors |
Example: Log Configuration in rabbitmq.conf
# Set log level to debug for detailed operation tracking
log.console.level = debug
# Enable file logging with rotation
log.file = /var/log/rabbitmq/rabbit.log
log.file.level = info
log.file.rotation.date = ".<date>"
log.file.rotation.size = 10M
Note: Log files can grow very large in high-volume environments. Consider implementing log rotation and archiving strategies to manage disk space.
Example: Using rabbitmqctl to view logs
rabbitmqctl status
rabbitmqctl eval "logger:info("Message processing delay detected")"
Log Analysis Best Practices
- Filter logs by log level to focus on relevant information
- Use structured logging formats for easier parsing and analysis
- Monitor log file sizes to prevent disk space exhaustion
- Correlate logs with system metrics for comprehensive troubleshooting
9. Using Management API
| Endpoint | Method | Description | Notes |
|---|---|---|---|
| /api/queues/{vhost}/{queue} | GET | Retrieve queue details | Requires vhost and queue name |
| /api/exchanges/{vhost}/{exchange} | GET | Retrieve exchange details | Use for verifying exchange configuration |
| /api/connections | GET | List active connections | Useful for monitoring connection health |
| /api/channels | GET | List active channels | Helps identify channel issues |
Note: The Management API requires the
rabbitmq_management plugin to be enabled.
Example: Deleting a Queue via API
curl -u guest:guest -X DELETE http://localhost:15672/api/queues/%2f/my-queue
10. Inspecting Message Content
| Property | Description | Example |
|---|---|---|
| message.id | Unique identifier for the message | msg12345 |
| message.correlation_id | Used to correlate related messages | corr56789 |
| message.reply_to | Queue name to reply to | reply.queue |
| message.timestamp | Timestamp when message was sent | 1634567890 |
| message.type | Message type for routing | order.created |
| message.headers | Additional metadata as key-value pairs | |
Example: Fetching Message Content via Management Plugin
curl -u guest:guest http://localhost:15672/api/queues/%2f/my-queue/get \
-d '{"count":1,"ackmode":"ack_requeue_false","encoding":"auto"}'Example: Inspecting Message Headers Programmatically (JavaScript)
const amqp = require('amqplib');
amqp.connect('amqp://localhost').then(conn => {
return conn.createChannel();
}).then(ch => {
ch.consume('my-queue', msg => {
console.log('Headers:', msg.properties.headers);
console.log('Type:', msg.properties.type);
ch.ack(msg);
});
});
Note: Message content is not stored by default in RabbitMQ. To inspect message content, you must enable
message-timestamp or use the management plugin.
11. Testing with PerfTest
RabbitMQ provides the perf_test tool to help you test and benchmark message throughput, latency, and other performance characteristics of your RabbitMQ setup.
Example: Running PerfTest
./perf_test.sh -u amqp://guest:guest@localhost:5672 -e test_exchange -q test_queue -m 1000| Option | Description | Default Value |
|---|---|---|
| -u | AMQP URI to connect to the broker | amqp://localhost:5672 |
| -e | Exchange name to bind the queue to | test_exchange |
| -q | Queue name to use for testing | test_queue |
| -m | Number of messages to send | 1000 |
| -c | Number of concurrent publishers | 1 |
| -p | Message payload size in bytes | 1024 |
Note: PerfTest is part of the RabbitMQ examples and must be built from source or downloaded separately.
Warning: Running PerfTest with large message counts can consume significant system resources and may impact other services.
12. Resolving Common Issues
Common RabbitMQ Issues and Solutions
- Connection Refused: Check if RabbitMQ service is running and port 5672 is accessible.
- Message Not Delivered: Verify queue and exchange bindings; ensure correct routing keys.
- High Memory Usage: Review message TTLs, purge unused queues, and optimize consumers.
- Consumer Lag: Increase prefetch count or scale consumer instances to handle load.
- Broker Not Responding: Restart RabbitMQ or check logs for critical errors.
| Issue | Possible Cause | Solution |
|---|---|---|
| Connection Refused | localhost:5672 not accessible or service down |
Start RabbitMQ server with rabbitmq-server or check firewall rules |
| Message Not Delivered | Incorrect routing key or missing binding | Verify exchange, queue, and binding configuration using rabbitmqadmin |
| High Memory Usage | Unbounded queues or long-running consumers | Set queue limits, enable message TTLs, or use max-length policy |
| Consumer Lag | Low prefetch count or slow consumer processing | Increase prefetchCount or scale consumers to match message rate |
| Broker Not Responding | Resource exhaustion or internal error | Restart RabbitMQ or inspect rabbit.log for stack traces |
Note: Use
rabbitmqctl status to inspect node health and memory usage.
Warning: Avoid setting
max-length too low, which may cause message loss.