Skip to main content
Version: 0.4

Forward messages to RabbitMQ

This guide copies messages from NATS into a RabbitMQ queue. The RabbitMQ Shovel plugin connects to the gateway as an AMQP 1.0 client, reads an address, and sends each message on to RabbitMQ. NATS, MQTT and AMQP clients publish as usual, and RabbitMQ consumers read the queue. A message stays in its stream until RabbitMQ has it, so a RabbitMQ outage delays messages but does not lose them.

Before you start, you need:

  • The mq-gateway binary, with an AMQP listener.
  • A NATS server with JetStream.
  • RabbitMQ with the rabbitmq_shovel and rabbitmq_shovel_management plugins.
  • A NATS user for the shovel, named shovel here. Configure listeners and auth sets its permissions.
  • A NATS user that can create consumers, and the nats CLI.

1. Bind the subjects​

nats:
url: nats://nats.example.com:4222
user: gateway
password: env://GATEWAY_PASSWORD
bindings:
- { subjects: [orders.>], stream: ORDERS }
amqp:
bindings: [ORDERS]
delivery-counts: { bucket: AMQP_COUNTS }

The shovel always asks for a durable source. The gateway gives a durable source only on a subject that a binding stores. On any other subject, the gateway refuses the shovel, and RabbitMQ shows its state as terminated.

Provision the ORDERS stream and the AMQP_COUNTS bucket as in Provision the streams and buckets. Set the stream's max_age longer than the longest RabbitMQ outage you expect. A message that the stream removes before RabbitMQ has it is lost.

2. Make a queue for the shovel​

By default, all receivers on an address share one queue. The gateway removes that queue when no receiver uses it for amqp.queue-inactive (1h). A new queue starts at the first message the stream holds. So after a long outage, RabbitMQ gets those messages again. A consumer that you make with no inactive threshold stays until you remove it.

Make a pull consumer on the stream:

nats consumer add ORDERS rabbitmq --pull --ack explicit --filter 'orders.>' --deliver new --defaults
  • The filter must be the address that the shovel reads.
  • --deliver new starts at the next message. Use --deliver all to also copy the messages that the stream holds now.

3. Create the shovel​

Create a quorum queue named orders in RabbitMQ. A quorum queue keeps every message on disk. A classic queue can lose a message that the gateway sent as not durable, such as a plain NATS publish. Then create the shovel:

rabbitmqctl set_parameter shovel nats-orders '{
"src-protocol": "amqp10",
"src-uri": "amqp://shovel:secret@gateway.example.com:5672?sasl=plain",
"src-address": "orders.>::rabbitmq",
"dest-protocol": "amqp10",
"dest-uri": "amqp://rabbit:secret@localhost:5672?sasl=plain",
"dest-address": "/queues/orders",
"ack-mode": "on-confirm"
}'
  • src-uri connects to the gateway as the NATS user shovel. Use amqps:// when the listener serves TLS.
  • src-address is the address, then :: and the queue from step 2.
  • dest-address is the RabbitMQ queue.
  • ack-mode: on-confirm makes the shovel acknowledge a message to the gateway only after RabbitMQ stores it. The other modes can lose messages when RabbitMQ stops.

Check it works​

  1. Publish a message from NATS:

    nats pub orders.us '{"order":1}' -H 'Mqg-Prop-source:checkout'
  2. In the RabbitMQ management UI, open Admin, then Shovel Status. The shovel nats-orders is running.

  3. Read the message from the RabbitMQ queue orders. Its application property source is checkout.

  4. Show the consumer:

    nats consumer info ORDERS rabbitmq

    The consumer shows 0 unprocessed messages and 0 pending acknowledgments.

Stop RabbitMQ and publish again. The consumer shows the new messages as unprocessed or pending acknowledgment. Start RabbitMQ, and the shovel sends the messages that waited.

What RabbitMQ gets​

  • The body. A NATS payload arrives as one data section. See Mapping to NATS.
  • Application properties. For a message that a NATS client published, they come from its Mqg-Prop-<key> headers. Other NATS headers do not become application properties. An AMQP sender's message keeps its own. See Mapping to NATS.
  • Duplicates. When the shovel's connection ends, the gateway delivers again each message that the shovel had not acknowledged. RabbitMQ can already hold some of these messages. A RabbitMQ consumer that must not process a message twice should discard duplicates.