Forward messages to RabbitMQ
This guide copies messages from NATS into a RabbitMQ queue. The RabbitMQ Shovel plugin connects to the gateway as an AMQP 1.0 client, reads an address, and sends each message on to RabbitMQ. NATS, MQTT and AMQP clients publish as usual, and RabbitMQ consumers read the queue. A message stays in its stream until RabbitMQ has it, so a RabbitMQ outage delays messages but does not lose them.
Before you start, you need:
- The
mq-gatewaybinary, with an AMQP listener. - A NATS server with JetStream.
- RabbitMQ with the
rabbitmq_shovelandrabbitmq_shovel_managementplugins. - A NATS user for the shovel, named
shovelhere. Configure listeners and auth sets its permissions. - A NATS user that can create consumers, and the
natsCLI.
1. Bind the subjects
nats:
url: nats://nats.example.com:4222
user: gateway
password: env://GATEWAY_PASSWORD
bindings:
- { subjects: [orders.>], stream: ORDERS }
amqp:
bindings: [ORDERS]
delivery-counts: { bucket: AMQP_COUNTS }
The shovel always asks for a durable source. The gateway gives a durable
source only on a subject that a binding stores. On any other subject, the
gateway refuses the shovel, and RabbitMQ shows its state as
terminated.
Provision the ORDERS stream and the AMQP_COUNTS bucket as in
Provision the streams and buckets.
Set the stream's max_age longer than the longest RabbitMQ outage you
expect. A message that the stream removes
before RabbitMQ has it is lost.
2. Make a queue for the shovel
By default, all receivers on an address share one queue. The gateway
removes that queue when no receiver uses it for amqp.queue-inactive
(1h). A new queue starts at the first message the stream holds. So after
a long outage, RabbitMQ gets those messages again. A consumer that you
make with no inactive threshold stays until you remove it.
Make a pull consumer on the stream:
nats consumer add ORDERS rabbitmq --pull --ack explicit --filter 'orders.>' --deliver new --defaults
- The filter must be the address that the shovel reads.
--deliver newstarts at the next message. Use--deliver allto also copy the messages that the stream holds now.
3. Create the shovel
Create a quorum queue named orders in RabbitMQ. A quorum queue keeps
every message on disk. A classic queue can lose a message that the
gateway sent as not durable, such as a plain NATS publish. Then create
the shovel:
rabbitmqctl set_parameter shovel nats-orders '{
"src-protocol": "amqp10",
"src-uri": "amqp://shovel:secret@gateway.example.com:5672?sasl=plain",
"src-address": "orders.>::rabbitmq",
"dest-protocol": "amqp10",
"dest-uri": "amqp://rabbit:secret@localhost:5672?sasl=plain",
"dest-address": "/queues/orders",
"ack-mode": "on-confirm"
}'
src-uriconnects to the gateway as the NATS usershovel. Useamqps://when the listener serves TLS.src-addressis the address, then::and the queue from step 2.dest-addressis the RabbitMQ queue.ack-mode: on-confirmmakes the shovel acknowledge a message to the gateway only after RabbitMQ stores it. The other modes can lose messages when RabbitMQ stops.
Check it works
-
Publish a message from NATS:
nats pub orders.us '{"order":1}' -H 'Mqg-Prop-source:checkout' -
In the RabbitMQ management UI, open Admin, then Shovel Status. The shovel
nats-ordersis running. -
Read the message from the RabbitMQ queue
orders. Its application propertysourceischeckout. -
Show the consumer:
nats consumer info ORDERS rabbitmqThe consumer shows 0 unprocessed messages and 0 pending acknowledgments.
Stop RabbitMQ and publish again. The consumer shows the new messages as unprocessed or pending acknowledgment. Start RabbitMQ, and the shovel sends the messages that waited.
What RabbitMQ gets
- The body. A NATS payload arrives as one
datasection. See Mapping to NATS. - Application properties. For a message that a NATS client
published, they come from its
Mqg-Prop-<key>headers. Other NATS headers do not become application properties. An AMQP sender's message keeps its own. See Mapping to NATS. - Duplicates. When the shovel's connection ends, the gateway delivers again each message that the shovel had not acknowledged. RabbitMQ can already hold some of these messages. A RabbitMQ consumer that must not process a message twice should discard duplicates.