Deploy for high availability

This guide shows how to run three NATS servers and two or more gateways. With this setup, clients continue to work when one gateway or one NATS server stops.

Choose a setup

Setup	Use it for	If a gateway stops
One process, embedded NATS (nats.embed)	Trials, a single machine	All service stops until the process restarts
One gateway, external NATS	Development, small sites	Clients wait for the restart
Several gateways, NATS cluster	Production	Clients reconnect to another gateway and resume their sessions
Gateways on a leaf node	A site with a link to a central NATS system	Same as the row above. For traffic that must continue when the link is down, bind streams to a JetStream domain at the site

This guide uses the third setup.

Before you start, you need:

• A config file.
• The NATS users from Configure listeners and auth.

1. Run a three-server NATS cluster

Use the same configuration on each server. Change only server_name:

server_name: nats-1
listen: 0.0.0.0:4222
http: 8222
jetstream { store_dir: /var/lib/nats }
cluster {
  name: mq
  listen: 0.0.0.0:6222
  routes: [nats://nats-1:6222, nats://nats-2:6222, nats://nats-3:6222]
}
include "accounts.conf"

Then provision the streams and buckets. Set three replicas for each:

export ORDERS_REPLICAS=3 KV_MQTT_SESSIONS_REPLICAS=3

2. Start the gateways

Use the same config file on each gateway. In nats.url, list all NATS servers. This spreads the gateway connections across the cluster. Name a sessions bucket. Clients can then move between gateways.

nats:
  url: nats://nats-1:4222,nats://nats-2:4222,nats://nats-3:4222
  user: gateway
  password: env://GATEWAY_PASSWORD
monitor: { addr: ":8222" }
bindings:
  - { subjects: [orders.>], stream: ORDERS }
mqtt:
  bindings: [ORDERS]
  sessions: { bucket: MQTT_SESSIONS }

Check the config against the cluster. Then start each gateway with its own --id. Two gateways must never use the same id.

mq-gateway check --config gateway.yaml
mq-gateway --config gateway.yaml --id gw-1     # first host
mq-gateway --config gateway.yaml --id gw-2     # second host

3. Give clients one address

Put a TCP load balancer in front of the MQTT and AMQP ports of the gateways. Set its health check to /readyz on monitor.addr:

curl -i http://gw-1:8222/readyz

/readyz returns 200 while the gateway serves clients. It returns 503 during start-up, during a reconnect to NATS, and in lame duck mode.

A DNS name with one record for each gateway also works. This works only if clients try the next address when a connection attempt fails.

4. Check that a session moves

Connect a persistent session to one gateway. Then connect the same client ID to the other gateway:

mqttx sub -h gw-1 -p 1883 -u device -P device-secret -q 1 -t 'orders/#' -i device-7 --no-clean -se 3600 -rp 0
mqttx sub -h gw-2 -p 1883 -u device -P device-secret -q 1 -t 'orders/#' -i device-7 --no-clean -se 3600

gw-1 disconnects the first client. The second client gets the session. The session has its subscription and the messages published between the two connections.

On Kubernetes

Run two or more replicas. Give each replica its own id. You can put the pod name in MQGW_ID.

Start lame duck mode in a preStop hook. A rollout then moves clients before a pod stops. Set the grace period to ldm.duration plus 20s:

terminationGracePeriodSeconds: 140   # ldm.duration 2m + 20s
containers:
  - name: mq-gateway
    lifecycle:
      preStop:
        exec:
          command: ["/usr/local/bin/mq-gateway", "signal", "ldm"]
    readinessProbe:
      httpGet: {path: /readyz, port: 8222}
    livenessProbe:
      httpGet: {path: /livez, port: 8222}

Use /livez for the liveness probe. Do not use /readyz. /readyz fails during lame duck mode. As a liveness probe, it would stop the pod before the gateway moves all its clients.

What clients see

• A gateway restarts in lame duck mode: the gateway moves its clients to the other gateways, a few at a time. The clients resume their sessions there. See Change the config.
• A gateway fails: its clients reconnect to the other gateways immediately. If no client resumes a session, the other gateways end that session two to three mqtt.sessions.sweep intervals later (2–3 minutes by default). They publish its will at that time.
• A NATS server stops: most clients stay connected while their NATS connections move. Stored deliveries pause for a few seconds while the stream elects a new leader. The gateway may disconnect a publisher that has a message in flight. The publisher sends the message again when it reconnects.

To restart the NATS servers, do these steps for one server at a time:

1. Stop the server in lame duck mode (nats-server --signal ldm=<pid>).
2. Wait until the process exits.
3. Start the server again.
4. Wait until its /healthz returns 200.
5. Go to the next server.