# Deploy for high availability

This guide shows how to run three NATS servers and two or more gateways. With this setup, clients continue to work when one gateway or one NATS server stops.

## Choose a setup

| Setup                                     | Use it for                                  | If a gateway stops                                                                                                            |
| ----------------------------------------- | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| One process, embedded NATS (`nats.embed`) | Trials, a single machine                    | All service stops until the process restarts                                                                                  |
| One gateway, external NATS                | Development, small sites                    | Clients wait for the restart                                                                                                  |
| Several gateways, NATS cluster            | Production                                  | Clients reconnect to another gateway and resume their sessions                                                                |
| Gateways on a leaf node                   | A site with a link to a central NATS system | Same as the row above. For traffic that must continue when the link is down, bind streams to a JetStream `domain` at the site |

This guide uses the third setup.

Before you start, you need:

* A config file.
* The NATS users from [Configure listeners and auth](/mq-gateway/0.4/how-to/configure-listeners-and-auth.md).

## 1. Run a three-server NATS cluster

Use the same configuration on each server. Change only `server_name`:

```
server_name: nats-1

listen: 0.0.0.0:4222

http: 8222

jetstream { store_dir: /var/lib/nats }

cluster {

  name: mq

  listen: 0.0.0.0:6222

  routes: [nats://nats-1:6222, nats://nats-2:6222, nats://nats-3:6222]

}

include "accounts.conf"
```

Then [provision the streams and buckets](/mq-gateway/0.4/how-to/provision-streams-and-buckets.md). Set three replicas for each:

```
export ORDERS_REPLICAS=3 KV_MQTT_SESSIONS_REPLICAS=3
```

## 2. Start the gateways

Use the same config file on each gateway. In `nats.url`, list all NATS servers. This spreads the gateway connections across the cluster. Name a sessions bucket. Clients can then move between gateways.

```
nats:

  url: nats://nats-1:4222,nats://nats-2:4222,nats://nats-3:4222

  user: gateway

  password: env://GATEWAY_PASSWORD

monitor: { addr: ":8222" }

bindings:

  - { subjects: [orders.>], stream: ORDERS }

mqtt:

  bindings: [ORDERS]

  sessions: { bucket: MQTT_SESSIONS }
```

Check the config against the cluster. Then start each gateway with its own `--id`. Two gateways must never use the same id.

```
mq-gateway check --config gateway.yaml

mq-gateway --config gateway.yaml --id gw-1     # first host

mq-gateway --config gateway.yaml --id gw-2     # second host
```

## 3. Give clients one address

Put a TCP load balancer in front of the MQTT and AMQP ports of the gateways. Set its health check to `/readyz` on `monitor.addr`:

```
curl -i http://gw-1:8222/readyz
```

`/readyz` returns `200` while the gateway serves clients. It returns `503` during start-up, during a reconnect to NATS, and in lame duck mode.

A DNS name with one record for each gateway also works. This works only if clients try the next address when a connection attempt fails.

## 4. Check that a session moves

Connect a persistent session to one gateway. Then connect the same client ID to the other gateway:

```
mqttx sub -h gw-1 -p 1883 -u device -P device-secret -q 1 -t 'orders/#' -i device-7 --no-clean -se 3600 -rp 0

mqttx sub -h gw-2 -p 1883 -u device -P device-secret -q 1 -t 'orders/#' -i device-7 --no-clean -se 3600
```

`gw-1` disconnects the first client. The second client gets the session. The session has its subscription and the messages published between the two connections.

## On Kubernetes

The Helm chart does all of this. See [Deploy on Kubernetes](/mq-gateway/0.4/how-to/deploy-on-kubernetes.md). Without the chart, set these yourself.

Run two or more replicas. Give each replica its own id. You can put the pod name in `MQG_ID`.

Start lame duck mode in a `preStop` hook. A rollout then moves clients before a pod stops. Set the grace period to `ldm.duration` plus 20s:

```
terminationGracePeriodSeconds: 140   # ldm.duration 2m + 20s

containers:

  - name: mq-gateway

    lifecycle:

      preStop:

        exec:

          command: ["/usr/local/bin/mq-gateway", "signal", "ldm"]

    readinessProbe:

      httpGet: {path: /readyz, port: 8222}

    livenessProbe:

      httpGet: {path: /livez, port: 8222}
```

Use `/livez` for the liveness probe. Do not use `/readyz`. `/readyz` fails during lame duck mode. As a liveness probe, it would stop the pod before the gateway moves all its clients.

## What clients see

* **A gateway restarts in lame duck mode:** the gateway moves its clients to the other gateways, a few at a time. The clients resume their sessions there. See [Change the config](/mq-gateway/0.4/how-to/change-the-config.md).
* **A gateway fails:** its clients reconnect to the other gateways immediately. If no client resumes a session, the other gateways end that session two to three `mqtt.sessions.sweep` intervals later (2–3 minutes by default). They publish its will at that time.
* **A NATS server stops:** most clients stay connected while their NATS connections move. Stored deliveries pause for a few seconds while the stream elects a new leader. The gateway may disconnect a publisher that has a message in flight. The publisher sends the message again when it reconnects.

To restart the NATS servers, do these steps for one server at a time:

1. Stop the server in lame duck mode (`nats-server --signal ldm=<pid>`).
2. Wait until the process exits.
3. Start the server again.
4. Wait until its `/healthz` returns `200`.
5. Go to the next server.
