# Deploy on Kubernetes

This guide runs two or more gateways behind one Service with Synadia's [Helm chart](https://artifacthub.io/packages/helm/synadia/mq-gateway). A rollout restarts them one at a time, so clients always have a gateway to reconnect to.

The chart runs the gateway only. NATS runs beside it, and you create the streams and buckets in NATS, as this guide shows.

Before you start, you need:

* A Kubernetes cluster and Helm 3.
* The [`nats` CLI](https://github.com/nats-io/natscli).
* A license if you run a [licensed build](/mq-gateway/0.4/how-to/install.md#run-a-licensed-build). Without one, you need credentials for `registry.synadia.io`.

## 1. Run NATS with JetStream

Skip this step if your cluster can reach a NATS system with JetStream. Otherwise, install three NATS servers with the [NATS Helm chart](https://github.com/nats-io/k8s/tree/main/helm/charts/nats). Three servers keep serving while one restarts:

```
helm repo add nats https://nats-io.github.io/k8s/helm/charts/

helm install nats nats/nats --namespace nats --create-namespace \

  --set config.cluster.enabled=true --set config.cluster.replicas=3 \

  --set config.jetstream.enabled=true
```

## 2. Write the values

The chart's `config` is the gateway's [config file](/mq-gateway/0.4/reference/config-file.md). This example serves MQTT and AMQP. It stores what clients publish to `orders.>` in the stream `ORDERS`. It keeps MQTT retained messages and sessions in two KV buckets:

```
# values.yaml

nats:

  url: nats://nats.nats.svc.cluster.local:4222



config:

  monitor:

    addr: ":8222"

  bindings:

    - { subjects: ["orders.>"], stream: ORDERS }

  mqtt:

    bindings: [ORDERS]

    retained: { bucket: MQTT_RETAINED }

    sessions: { bucket: MQTT_SESSIONS }

  amqp:

    bindings: [ORDERS]

    delivery-counts: { bucket: AMQP_COUNTS }
```

The sessions bucket lets a client resume on another gateway. Keep `monitor.addr`. Kubernetes probes the gateway's [health endpoints](/mq-gateway/0.4/reference/cli.md#health-endpoint) on it.

The NATS servers from step 1 need no authentication. If yours do, the gateway connects as its own NATS user. [Generate the permissions](/mq-gateway/0.4/how-to/configure-listeners-and-auth.md#2-generate-the-permissions) says what that user may do. Put its credentials file in a Secret that your values name:

```
kubectl create secret generic mq-gateway-nats --from-file=nats.creds=gateway.creds
```

```
nats:

  url: nats://nats.nats.svc.cluster.local:4222

  credentials:

    existingSecret: mq-gateway-nats
```

## 3. Install the gateway

With a license, put it in a Secret and name the Secret in your values. The chart then runs the licensed image, `registry.synadia.io/mq-gateway-licensed`, which pulls without credentials:

```
kubectl create secret generic mq-gateway-license --from-file=license.jwt=license.jwt
```

```
license:

  existingSecret: mq-gateway-license
```

Without a license, the chart runs `registry.synadia.io/mq-gateway`. Put the registry credentials that Synadia sent you in a Secret, and name it in your values:

```
kubectl create secret docker-registry synadia-registry \

  --docker-server registry.synadia.io \

  --docker-username "$SYNADIA_REGISTRY_USERNAME" \

  --docker-password "$SYNADIA_REGISTRY_PASSWORD"
```

```
image:

  pullSecrets:

    - name: synadia-registry
```

Install the chart:

```
helm repo add synadia https://synadia-io.github.io/helm-charts

helm install mq-gateway synadia/mq-gateway -f values.yaml

kubectl get pods -l app.kubernetes.io/name=mq-gateway
```

Two pods start and become ready. Until the stream and buckets exist, each gateway refuses the MQTT and AMQP clients that need them. It logs `account's startup check failed, trying again`.

## 4. Create the stream and buckets

The gateway never creates a stream or a bucket. `generate` prints the `nats` CLI commands that create the ones its config names. Run it in a gateway pod:

```
kubectl exec deploy/mq-gateway -- mq-gateway generate \

  --config /etc/mq-gateway/config/config.yaml --format nats-cli > provision.sh
```

The script stops before it creates anything if a limit has no value. [Fill in the limits](/mq-gateway/0.4/how-to/provision-streams-and-buckets.md#2-fill-in-the-limits) explains them. On the three NATS servers above, give each one `3` replicas.

Run the script as a NATS user that may create streams:

```
kubectl port-forward -n nats svc/nats 4222 &

NATS_URL=nats://127.0.0.1:4222 bash provision.sh
```

Each gateway finds the stream and buckets within about five seconds. It then serves the clients it refused.

## 5. Connect a client

The Service `mq-gateway` takes MQTT on port 1883 and AMQP on port 5672. It sends each connection to one of the gateways. Reach one with `kubectl port-forward`:

```
kubectl port-forward svc/mq-gateway 1883 &

mqttx sub -h 127.0.0.1 -p 1883 -q 1 -t 'orders/#' -i device-7 --no-clean -se 3600
```

Publish from a second terminal:

```
mqttx pub -h 127.0.0.1 -p 1883 -q 1 -t 'orders/eu' -m 'first order'
```

An AMQP client uses the address `orders.eu`, the same NATS subject.

To reach the gateways from outside the cluster:

1. Set `service.type: LoadBalancer`.
2. Set `tls.existingSecret` to a TLS Secret. Clients then connect with `mqtts://` and `amqps://`. [Configure listeners and auth](/mq-gateway/0.4/how-to/configure-listeners-and-auth.md) covers more ports.

## 6. Change the config

Edit `values.yaml` and upgrade:

```
helm upgrade mq-gateway synadia/mq-gateway -f values.yaml
```

The gateway reads its config only at start, so a change restarts the gateways one at a time. Each new gateway answers `/readyz` before an old one stops. A stopping gateway enters [lame duck mode](/mq-gateway/0.4/how-to/change-the-config.md#3-restart-the-gateways-one-at-a-time) and writes each session for another gateway to resume. Its clients reconnect through the Service to the other gateways.

A client with a persistent session, such as the `--no-clean` subscriber above, keeps its subscriptions. It also gets the QoS 1 and 2 messages published while it moved. A port-forward ends with the pod it reached, so start it again before the subscriber reconnects.

A stream or bucket that the new config names must exist before the upgrade. [While the restart runs](/mq-gateway/0.4/how-to/change-the-config.md#while-the-restart-runs) lists what else differs during a rollout.

## Check it works

* This command prints nothing:

  ```
  kubectl exec deploy/mq-gateway -- mq-gateway check \

    --config /etc/mq-gateway/config/config.yaml
  ```

* The subscriber above gets `first order`.

## See also

* The [chart's README](https://artifacthub.io/packages/helm/synadia/mq-gateway) lists every value, such as TLS to NATS and scheduling.
* [Deploy for high availability](/mq-gateway/0.4/how-to/deploy-for-high-availability.md) shows how gateways share sessions. Its [On Kubernetes](/mq-gateway/0.4/how-to/deploy-for-high-availability.md#on-kubernetes) section covers the lame duck hook and probes that the chart sets.
