Skip to main content
Version: 0.3

Deploy on Kubernetes

This guide runs two or more gateways behind one Service with Synadia's Helm chart. A rollout restarts them one at a time, so clients always have a gateway to reconnect to.

The chart runs the gateway only. NATS runs beside it, and you create the streams and buckets in NATS, as this guide shows.

Before you start, you need:

  • A Kubernetes cluster and Helm 3.
  • The nats CLI.
  • A license if you run a licensed build. Without one, you need credentials for registry.synadia.io.

1. Run NATS with JetStream​

Skip this step if your cluster can reach a NATS system with JetStream. Otherwise, install three NATS servers with the NATS Helm chart. Three servers keep serving while one restarts:

helm repo add nats https://nats-io.github.io/k8s/helm/charts/
helm install nats nats/nats --namespace nats --create-namespace \
  --set config.cluster.enabled=true --set config.cluster.replicas=3 \
  --set config.jetstream.enabled=true

2. Write the values​

The chart's config is the gateway's config file. This example serves MQTT and AMQP. It stores what clients publish to orders.> in the stream ORDERS. It keeps MQTT retained messages and sessions in two KV buckets:

# values.yaml
nats:
  url: nats://nats.nats.svc.cluster.local:4222

config:
  monitor:
    addr: ":8222"
  bindings:
    - { subjects: ["orders.>"], stream: ORDERS }
  mqtt:
    bindings: [ORDERS]
    retained: { bucket: MQTT_RETAINED }
    sessions: { bucket: MQTT_SESSIONS }
  amqp:
    bindings: [ORDERS]
    delivery-counts: { bucket: AMQP_COUNTS }

The sessions bucket lets a client resume on another gateway. Keep monitor.addr. Kubernetes probes the gateway's health endpoints on it.

The NATS servers from step 1 need no authentication. If yours do, the gateway connects as its own NATS user. Generate the permissions says what that user may do. Put its credentials file in a Secret that your values name:

kubectl create secret generic mq-gateway-nats --from-file=nats.creds=gateway.creds
nats:
  url: nats://nats.nats.svc.cluster.local:4222
  credentials:
    existingSecret: mq-gateway-nats

3. Install the gateway​

With a license, put it in a Secret and name the Secret in your values. The chart then runs the licensed image, registry.synadia.io/mq-gateway-licensed, which pulls without credentials:

kubectl create secret generic mq-gateway-license --from-file=license.jwt=license.jwt
license:
  existingSecret: mq-gateway-license

Without a license, the chart runs registry.synadia.io/mq-gateway. Put the registry credentials that Synadia sent you in a Secret, and name it in your values:

kubectl create secret docker-registry synadia-registry \
  --docker-server registry.synadia.io \
  --docker-username "$SYNADIA_REGISTRY_USERNAME" \
  --docker-password "$SYNADIA_REGISTRY_PASSWORD"
image:
  pullSecrets:
    - name: synadia-registry

Install the chart:

helm repo add synadia https://synadia-io.github.io/helm-charts
helm install mq-gateway synadia/mq-gateway -f values.yaml
kubectl get pods -l app.kubernetes.io/name=mq-gateway

Two pods start and become ready. Until the stream and buckets exist, each gateway refuses the MQTT and AMQP clients that need them. It logs account's startup check failed, trying again.

4. Create the stream and buckets​

The gateway never creates a stream or a bucket. generate prints the nats CLI commands that create the ones its config names. Run it in a gateway pod:

kubectl exec deploy/mq-gateway -- mq-gateway generate \
  --config /etc/mq-gateway/config/config.yaml --format nats-cli > provision.sh

The script stops before it creates anything if a limit has no value. Fill in the limits explains them. On the three NATS servers above, give each one 3 replicas.

Run the script as a NATS user that may create streams:

kubectl port-forward -n nats svc/nats 4222 &
NATS_URL=nats://127.0.0.1:4222 bash provision.sh

Each gateway finds the stream and buckets within about five seconds. It then serves the clients it refused.

5. Connect a client​

The Service mq-gateway takes MQTT on port 1883 and AMQP on port 5672. It sends each connection to one of the gateways. Reach one with kubectl port-forward:

kubectl port-forward svc/mq-gateway 1883 &
mqttx sub -h 127.0.0.1 -p 1883 -q 1 -t 'orders/#' -i device-7 --no-clean -se 3600

Publish from a second terminal:

mqttx pub -h 127.0.0.1 -p 1883 -q 1 -t 'orders/eu' -m 'first order'

An AMQP client uses the address orders.eu, the same NATS subject.

To reach the gateways from outside the cluster:

  1. Set service.type: LoadBalancer.
  2. Set tls.existingSecret to a TLS Secret. Clients then connect with mqtts:// and amqps://. Configure listeners and auth covers more ports.

6. Change the config​

Edit values.yaml and upgrade:

helm upgrade mq-gateway synadia/mq-gateway -f values.yaml

The gateway reads its config only at start, so a change restarts the gateways one at a time. Each new gateway answers /readyz before an old one stops. A stopping gateway enters lame duck mode and writes each session for another gateway to resume. Its clients reconnect through the Service to the other gateways.

A client with a persistent session, such as the --no-clean subscriber above, keeps its subscriptions. It also gets the QoS 1 and 2 messages published while it moved. A port-forward ends with the pod it reached, so start it again before the subscriber reconnects.

A stream or bucket that the new config names must exist before the upgrade. While the restart runs lists what else differs during a rollout.

Check it works​

  • This command prints nothing:

    kubectl exec deploy/mq-gateway -- mq-gateway check \
      --config /etc/mq-gateway/config/config.yaml

  • The subscriber above gets first order.

See also​