Federated Deployment

Monitor NATS systems at several sites and see all of them in one web UI.

In a federated deployment, each site runs Insights next to its own NATS system, and a central site serves the web UI. Each site connects to the central site's NATS server over a leafnode that carries only the account Insights uses. The monitored system's system account never leaves its site, and the central site never connects to the monitored systems.

Choose What Runs at Each Site

Each site runs one of two things. The network setup is the same for both. What differs is where the data is stored and checked.

	Insights node (recommended)	Collector
Runs at the site	insights, the full binary	insights-collector, a smaller binary that only scrapes
Database and checks	At the site	At the central site
Data kept at the site	History for db.retention.duration	A short buffer until the central site copies it
Runs at the central site	insights web, or any Insights node	An Insights node that stores and checks the site's data

Run a full Insights node at a site unless you have a reason not to. Each site keeps its own data and runs its own checks, and nothing is copied between sites.

Run a collector when a site can't spare the memory and disk for a database, or when monitoring data should not be kept at the site. The central site then copies the site's scrape stream with a JetStream mirror and does the storage and checks there. The collector has no DuckDB, no web UI, no Prometheus endpoint, and no embedded NATS server. It scrapes, publishes to a stream, and serves the part of the $INS API that needs no database.

You can mix both in one deployment.

If a central node can reach a monitored system directly, it doesn't need either: list the system under systems: with a scrape: block and the node scrapes it itself. See Multiple Systems on One Node.

Connect the Sites

Every site needs the following, whichever option it runs:

• A NATS server at the site with JetStream enabled and its own JetStream domain.
• A NATS server at the central site with JetStream enabled and a different domain.
• A leafnode from the site to the central site that carries the account Insights uses. This guide calls that account INSIGHTS. The site's system account is not bridged.
• A unique system.id for each site's system, and a unique node.id for each Insights process. This guide uses east for a site with an Insights node and west for a site with a collector.
• Network access from each site to the central site on the leafnode port (7422 by convention). The central site never connects to a site.

The nats CLI commands below were written against v0.4.x.

The system id names everything Insights creates for that system: the $INS.sys.<id>.* API, the scrape_<id> stream, and its subjects, $INS.sys.<id>.scrape.>. Because every subject includes the id, sites can share the INSIGHTS account without their subjects colliding. The subject namespace is never configured.

This is not access control: any user in the account can query every site. If sites must not see each other's data, give each Insights user permissions scoped to its own system (see Insights Permissions) Separate accounts per site would also stop the central web UI from finding the sites, because it discovers and queries them within one account.

Site NATS Server

A domain is how JetStream tells two JetStreams apart over one leafnode: a stream in one domain is reached from the other through the domain's API prefix, $JS.<domain>.API. Give each site's domain the same name as its system id, so $JS.west.API clearly refers to site west. Nothing requires it, but with several sites the alternative is remembering which domain is which.

Bridge only the INSIGHTS account:

# site-west.conf
server_name: west-1
listen: 0.0.0.0:4222

jetstream {
  domain: west
  store_dir: /data/nats/jetstream
}

leafnodes {
  remotes = [
    {
      url: "nats-leaf://insights:insights-password@central.example.com:7422"
      account: INSIGHTS
    }
  ]
}

accounts {
  INSIGHTS: {
    jetstream: enabled
    users: [ { user: insights, password: insights-password } ]
  }
  SYS: {
    users: [ { user: sys, password: sys-password } ]
  }
}
system_account: SYS

Two accounts matter here:

Account	Holds	Reachable from the central site
SYS	The monitored system's system account, which Insights scrapes	No
INSIGHTS	The scrape stream and the $INS.sys.<id>.* API	Yes

Insights at the site holds a connection to each: it reads from SYS locally and writes to INSIGHTS. Only INSIGHTS crosses the leafnode.

The examples on this page use passwords to stay short. In production, use your usual NATS authentication, such as JWT credentials, and point Insights at it with creds or context.

Sharing an account is not sharing a JetStream. The domains separate the two JetStreams and the account separates the traffic. A leaf server with no jetstream block does not gain one by being bridged into a JetStream-enabled account.

Central NATS Server

The central NATS server runs JetStream under its own domain, accepts leafnode connections, and defines the same INSIGHTS account. A leafnode joins the account of the user it logs in as, so sites that log in as insights join only that account.

# central.conf
server_name: central-1
listen: 0.0.0.0:4222

jetstream {
  domain: central
  store_dir: /data/nats/jetstream
}

leafnodes {
  port: 7422
}

accounts {
  INSIGHTS: {
    jetstream: enabled
    users: [ { user: insights, password: insights-password } ]
  }
}

Insights Permissions

If you restrict the users Insights connects with, this is a working baseline for a site node, a collector, or a central node. It is the same connection shape as any external NATS server, described in Sink connection permissions:

permissions {
  publish {
    allow = [
      "$INS.>",     # scrape data, discovery, and the Insights API
      "$JS.API.>",  # stream management
      "$JS.FC.>",   # flow control replies
      "_INBOX.>"    # request replies
    ]
  }
  subscribe {
    allow = [
      "_INBOX.>",   # request replies
      "$INS.>",     # discovery and the Insights API
      "$SRV.>"      # NATS service discovery
    ]
  }
}

Everything Insights owns is under $INS, including the scrape stream, so the grant doesn't have to list it. To limit a connection to one system, replace $INS.> with $INS.sys.<id>.>, $INS.node.<node_id>.>, and $INS.ping.

Node Ids

Each Insights process checks its identity over $INS.ping at startup, and that request crosses the leafnode. A node's node.id defaults to its system id when it serves one system, so two processes serving the same system (a collector and a single-system central node, for example) clash unless at least one is named. The second to start stops with:

node.id "west" is already in use on this NATS

Sharing a system id is fine when the roles differ: a collector holds the system's scrape role and the central node its catalog role.

Option 1: Insights Node at Each Site

The site node is an ordinary single-system Insights node. It scrapes the site's system, stores the data, and runs checks. What makes it part of a federation is its connection to the site's INSIGHTS account, and turning off the embedded NATS server so its stream lives on the site NATS server. There is no federation-specific config:

# east.yaml, run with: insights -c east.yaml
node:
  id: east

system:
  id: east

data-dir: /var/lib/insights

# The INSIGHTS account on the site NATS server. The central site reaches this
# node's API through the leafnode.
nats:
  server: nats://site-east:4222
  user: insights
  password: insights-password

# The site's system account, which is what Insights monitors.
sys:
  server: nats://site-east:4222
  user: sys
  password: sys-password

sink:
  embed: false

A trial build also needs a license: section on every site node.

The site node can also serve its own web UI. Leave web enabled if people at the site should use it directly.

At the central site, run a web UI connected to the central NATS server's INSIGHTS account. It needs no database and no license:

insights web \
  --server nats://central:4222 \
  --user insights --password insights-password \
  --web.hostname 0.0.0.0

The web UI finds every system that answers on the shared account over $INS.ping and lists them on its Systems page. It looks for systems when it starts, so restart it after you add a site. A full Insights node on the central NATS server does the same, so if the central site also monitors its own system, that node's web UI can show every site.

Check that every site is reachable from the central site:

insights system list --server nats://central:4222 --user insights --password insights-password
insights db query --server nats://central:4222 --user insights --password insights-password \
  --system east "SELECT count(*) FROM hx.servers"

Option 2: Collector at a Site

A collector scrapes the site's system and publishes the data to a stream on the site NATS server. A central Insights node copies that stream to the central NATS server with a JetStream mirror, then stores the data and runs checks.

The collector is the insights-collector release archive, the -collector image tag (registry.synadia.io/insights:1.2.0-collector), or task build:collector from source.

Collector Config

The collector monitors one system, so it uses the single-system config with no systems: list:

# collector.yaml, run with: insights-collector -c collector.yaml
node:
  # Must differ from the central node's node.id.
  id: west-collector

system:
  # Must match the system id configured on the central node.
  id: west

# The INSIGHTS account on the site NATS server, where the scrape stream lives.
nats:
  server: nats://site-west:4222
  user: insights
  password: insights-password

# The site's system account, which is what the collector scrapes.
sys:
  server: nats://site-west:4222
  user: sys
  password: sys-password

scraper:
  interval: 1m

sink:
  # How much data the stream keeps.
  retention: 6h

# A trial build also validates a license:
# license:
#   file: /etc/insights/license.jwt

Run it with insights-collector -c collector.yaml. serve is the collector's default command, so insights-collector serve -c collector.yaml does the same thing. From the collector image, with collector.yaml and the files it names mounted at /etc/insights:

docker run -v /path/to/collector:/etc/insights:ro \
  registry.synadia.io/insights:1.2.0-collector \
  -c /etc/insights/collector.yaml

The collector reads only the node, system, nats, sys, scraper, sink, license, updater, telemetry, and data-dir settings, plus log-level. It has no database or embedded stream store, so it doesn't need a data directory. If no NATS server is configured, it stops at startup:

collector requires an external NATS sink: set --server (or --context, or --scraper.nats.server) to the local leafnode that stores the scrape stream and serves the $INS API

See the collector build reference for what the binary includes, and insights-collector serve for its flags.

The Site Stream

On startup, the collector creates the scrape_west stream on the site NATS server with the subject $INS.sys.west.scrape.>. It derives the name from system.id; an explicit sink.stream still wins. If the stream already exists, the collector uses it without changing it, so a stream you created keeps your limits.

To replicate the stream, run a clustered site NATS server and set sink.replicas (and sink.storage for the storage class):

sink:
  replicas: 3
  storage: file

To create the stream yourself instead, run this against the site NATS server:

nats stream add scrape_west \
  --subjects '$INS.sys.west.scrape.>' \
  --storage file \
  --retention limits \
  --discard old \
  --replicas 1 \
  --defaults

The stream must carry exactly one subject, $INS.sys.<id>.scrape.>. The collector publishes every epoch under that namespace and doesn't inspect a stream it finds, so a stream that captures anything else stores none of its scrapes.

Central Node Config

On the central node, a collector site is a systems: entry with no scrape: block. With no scrape connection, the central node knows another process fills the stream, so it creates a mirror and only stores and checks the data:

# central.yaml, run with: insights -c central.yaml
node:
  id: central

data-dir: /var/lib/insights

# The INSIGHTS account on the central NATS server.
nats:
  server: nats://central:4222
  user: insights
  password: insights-password

# The mirror lives on the central NATS server, not in an embedded one.
sink:
  embed: false

db:
  memory-limit: 4GiB # per system

# A trial build also validates a license here, since this node indexes:
# license:
#   file: /etc/insights/license.jwt

systems:
  - id: west
    retention: 6h

The id must match the collector's system.id. It is the only value the two configs share: the central node's mirror_west, the collector's scrape_west, and the $INS.sys.west.* subjects all follow from it. The two nodes need different node.id values; see Node Ids.

A systems: list can mix entries: one with a scrape: block that the central node scrapes directly, another without one that a collector fills. See the systems configuration reference for every per-system setting.

How the Mirror Is Created

The central node names its mirror mirror_west and asks the collector for its stream details over $INS.sys.west.ops.sink.info: the stream name, subjects, JetStream domain, and scrape interval. It builds the mirror from that answer, reaching the site's stream through the site's domain, $JS.west.API. The central node has no other way to know the site's domain.

If the collector hasn't answered yet, the central node logs waiting for the scrape side and tries again every 15 seconds. It doesn't guess, because JetStream can't change a mirror after it is created, and a mirror whose API prefix reaches nothing reports lag: 0 and never sets mirror.error. Each system waits separately, so one unreachable collector doesn't hold up the others.

If you reach the site's JetStream through a subject mapping or an account import instead of a domain, the collector can't know the prefix, because the mapping rewrites the subject before it reaches the collector's side. Set it on the central node:

systems:
  - id: west
    mirror-api-prefix: JS.west.API

To set the mirror's name, replicas, or storage, add a stream: block. Any setting you leave out comes from the top-level sink: section:

systems:
  - id: west
    stream:
      name: mirror_west
      replicas: 3 # requires a clustered central NATS server
      storage: file

If a stream with that name already exists, the central node uses it without changing it. stream.name can't be the collector's stream name (scrape_west): the central node would be sharing the collector's stream instead of mirroring it, so it logs that and retries without creating anything.

Retention

retention limits how much data the stream and the mirror keep. The central node's database keeps history for db.retention.duration (32 days by default). Insights removes whole epochs, oldest first, so the indexer never reads half an epoch. The streams Insights creates have no max_age, so nothing else bounds them.

The collector trims its stream. The central node trims the mirror, using the scrape interval the collector reports: retention divided by the interval is the number of epochs to keep. A central node that can't purge, or whose collector never answers, logs it and the mirror keeps growing.

After an outage, the site's stream may hold far more than retention, and the mirror copies it faster than the indexer reads it. The central node doesn't trim the mirror until it has caught up with the site's stream, so the indexer doesn't lose epochs it hasn't read yet.

A stream you created with --max-age keeps it. That works, but the oldest epoch can then be cut in half instead of removed whole.

Check the Mirror

Look at the site's stream from the central site:

nats stream info scrape_west --js-domain west

Look at the mirror on the central NATS server. The mirror section shows the source and how far behind it is:

nats stream info mirror_west
nats stream report

A working mirror catches up to the site's stream and then keeps pace with it. Confirm that new epochs are stored:

insights db query --system west "SELECT max(epoch) FROM hx.servers"

The result should advance with each scrape. insights node list shows both nodes: the collector serves west (scrape) and the central node serves west; see Selecting a system or node. The central node's web UI serves the system under /systems/west/.

If the mirror stays empty or doesn't catch up, check:

• The domain. The domain in the site NATS server config must match the one the mirror uses, for example west and not West.
• Permissions. The central site's INSIGHTS account must be able to reach $JS.west.API.> across the leafnode and receive the replies. See Mirror Permissions.
• The leafnode. Nothing is mirrored while the leafnode is down. Check nats server report connections or the server logs on both sides.

Fix a Wrong Mirror

JetStream can't change a mirror's source after it is created, and a mirror pointed at the wrong place doesn't report an error. It just stays empty. On startup the central node compares an existing mirror with what the collector reports and logs any difference:

level=WARN msg="mirror does not match origin" system=west detail="origin name: mirror has \"scrape_edge\", origin reports \"scrape_west\""

It compares the origin stream name, the subject prefix, and the external API prefix (only when it has the $JS.<domain>.API form a domain produces), and whether the origin exists at all.

To fix it, delete the mirror and restart the central node, which creates a new one:

nats stream rm mirror_west

The new mirror copies whatever the site's stream still holds, and the central node's database keeps the older epochs it already stored.

To create the mirror yourself instead, use a JSON stream config, because the --mirror flag can't point at another domain:

{
  "name": "mirror_west",
  "storage": "file",
  "retention": "limits",
  "discard": "old",
  "num_replicas": 1,
  "mirror": {
    "name": "scrape_west",
    "filter_subject": "$INS.sys.west.scrape.>",
    "external": {
      "api": "$JS.west.API"
    }
  }
}

nats stream add mirror_west --config mirror.json

Then set stream: { name: mirror_west } on the system entry. Keep these rules:

• Set filter_subject to $INS.sys.<id>.scrape.>. A mirror has no subjects of its own, so the central node reads the subject namespace from filter_subject alone. Without it, the central node logs mirror has no filter subject on every retry and doesn't store the system.
• Don't add subject_transforms. The stored subjects must stay as the collector published them.
• Give the mirror a different name from the site's stream. $JS.FC.<stream>.> flow control subjects carry no domain, so streams with the same name in two domains of one account can't be mirrored reliably.
• If you set max_age, the JSON form takes nanoseconds (21600000000000 is 6h). Insights sets no max_age on the streams it creates; see Retention.

Mirror Permissions

The central NATS server reads the site's stream on the mirror's behalf. This is a server-to-server path, separate from the central node's own client permissions: $JS.API.> is the central domain, which the central node drives, and $JS.west.API.> is the site's, which the central server drives for the mirror. With the INSIGHTS account bridged and JetStream enabled on both sides, as above, this works without extra configuration. The examples use this arrangement.

If you restrict the leafnode or split the sites into separate accounts, these subjects must pass in the right direction:

• $JS.west.API.CONSUMER.CREATE.scrape_west and $JS.west.API.CONSUMER.CREATE.scrape_west.>, which create the mirror's consumer.
• $JS.FC.>, flow control replies.
• $JSC.R.>, consumer create replies.
• $JS.M.>, the mirrored messages and heartbeats.

If one is missing, the mirror exists but stops copying. The direction of each grant depends on which side accepts the leafnode and how accounts are imported, so treat this as a list of subjects rather than a ready-made permission block, and check a restricted setup with Check the Mirror. See the NATS JetStream wire API reference.

If the site and central sides use different accounts, mirroring becomes a cross-account JetStream setup: the site account exports its JetStream API ($JS.west.API.>), the central account imports it, and the delivery subjects above must be shared the same way. Start from the shared account arrangement unless account isolation across the leafnode is a hard requirement.

If you narrow the central node's own $JS.API.> permissions, keep these:

• $JS.API.STREAM.CREATE.mirror_west, unless you create the mirror yourself.
• $JS.API.STREAM.PURGE.mirror_west. Without it, the mirror is never trimmed and grows without limit, and the central node logs mirror prune failed.
• $JS.API.INFO, so the central node can read its own domain when it checks the mirror. Without it, the central node skips the checks that depend on the domain.

The request to each collector, $INS.sys.<id>.ops.sink.info, is already covered by $INS.>.