# Federated deployment

Monitor NATS systems at several sites and see all of them in one web UI.

In a federated deployment, each site runs Insights next to its own NATS system, and a central site serves the web UI. Each site connects to the central site's NATS server over a leafnode that carries only the account Insights uses. The monitored system's system account never leaves its site, and the central site never connects to the monitored systems.

![A central site connected over leafnodes to a site running an Insights node and a site running a collector](data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCA3NjAgNDkwIiB3aWR0aD0iNzYwIiBoZWlnaHQ9IjQ5MCIgcm9sZT0iaW1nIiBhcmlhLWxhYmVsbGVkYnk9InRpdGxlIGRlc2MiPgogIDx0aXRsZSBpZD0idGl0bGUiPkZlZGVyYXRlZCBJbnNpZ2h0cyBkZXBsb3ltZW50PC90aXRsZT4KICA8ZGVzYyBpZD0iZGVzYyI+QSBjZW50cmFsIHNpdGUgcnVucyBhIE5BVFMgc2VydmVyIGFuZCBJbnNpZ2h0cy4gVHdvIHNpdGVzIGNvbm5lY3QgdG8gaXQgb3ZlciBsZWFmbm9kZXMgdGhhdCBjYXJyeSBvbmx5IHRoZSBJbnNpZ2h0cyBhY2NvdW50LiBTaXRlIGVhc3QgcnVucyBhIGZ1bGwgSW5zaWdodHMgbm9kZSB0aGF0IHNjcmFwZXMsIHN0b3JlcywgYW5kIGNoZWNrcyBpdHMgb3duIHN5c3RlbS4gU2l0ZSB3ZXN0IHJ1bnMgYSBjb2xsZWN0b3IgdGhhdCBvbmx5IHNjcmFwZXMgYW5kIHB1Ymxpc2hlcyB0byBhIHN0cmVhbSwgd2hpY2ggdGhlIGNlbnRyYWwgc2l0ZSBtaXJyb3JzIGFuZCBzdG9yZXMuIEVhY2ggc2l0ZSdzIHN5c3RlbSBhY2NvdW50IHN0YXlzIGF0IHRoZSBzaXRlLjwvZGVzYz4KICA8ZGVmcz4KICAgIDxtYXJrZXIgaWQ9ImFycm93IiB2aWV3Qm94PSIwIDAgMTAgMTAiIHJlZlg9IjkiIHJlZlk9IjUiIG1hcmtlcldpZHRoPSI3IiBtYXJrZXJIZWlnaHQ9IjciIG9yaWVudD0iYXV0by1zdGFydC1yZXZlcnNlIj4KICAgICAgPHBhdGggZD0iTTAsMCBMMTAsNSBMMCwxMCB6IiBmaWxsPSIjNDc1NTY5Ii8+CiAgICA8L21hcmtlcj4KICAgIDxzdHlsZT4KICAgICAgdGV4dCB7IGZvbnQtZmFtaWx5OiB1aS1zYW5zLXNlcmlmLCBzeXN0ZW0tdWksIC1hcHBsZS1zeXN0ZW0sICJTZWdvZSBVSSIsIFJvYm90bywgc2Fucy1zZXJpZjsgZmlsbDogIzBmMTcyYTsgfQogICAgICAuc2l0ZSB7IGZvbnQtc2l6ZTogMTZweDsgZm9udC13ZWlnaHQ6IDYwMDsgfQogICAgICAubmFtZSB7IGZvbnQtc2l6ZTogMTVweDsgZm9udC13ZWlnaHQ6IDYwMDsgdGV4dC1hbmNob3I6IG1pZGRsZTsgfQogICAgICAuc3ViIHsgZm9udC1zaXplOiAxM3B4OyBmaWxsOiAjNDc1NTY5OyB0ZXh0LWFuY2hvcjogbWlkZGxlOyB9CiAgICAgIC5lZGdlIHsgZm9udC1zaXplOiAxM3B4OyBmaWxsOiAjMzM0MTU1OyB9CiAgICAgIC5ub3RlIHsgZm9udC1zaXplOiAxMi41cHg7IGZpbGw6ICM2NDc0OGI7IGZvbnQtc3R5bGU6IGl0YWxpYzsgfQogICAgICAuZmxvdyB7IHN0cm9rZTogIzQ3NTU2OTsgc3Ryb2tlLXdpZHRoOiAxLjU7IGZpbGw6IG5vbmU7IH0KICAgICAgLmxlYWYgeyBzdHJva2U6ICMwZTc0OTA7IHN0cm9rZS13aWR0aDogMjsgc3Ryb2tlLWRhc2hhcnJheTogNiA1OyBmaWxsOiBub25lOyB9CiAgICA8L3N0eWxlPgogIDwvZGVmcz4KCiAgPHJlY3QgeD0iMC41IiB5PSIwLjUiIHdpZHRoPSI3NTkiIGhlaWdodD0iNDg5IiByeD0iMTIiIGZpbGw9IiNmZmZmZmYiIHN0cm9rZT0iI2UyZThmMCIvPgoKICA8IS0tIENlbnRyYWwgc2l0ZSAtLT4KICA8cmVjdCB4PSIyMCIgeT0iMjAiIHdpZHRoPSI3MjAiIGhlaWdodD0iMTQwIiByeD0iMTAiIGZpbGw9IiNmOGZhZmMiIHN0cm9rZT0iI2NiZDVlMSIvPgogIDx0ZXh0IGNsYXNzPSJzaXRlIiB4PSIzNiIgeT0iNDQiPkNlbnRyYWwgc2l0ZTwvdGV4dD4KICA8cmVjdCB4PSI2MCIgeT0iNjAiIHdpZHRoPSIyODAiIGhlaWdodD0iODAiIHJ4PSI4IiBmaWxsPSIjZWNmZWZmIiBzdHJva2U9IiMwODkxYjIiLz4KICA8dGV4dCBjbGFzcz0ibmFtZSIgeD0iMjAwIiB5PSI4OCI+TkFUUyBzZXJ2ZXI8L3RleHQ+CiAgPHRleHQgY2xhc3M9InN1YiIgeD0iMjAwIiB5PSIxMDYiPkluc2lnaHRzIGFjY291bnQgwrcgZG9tYWluOiBjZW50cmFsPC90ZXh0PgogIDx0ZXh0IGNsYXNzPSJzdWIiIHg9IjIwMCIgeT0iMTI0Ij5taXJyb3Jfd2VzdCBzdHJlYW08L3RleHQ+CiAgPHJlY3QgeD0iNDIwIiB5PSI2MCIgd2lkdGg9IjI4MCIgaGVpZ2h0PSI4MCIgcng9IjgiIGZpbGw9IiNlZWYyZmYiIHN0cm9rZT0iIzRmNDZlNSIvPgogIDx0ZXh0IGNsYXNzPSJuYW1lIiB4PSI1NjAiIHk9Ijg4Ij5JbnNpZ2h0czwvdGV4dD4KICA8dGV4dCBjbGFzcz0ic3ViIiB4PSI1NjAiIHk9IjEwNiI+V2ViIFVJIGZvciBldmVyeSBzaXRlPC90ZXh0PgogIDx0ZXh0IGNsYXNzPSJzdWIiIHg9IjU2MCIgeT0iMTI0Ij5TdG9yZXMgYW5kIGNoZWNrcyBjb2xsZWN0b3Igc2l0ZXM8L3RleHQ+CiAgPHBhdGggY2xhc3M9ImZsb3ciIGQ9Ik0zNDQsMTAwIEw0MTYsMTAwIiBtYXJrZXItc3RhcnQ9InVybCgjYXJyb3cpIiBtYXJrZXItZW5kPSJ1cmwoI2Fycm93KSIvPgoKICA8IS0tIExlYWZub2RlcyAtLT4KICA8cGF0aCBjbGFzcz0ibGVhZiIgZD0iTTE5NSwyNjQgTDE5NSwxNDQiLz4KICA8dGV4dCBjbGFzcz0iZWRnZSIgeD0iMjA1IiB5PSIyMDAiPmxlYWZub2RlPC90ZXh0PgogIDx0ZXh0IGNsYXNzPSJlZGdlIiB4PSIyMDUiIHk9IjIxNSI+SW5zaWdodHMgYWNjb3VudCBvbmx5PC90ZXh0PgogIDxwYXRoIGNsYXNzPSJsZWFmIiBkPSJNNTY1LDI2NCBMMzAwLDE0NCIvPgogIDx0ZXh0IGNsYXNzPSJlZGdlIiB4PSI0NzAiIHk9IjE5MiI+bGVhZm5vZGU8L3RleHQ+CiAgPHRleHQgY2xhc3M9ImVkZ2UiIHg9IjQ3MCIgeT0iMjA3Ij5zdHJlYW0gbWlycm9yZWQgdG8gY2VudHJhbDwvdGV4dD4KCiAgPCEtLSBTaXRlIGVhc3Q6IGZ1bGwgSW5zaWdodHMgbm9kZSAtLT4KICA8cmVjdCB4PSIyMCIgeT0iMjQwIiB3aWR0aD0iMzUwIiBoZWlnaHQ9IjIzMCIgcng9IjEwIiBmaWxsPSIjZjhmYWZjIiBzdHJva2U9IiNjYmQ1ZTEiLz4KICA8cmVjdCB4PSI0MCIgeT0iMjY4IiB3aWR0aD0iMzEwIiBoZWlnaHQ9IjU0IiByeD0iOCIgZmlsbD0iI2VjZmVmZiIgc3Ryb2tlPSIjMDg5MWIyIi8+CiAgPHRleHQgY2xhc3M9Im5hbWUiIHg9IjE5NSIgeT0iMjkwIj5TaXRlIE5BVFMgc2VydmVyPC90ZXh0PgogIDx0ZXh0IGNsYXNzPSJzdWIiIHg9IjE5NSIgeT0iMzA4Ij5JbnNpZ2h0cyBhY2NvdW50IMK3IGRvbWFpbjogZWFzdDwvdGV4dD4KICA8cmVjdCB4PSI0MCIgeT0iMzY2IiB3aWR0aD0iMTMwIiBoZWlnaHQ9IjY0IiByeD0iOCIgZmlsbD0iI2ZmZmZmZiIgc3Ryb2tlPSIjOTRhM2I4Ii8+CiAgPHRleHQgY2xhc3M9Im5hbWUiIHg9IjEwNSIgeT0iMzkyIj5Nb25pdG9yZWQgTkFUUzwvdGV4dD4KICA8dGV4dCBjbGFzcz0ic3ViIiB4PSIxMDUiIHk9IjQxMCI+c3lzdGVtIGFjY291bnQ8L3RleHQ+CiAgPHJlY3QgeD0iMjEwIiB5PSIzNjYiIHdpZHRoPSIxNDAiIGhlaWdodD0iNjQiIHJ4PSI4IiBmaWxsPSIjZWVmMmZmIiBzdHJva2U9IiM0ZjQ2ZTUiLz4KICA8dGV4dCBjbGFzcz0ibmFtZSIgeD0iMjgwIiB5PSIzOTIiPkluc2lnaHRzIG5vZGU8L3RleHQ+CiAgPHRleHQgY2xhc3M9InN1YiIgeD0iMjgwIiB5PSI0MTAiPnNjcmFwZSwgc3RvcmUsIGNoZWNrPC90ZXh0PgogIDxwYXRoIGNsYXNzPSJmbG93IiBkPSJNMTc0LDM5OCBMMjA2LDM5OCIgbWFya2VyLWVuZD0idXJsKCNhcnJvdykiLz4KICA8cGF0aCBjbGFzcz0iZmxvdyIgZD0iTTI4MCwzNjIgTDI4MCwzMjYiIG1hcmtlci1lbmQ9InVybCgjYXJyb3cpIi8+CiAgPHRleHQgY2xhc3M9ImVkZ2UiIHg9IjI4OCIgeT0iMzQ4Ij5BUEk8L3RleHQ+CiAgPHRleHQgY2xhc3M9InNpdGUiIHg9IjM2IiB5PSIyNTgiIGR5PSIwIiBzdHlsZT0iZm9udC1zaXplOjEzLjVweDtmaWxsOiMzMzQxNTUiPlNpdGUgZWFzdDogSW5zaWdodHMgbm9kZTwvdGV4dD4KICA8dGV4dCBjbGFzcz0ibm90ZSIgeD0iMzYiIHk9IjQ1NiI+U3lzdGVtIGFjY291bnQgc3RheXMgYXQgdGhlIHNpdGU8L3RleHQ+CgogIDwhLS0gU2l0ZSB3ZXN0OiBjb2xsZWN0b3IgLS0+CiAgPHJlY3QgeD0iMzkwIiB5PSIyNDAiIHdpZHRoPSIzNTAiIGhlaWdodD0iMjMwIiByeD0iMTAiIGZpbGw9IiNmOGZhZmMiIHN0cm9rZT0iI2NiZDVlMSIvPgogIDxyZWN0IHg9IjQxMCIgeT0iMjY4IiB3aWR0aD0iMzEwIiBoZWlnaHQ9IjU0IiByeD0iOCIgZmlsbD0iI2VjZmVmZiIgc3Ryb2tlPSIjMDg5MWIyIi8+CiAgPHRleHQgY2xhc3M9Im5hbWUiIHg9IjU2NSIgeT0iMjkwIj5TaXRlIE5BVFMgc2VydmVyPC90ZXh0PgogIDx0ZXh0IGNsYXNzPSJzdWIiIHg9IjU2NSIgeT0iMzA4Ij5zY3JhcGVfd2VzdCBzdHJlYW0gwrcgZG9tYWluOiB3ZXN0PC90ZXh0PgogIDxyZWN0IHg9IjQxMCIgeT0iMzY2IiB3aWR0aD0iMTMwIiBoZWlnaHQ9IjY0IiByeD0iOCIgZmlsbD0iI2ZmZmZmZiIgc3Ryb2tlPSIjOTRhM2I4Ii8+CiAgPHRleHQgY2xhc3M9Im5hbWUiIHg9IjQ3NSIgeT0iMzkyIj5Nb25pdG9yZWQgTkFUUzwvdGV4dD4KICA8dGV4dCBjbGFzcz0ic3ViIiB4PSI0NzUiIHk9IjQxMCI+c3lzdGVtIGFjY291bnQ8L3RleHQ+CiAgPHJlY3QgeD0iNTgwIiB5PSIzNjYiIHdpZHRoPSIxNDAiIGhlaWdodD0iNjQiIHJ4PSI4IiBmaWxsPSIjZmVmY2U4IiBzdHJva2U9IiNjYThhMDQiLz4KICA8dGV4dCBjbGFzcz0ibmFtZSIgeD0iNjUwIiB5PSIzOTIiPkNvbGxlY3RvcjwvdGV4dD4KICA8dGV4dCBjbGFzcz0ic3ViIiB4PSI2NTAiIHk9IjQxMCI+c2NyYXBlIG9ubHk8L3RleHQ+CiAgPHBhdGggY2xhc3M9ImZsb3ciIGQ9Ik01NDQsMzk4IEw1NzYsMzk4IiBtYXJrZXItZW5kPSJ1cmwoI2Fycm93KSIvPgogIDxwYXRoIGNsYXNzPSJmbG93IiBkPSJNNjUwLDM2MiBMNjUwLDMyNiIgbWFya2VyLWVuZD0idXJsKCNhcnJvdykiLz4KICA8dGV4dCBjbGFzcz0iZWRnZSIgeD0iNjU4IiB5PSIzNDgiPnB1Ymxpc2g8L3RleHQ+CiAgPHRleHQgY2xhc3M9InNpdGUiIHg9IjQwNiIgeT0iMjU4IiBzdHlsZT0iZm9udC1zaXplOjEzLjVweDtmaWxsOiMzMzQxNTUiPlNpdGUgd2VzdDogY29sbGVjdG9yPC90ZXh0PgogIDx0ZXh0IGNsYXNzPSJub3RlIiB4PSI0MDYiIHk9IjQ1NiI+U3lzdGVtIGFjY291bnQgc3RheXMgYXQgdGhlIHNpdGU8L3RleHQ+Cjwvc3ZnPgo=)

## Choose what runs at each site

Each site runs one of two things. The network setup is the same for both. What differs is where the data is stored and checked.

|                          | Insights node (recommended)          | Collector                                                |
| ------------------------ | ------------------------------------ | -------------------------------------------------------- |
| Runs at the site         | `insights`, the full binary          | `insights-collector`, a smaller binary that only scrapes |
| Database and checks      | At the site                          | At the central site                                      |
| Data kept at the site    | History for `db.retention.duration`  | A short buffer until the central site copies it          |
| Runs at the central site | `insights web`, or any Insights node | An Insights node that stores and checks the site's data  |

Run a full **Insights node** at a site unless you have a reason not to. Each site keeps its own data and runs its own checks, and nothing is copied between sites.

Run a **collector** when a site can't spare the memory and disk for a database, or when monitoring data should not be kept at the site. The central site then copies the site's scrape stream with a JetStream mirror and does the storage and checks there. The collector has no DuckDB, no web UI, no Prometheus endpoint, and no embedded NATS server. It scrapes, publishes to a stream, and serves the part of the `$INS` API that needs no database.

You can mix both in one deployment.

If a central node can reach a monitored system directly, it doesn't need either: list the system under `systems:` with a `scrape:` block and the node scrapes it itself. See [Multiple Systems on One Node](/insights/1.2/guides/deployment.md#multiple-systems-on-one-node).

## Connect the sites

Every site needs the following, whichever option it runs:

* A NATS server at the site with JetStream enabled and its own JetStream domain.
* A NATS server at the central site with JetStream enabled and a different domain.
* A leafnode from the site to the central site that carries the account Insights uses. This guide calls that account `INSIGHTS`. The site's system account is not bridged.
* A unique `system.id` for each site's system, and a unique `node.id` for each Insights process. This guide uses `east` for a site with an Insights node and `west` for a site with a collector.
* Network access from each site to the central site on the leafnode port (`7422` by convention). The central site never connects to a site.

The `nats` CLI commands below were written against v0.4.x.

The system id names everything Insights creates for that system: the `$INS.sys.<id>.*` API, the `scrape_<id>` stream, and its subjects, `$INS.sys.<id>.scrape.>`. Because every subject includes the id, sites can share the `INSIGHTS` account without their subjects colliding. The subject namespace is never configured.

This is not access control: any user in the account can query every site. If sites must not see each other's data, give each Insights user permissions scoped to its own system (see [Insights Permissions](#insights-permissions)) Separate accounts per site would also stop the central web UI from finding the sites, because it discovers and queries them within one account.

### Site NATS server

A domain is how JetStream tells two JetStreams apart over one leafnode: a stream in one domain is reached from the other through the domain's API prefix, `$JS.<domain>.API`. Give each site's domain the same name as its system id, so `$JS.west.API` clearly refers to site `west`. Nothing requires it, but with several sites the alternative is remembering which domain is which.

Bridge only the `INSIGHTS` account:

```
# site-west.conf

server_name: west-1

listen: 0.0.0.0:4222



jetstream {

  domain: west

  store_dir: /data/nats/jetstream

}



leafnodes {

  remotes = [

    {

      url: "nats-leaf://insights:insights-password@central.example.com:7422"

      account: INSIGHTS

    }

  ]

}



accounts {

  INSIGHTS: {

    jetstream: enabled

    users: [ { user: insights, password: insights-password } ]

  }

  SYS: {

    users: [ { user: sys, password: sys-password } ]

  }

}

system_account: SYS
```

Two accounts matter here:

| Account    | Holds                                                         | Reachable from the central site |
| ---------- | ------------------------------------------------------------- | ------------------------------- |
| `SYS`      | The monitored system's system account, which Insights scrapes | No                              |
| `INSIGHTS` | The scrape stream and the `$INS.sys.<id>.*` API               | Yes                             |

Insights at the site holds a connection to each: it reads from `SYS` locally and writes to `INSIGHTS`. Only `INSIGHTS` crosses the leafnode.

The examples on this page use passwords to stay short. In production, use your usual NATS authentication, such as JWT credentials, and point Insights at it with `creds` or `context`.

Sharing an account is not sharing a JetStream. The domains separate the two JetStreams and the account separates the traffic. A leaf server with no `jetstream` block does not gain one by being bridged into a JetStream-enabled account.

### Central NATS server

The central NATS server runs JetStream under its own domain, accepts leafnode connections, and defines the same `INSIGHTS` account. A leafnode joins the account of the user it logs in as, so sites that log in as `insights` join only that account.

```
# central.conf

server_name: central-1

listen: 0.0.0.0:4222



jetstream {

  domain: central

  store_dir: /data/nats/jetstream

}



leafnodes {

  port: 7422

}



accounts {

  INSIGHTS: {

    jetstream: enabled

    users: [ { user: insights, password: insights-password } ]

  }

}
```

### Insights permissions

If you restrict the users Insights connects with, this is a working baseline for a site node, a collector, or a central node. It is the same connection shape as any external NATS server, described in [Sink connection permissions](/insights/1.2/guides/deployment.md#sink-connection-permissions):

```
permissions {

  publish {

    allow = [

      "$INS.>",     # scrape data, discovery, and the Insights API

      "$JS.API.>",  # stream management

      "$JS.FC.>",   # flow control replies

      "_INBOX.>"    # request replies

    ]

  }

  subscribe {

    allow = [

      "_INBOX.>",   # request replies

      "$INS.>",     # discovery and the Insights API

      "$SRV.>"      # NATS service discovery

    ]

  }

}
```

Everything Insights owns is under `$INS`, including the scrape stream, so the grant doesn't have to list it. To limit a connection to one system, replace `$INS.>` with `$INS.sys.<id>.>`, `$INS.node.<node_id>.>`, and `$INS.ping`.

### Node ids

Each Insights process checks its identity over `$INS.ping` at startup, and that request crosses the leafnode. A node's `node.id` defaults to its system id when it serves one system, so two processes serving the same system (a collector and a single-system central node, for example) clash unless at least one is named. The second to start stops with:

```
node.id "west" is already in use on this NATS
```

Sharing a system id is fine when the roles differ: a collector holds the system's `scrape` role and the central node its `catalog` role.

## Option 1: Insights node at each site

The site node is an ordinary single-system Insights node. It scrapes the site's system, stores the data, and runs checks. What makes it part of a federation is its connection to the site's `INSIGHTS` account, and turning off the embedded NATS server so its stream lives on the site NATS server. There is no federation-specific config:

```
# east.yaml, run with: insights -c east.yaml

node:

  id: east



system:

  id: east



data-dir: /var/lib/insights



# The INSIGHTS account on the site NATS server. The central site reaches this

# node's API through the leafnode.

nats:

  server: nats://site-east:4222

  user: insights

  password: insights-password



# The site's system account, which is what Insights monitors.

sys:

  server: nats://site-east:4222

  user: sys

  password: sys-password



sink:

  embed: false
```

A [trial build](/insights/1.2/trial.md#configuring-the-license) also needs a `license:` section on every site node.

The site node can also serve its own web UI. Leave `web` enabled if people at the site should use it directly.

At the central site, run a web UI connected to the central NATS server's `INSIGHTS` account. It needs no database and no license:

```
insights web \

  --server nats://central:4222 \

  --user insights --password insights-password \

  --web.hostname 0.0.0.0
```

The web UI finds every system that answers on the shared account over `$INS.ping` and lists them on its Systems page. It looks for systems when it starts, so restart it after you add a site. A full Insights node on the central NATS server does the same, so if the central site also monitors its own system, that node's web UI can show every site.

Check that every site is reachable from the central site:

```
insights system list --server nats://central:4222 --user insights --password insights-password

insights db query --server nats://central:4222 --user insights --password insights-password \

  --system east "SELECT count(*) FROM hx.servers"
```

## Option 2: Collector at a site

A collector scrapes the site's system and publishes the data to a stream on the site NATS server. A central Insights node copies that stream to the central NATS server with a JetStream mirror, then stores the data and runs checks.

The collector is the `insights-collector` release archive, the `-collector` image tag (`registry.synadia.io/insights:1.2.2-collector`), or `task build:collector` from source.

### Collector config

The collector monitors one system, so it uses the single-system config with no `systems:` list:

```
# collector.yaml, run with: insights-collector -c collector.yaml

node:

  # Must differ from the central node's node.id.

  id: west-collector



system:

  # Must match the system id configured on the central node.

  id: west



# The INSIGHTS account on the site NATS server, where the scrape stream lives.

nats:

  server: nats://site-west:4222

  user: insights

  password: insights-password



# The site's system account, which is what the collector scrapes.

sys:

  server: nats://site-west:4222

  user: sys

  password: sys-password



scraper:

  interval: 1m



sink:

  # How much data the stream keeps.

  retention: 6h



# A trial build also validates a license:

# license:

#   file: /etc/insights/license.jwt
```

Run it with `insights-collector -c collector.yaml`. `serve` is the collector's default command, so `insights-collector serve -c collector.yaml` does the same thing. From the collector image, with `collector.yaml` and the files it names mounted at `/etc/insights`:

```
docker run -v /path/to/collector:/etc/insights:ro \

  registry.synadia.io/insights:1.2.2-collector \

  -c /etc/insights/collector.yaml
```

The collector reads only the `node`, `system`, `nats`, `sys`, `scraper`, `sink`, `license`, `updater`, `telemetry`, and `data-dir` settings, plus `log-level`. It has no database or embedded stream store, so it doesn't need a data directory. If no NATS server is configured, it stops at startup:

```
collector requires an external NATS sink: set --server (or --context, or --scraper.nats.server) to the local leafnode that stores the scrape stream and serves the $INS API
```

See the collector build reference for what the binary includes, and `insights-collector serve` for its flags.

### The site stream

On startup, the collector creates the `scrape_west` stream on the site NATS server with the subject `$INS.sys.west.scrape.>`. It derives the name from `system.id`; an explicit `sink.stream` still wins. If the stream already exists, the collector uses it without changing it, so a stream you created keeps your limits.

To replicate the stream, run a clustered site NATS server and set `sink.replicas` (and `sink.storage` for the storage class):

```
sink:

  replicas: 3

  storage: file
```

To create the stream yourself instead, run this against the site NATS server:

```
nats stream add scrape_west \

  --subjects '$INS.sys.west.scrape.>' \

  --storage file \

  --retention limits \

  --discard old \

  --replicas 1 \

  --defaults
```

The stream must carry exactly one subject, `$INS.sys.<id>.scrape.>`. The collector publishes every epoch under that namespace and doesn't inspect a stream it finds, so a stream that captures anything else stores none of its scrapes.

### Central node config

On the central node, a collector site is a `systems:` entry with no `scrape:` block. With no scrape connection, the central node knows another process fills the stream, so it creates a mirror and only stores and checks the data:

```
# central.yaml, run with: insights -c central.yaml

node:

  id: central



data-dir: /var/lib/insights



# The INSIGHTS account on the central NATS server.

nats:

  server: nats://central:4222

  user: insights

  password: insights-password



# The mirror lives on the central NATS server, not in an embedded one.

sink:

  embed: false



db:

  memory-limit: 4GiB # per system



# A trial build also validates a license here, since this node indexes:

# license:

#   file: /etc/insights/license.jwt



systems:

  - id: west

    retention: 6h
```

The `id` must match the collector's `system.id`. It is the only value the two configs share: the central node's `mirror_west`, the collector's `scrape_west`, and the `$INS.sys.west.*` subjects all follow from it. The two nodes need different `node.id` values; see [Node Ids](#node-ids).

A `systems:` list can mix entries: one with a `scrape:` block that the central node scrapes directly, another without one that a collector fills. See the `systems` configuration reference for every per-system setting.

### How the mirror is created

The central node names its mirror `mirror_west` and asks the collector for its stream details over `$INS.sys.west.ops.sink.info`: the stream name, subjects, JetStream domain, and scrape interval. It builds the mirror from that answer, reaching the site's stream through the site's domain, `$JS.west.API`. The central node has no other way to know the site's domain.

If the collector hasn't answered yet, the central node logs `waiting for the scrape side` and tries again every 15 seconds. It doesn't guess, because JetStream can't change a mirror after it is created, and a mirror whose API prefix reaches nothing reports `lag: 0` and never sets `mirror.error`. Each system waits separately, so one unreachable collector doesn't hold up the others.

If you reach the site's JetStream through a subject mapping or an account import instead of a domain, the collector can't know the prefix, because the mapping rewrites the subject before it reaches the collector's side. Set it on the central node:

```
systems:

  - id: west

    mirror-api-prefix: JS.west.API
```

To set the mirror's name, replicas, or storage, add a `stream:` block. Any setting you leave out comes from the top-level `sink:` section:

```
systems:

  - id: west

    stream:

      name: mirror_west

      replicas: 3 # requires a clustered central NATS server

      storage: file
```

If a stream with that name already exists, the central node uses it without changing it. `stream.name` can't be the collector's stream name (`scrape_west`): the central node would be sharing the collector's stream instead of mirroring it, so it logs that and retries without creating anything.

### Retention

`retention` limits how much data the stream and the mirror keep. The central node's database keeps history for `db.retention.duration` (32 days by default). Insights removes whole epochs, oldest first, so the indexer never reads half an epoch. The streams Insights creates have no `max_age`, so nothing else bounds them.

The collector trims its stream. The central node trims the mirror, using the scrape interval the collector reports: retention divided by the interval is the number of epochs to keep. A central node that can't purge, or whose collector never answers, logs it and the mirror keeps growing.

After an outage, the site's stream may hold far more than `retention`, and the mirror copies it faster than the indexer reads it. The central node doesn't trim the mirror until it has caught up with the site's stream, so the indexer doesn't lose epochs it hasn't read yet.

A stream you created with `--max-age` keeps it. That works, but the oldest epoch can then be cut in half instead of removed whole.

### Check the mirror

Look at the site's stream from the central site:

```
nats stream info scrape_west --js-domain west
```

Look at the mirror on the central NATS server. The mirror section shows the source and how far behind it is:

```
nats stream info mirror_west

nats stream report
```

A working mirror catches up to the site's stream and then keeps pace with it. Confirm that new epochs are stored:

```
insights db query --system west "SELECT max(epoch) FROM hx.servers"
```

The result should advance with each scrape. `insights node list` shows both nodes: the collector serves `west (scrape)` and the central node serves `west`; see Selecting a system or node. The central node's web UI serves the system under `/systems/west/`.

If the mirror stays empty or doesn't catch up, check:

* **The domain.** The domain in the site NATS server config must match the one the mirror uses, for example `west` and not `West`.
* **Permissions.** The central site's `INSIGHTS` account must be able to reach `$JS.west.API.>` across the leafnode and receive the replies. See [Mirror Permissions](#mirror-permissions).
* **The leafnode.** Nothing is mirrored while the leafnode is down. Check `nats server report connections` or the server logs on both sides.

### Fix a wrong mirror

JetStream can't change a mirror's source after it is created, and a mirror pointed at the wrong place doesn't report an error. It just stays empty. On startup the central node compares an existing mirror with what the collector reports and logs any difference:

```
level=WARN msg="mirror does not match origin" system=west detail="origin name: mirror has \"scrape_edge\", origin reports \"scrape_west\""
```

It compares the origin stream name, the subject prefix, and the external API prefix (only when it has the `$JS.<domain>.API` form a domain produces), and whether the origin exists at all.

To fix it, delete the mirror and restart the central node, which creates a new one:

```
nats stream rm mirror_west
```

The new mirror copies whatever the site's stream still holds, and the central node's database keeps the older epochs it already stored.

To create the mirror yourself instead, use a JSON stream config, because the `--mirror` flag can't point at another domain:

```
{

  "name": "mirror_west",

  "storage": "file",

  "retention": "limits",

  "discard": "old",

  "num_replicas": 1,

  "mirror": {

    "name": "scrape_west",

    "filter_subject": "$INS.sys.west.scrape.>",

    "external": {

      "api": "$JS.west.API"

    }

  }

}
```

```
nats stream add mirror_west --config mirror.json
```

Then set `stream: { name: mirror_west }` on the system entry. Keep these rules:

* Set `filter_subject` to `$INS.sys.<id>.scrape.>`. A mirror has no subjects of its own, so the central node reads the subject namespace from `filter_subject` alone. Without it, the central node logs `mirror has no filter subject` on every retry and doesn't store the system.
* Don't add `subject_transforms`. The stored subjects must stay as the collector published them.
* Give the mirror a different name from the site's stream. `$JS.FC.<stream>.>` flow control subjects carry no domain, so streams with the same name in two domains of one account can't be mirrored reliably.
* If you set `max_age`, the JSON form takes nanoseconds (`21600000000000` is 6h). Insights sets no `max_age` on the streams it creates; see [Retention](#retention).

### Mirror permissions

The central NATS server reads the site's stream on the mirror's behalf. This is a server-to-server path, separate from the central node's own client permissions: `$JS.API.>` is the central domain, which the central node drives, and `$JS.west.API.>` is the site's, which the central server drives for the mirror. With the `INSIGHTS` account bridged and JetStream enabled on both sides, as above, this works without extra configuration. The examples use this arrangement.

If you restrict the leafnode or split the sites into separate accounts, these subjects must pass in the right direction:

* `$JS.west.API.CONSUMER.CREATE.scrape_west` and `$JS.west.API.CONSUMER.CREATE.scrape_west.>`, which create the mirror's consumer.
* `$JS.FC.>`, flow control replies.
* `$JSC.R.>`, consumer create replies.
* `$JS.M.>`, the mirrored messages and heartbeats.

If one is missing, the mirror exists but stops copying. The direction of each grant depends on which side accepts the leafnode and how accounts are imported, so treat this as a list of subjects rather than a ready-made permission block, and check a restricted setup with [Check the Mirror](#check-the-mirror). See the NATS [JetStream wire API reference](https://docs.nats.io/reference/reference-protocols/nats_api_reference#stream-source-and-mirror).

If the site and central sides use different accounts, mirroring becomes a cross-account JetStream setup: the site account exports its JetStream API (`$JS.west.API.>`), the central account imports it, and the delivery subjects above must be shared the same way. Start from the shared account arrangement unless account isolation across the leafnode is a hard requirement.

If you narrow the central node's own `$JS.API.>` permissions, keep these:

* `$JS.API.STREAM.CREATE.mirror_west`, unless you create the mirror yourself.
* `$JS.API.STREAM.PURGE.mirror_west`. Without it, the mirror is never trimmed and grows without limit, and the central node logs `mirror prune failed`.
* `$JS.API.INFO`, so the central node can read its own domain when it checks the mirror. Without it, the central node skips the checks that depend on the domain.

The request to each collector, `$INS.sys.<id>.ops.sink.info`, is already covered by `$INS.>`.
