# Deployment

Insights is a single Go binary with independently toggleable subsystems. How you configure them determines the deployment topology.

## Subsystems

| Subsystem       | Flag           | Default | Description                                                                         |
| --------------- | -------------- | ------- | ----------------------------------------------------------------------------------- |
| Scraper         | `--scraper`    | `true`  | Scrapes `$SYS` endpoints from a target NATS system                                  |
| Indexer         | `--indexer`    | `true`  | Processes scrape stream into DuckDB                                                 |
| Web UI          | `--web`        | `true`  | HTTP server for the dashboard                                                       |
| Embedded Sink   | `--sink.embed` | `true`  | In-process NATS server with JetStream                                               |
| Simulator       | `--simulator`  | `false` | In-process NATS clusters with traffic workloads                                     |
| Prometheus      | `--prometheus` | `true`  | Prometheus-compatible `/metrics` endpoint                                           |
| Version checker | `--updater`    | `true`  | Periodic check for a newer release (notification only)                              |
| Telemetry       | `--telemetry`  | `true`  | Product usage pulse, in [trial builds](/insights/1.2/trial.md) only (see Telemetry) |

The API service (NATS micro) starts on any node that indexes or scrapes a system. A web-only node is a client of it and serves no API of its own.

Each toggle is a shorthand: `--web=false` on the command line, `INSIGHTS_WEB=false` in the environment. Both spell the same thing as the verbose `--web.enabled=false` / `INSIGHTS_WEB_ENABLED=false`, which still parse but are hidden from `--help`. In YAML, use a `web:` section with `enabled:`; see the Configuration reference.

## Deployment topologies

The following diagrams show which subsystems run in each topology.

### Demo (simulator)

Zero-config mode for evaluation. Starts an embedded NATS simulator with traffic workloads. All data is ephemeral.

```
./insights --simulator
```

Everything runs in-process: simulator, embedded sink, scraper, indexer, web UI. No external NATS required.

Choose a profile to control deployment size:

```
./insights --simulator --simulator.profile super-medium
```

Available profiles: `core-{small,medium,large}`, `js-{small,medium,large}`, `super-{small,medium,large}`, `leaf-{small,medium,large}`, `super-leaf-{small,medium,large}`.

### All-in-one (ephemeral)

Monitor a real NATS deployment. Requires system account credentials for `$SYS` access.

```
./insights \

  --sys.server nats://target:4222 \

  --sys.creds /path/to/sys.creds
```

Or with a NATS context:

```
./insights --sys.context my-system
```

The database and the JetStream store are written to a temporary directory, and the data is lost on restart.

### All-in-one (persistent)

Same as above with durable storage that survives restarts.

```
./insights \

  --sys.server nats://target:4222 \

  --sys.creds /path/to/sys.creds \

  --data-dir /data/insights
```

| Flag       | Default    | Purpose                                                                                                                                                                                           |
| ---------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `data-dir` | (temp dir) | Base directory for DuckDB (`insights.db`, or `systems/<id>.db` per system on a node with several systems), the embedded server's JetStream (`nats/`), and the identity snapshot (`identity.json`) |

The default `web.session-seed` is `"insights"`, which preserves sessions across restarts out of the box. For production, set it to a unique secret:

```
--web.session-seed "your-secret-seed"
```

### External NATS sink

Disable the embedded NATS server and use an external NATS cluster with JetStream for the scrape stream. This decouples the scraper from the indexer and enables distributed topologies.

```
./insights \

  --sys.server nats://target:4222 \

  --sys.creds /path/to/sys.creds \

  --sink.embed=false \

  --server nats://sink-cluster:4222 \

  --creds /path/to/sink.creds \

  --data-dir /data/insights
```

The `nats.*` config is the top-level connection used by the API, indexer, and scraper sink (unless overridden). A node that scrapes a system creates that system's stream on whichever sink it writes to, embedded or external, named `scrape_<id>` and capturing `$INS.sys.<id>.scrape.>`, where `<id>` is its `system.id`. A stream that already exists is used as-is and never updated, so one you pre-created keeps its own limits.

#### Stream shape

The stream is a buffer between scraper and indexer; the database keeps the history. Size it with the `sink` settings:

| Flag             | Default | Purpose                                                                                                                      |
| ---------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `sink.retention` | `24h`   | Buffer depth, enforced by pruning whole epochs rather than by the stream's `max_age` (`0` = unlimited)                       |
| `sink.replicas`  | `1`     | Stream replicas. Above 1 requires a clustered external sink; the embedded server is a single node and the config is rejected |
| `sink.storage`   | `file`  | `file` or `memory`. Memory storage suits a deployment that can afford to lose the buffer on restart                          |

```
sink:

  embed: false

  retention: 6h

  replicas: 3

  storage: file
```

#### Separate NATS connections

You can override the NATS connection for the indexer or scraper independently:

```
nats:

  server: nats://api-cluster:4222



indexer:

  nats:

    server: nats://data-cluster:4222



scraper:

  nats:

    server: nats://data-cluster:4222
```

The indexer uses `indexer.nats`, or the top-level `nats` if that is unset. The scraper uses `scraper.nats`, then the indexer's connection, then the top-level `nats`.

#### Sink connection permissions

An unrestricted JetStream user needs no grants. For a least-privilege user, grant the subjects below. The connection creates and fills the scrape stream, consumes it for the indexer, serves the `$INS` API (NATS micro), and stores webhook delivery history.

Insights **publishes** to:

| Subject     | Purpose                                                                                                                           |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `$INS.>`    | `$INS` API replies, scrape stream (`$INS.sys.<id>.scrape.>`), and webhook delivery log (`$INS.node.<node_id>.webhook.delivery.*`) |
| `$JS.API.>` | Create, inspect, and purge the scrape stream and webhook delivery stream; drive the indexer consumer                              |
| `$JS.FC.>`  | JetStream consumer flow-control replies                                                                                           |
| `_INBOX.>`  | Reply inbox for API, micro, and JetStream requests                                                                                |

Insights **subscribes** to:

| Subject    | Purpose                             |
| ---------- | ----------------------------------- |
| `$INS.>`   | The `$INS` API this node serves     |
| `$SRV.>`   | NATS micro discovery                |
| `_INBOX.>` | Replies to requests this node makes |

Endpoint configuration is held in memory; only the delivery history is stored in JetStream. The webhook delivery stream (`webhook-deliveries_<node_id>`) is created only when at least one notification endpoint is configured, so its `$JS.API.>` grant is needed only on a node that delivers notifications; `<node_id>` is `node.id`, which on a node serving a system defaults to that system's id. To scope the connection to one system, narrow `$INS.>` to `$INS.sys.<id>.>`, `$INS.node.<node_id>.>`, and `$INS.ping`; `$JS.API.>` narrows to `INFO`, `STREAM.CREATE`/`UPDATE`/`INFO`/`PURGE`, `CONSUMER.>`, and `STREAM.MSG.GET`. `INFO` is the account info a scraping node reads to report its JetStream domain. Scrape streams are created once and never updated, but the webhook delivery store reconciles its stream with `CreateOrUpdateStream`, so `STREAM.UPDATE` is needed on every restart after the first on a node that delivers notifications; drop `STREAM.CREATE` if the streams are pre-created.

#### Server config

Attach the permissions to a user on the sink's JetStream-enabled account:

```
accounts {

  INSIGHTS {

    jetstream: enabled

    users = [

      {

        # the sink user's public nkey (U…)

        nkey: UAINSIGHTSSINKEXAMPLEPUBLICUSERNKEY0000000000000000000

        permissions {

          publish {

            allow = [

              "$INS.>"

              "$JS.API.>"

              "$JS.FC.>"

              "_INBOX.>"

            ]

          }

          subscribe {

            allow = [

              "$INS.>"

              "$SRV.>"

              "_INBOX.>"

            ]

          }

        }

      }

    ]

  }

}
```

#### Decentralized JWT (`nats auth`)

Enable JetStream on the account, then attach the permissions to a **scoped signing key** (a reusable template) or **directly to the user**:

```
nats auth account edit INSIGHTS --jetstream --js-memory 1G --js-disk 10G



nats auth account keys add INSIGHTS insights-sink \

  --pub-allow '$INS.>' \

  --pub-allow '$JS.API.>' \

  --pub-allow '$JS.FC.>' \

  --pub-allow '_INBOX.>' \

  --sub-allow '$INS.>' \

  --sub-allow '$SRV.>' \

  --sub-allow '_INBOX.>'



# issue a user under that scoped key (it inherits the template) and write its creds

nats auth user add insights-sink INSIGHTS --key <SIGNING_KEY> --defaults

nats auth user credential sink.creds insights-sink INSIGHTS
```

Attach the same `--pub-allow`/`--sub-allow` flags directly to `nats auth user add … --defaults` to skip the signing-key template.

### Without the web UI

Run without the web UI. The NATS micro API remains available for direct `nats req` queries.

```
./insights \

  --sys.server nats://target:4222 \

  --sys.creds /path/to/sys.creds \

  --web=false
```

Query via the NATS CLI:

```
nats req '$INS.sys.default.db.query' '{"sql": "SELECT * FROM hx.servers LIMIT 5"}'
```

`default` is the `system.id` a node uses when none is set; every system-scoped subject carries it. `insights db query` resolves the id for you.

### Indexer-only

Consume from an existing scrape stream without running the scraper. Useful when a separate instance or process handles scraping.

```
./insights \

  --scraper=false \

  --sink.embed=false \

  --server nats://data-cluster:4222 \

  --system.id prod \

  --node.id prod-indexer \

  --data-dir /data/insights
```

This node neither scrapes nor mirrors the system, so it creates no stream: `scrape_<system.id>` must already exist, and `system.id` must match the node that fills it. The two nodes share a system id in different roles (scrape and catalog), which is allowed. They must not share a node id, and `node.id` defaults to the system id on both, so name at least one of them. Insights checks both rules over `$INS.ping` at startup and refuses to start on a clash.

### Split web tier

Run the web UI on its own node as a pure client of a remote indexer/scraper node over NATS. The web tier owns no database, runs no scraper or indexer, and validates no license. It discovers the systems on its NATS over `$INS.ping` and each backend's capabilities (geo enrichment, realtime feed, scrape interval, retention, license) over `$INS.sys.<id>.ops.info`. Useful for placing the UI close to operators, scaling it independently, or exposing it in a different network zone from the data plane.

```
insights web \

  --server nats://data-cluster:4222 \

  --web.hostname 0.0.0.0 --web.port 8080
```

`insights web` reads the `web:`, `prometheus:` and `updater:` sections of a config file, and `telemetry.enabled`, as well as their flags.

The same read-only NATS query API also backs `insights mcp` (a read-only MCP server for AI assistants) and `insights http` (an HTTP gateway for HTTP-only clients such as Grafana's Infinity data source). Each runs as an independent pure client. See the CLI reference.

### Multiple systems on one node

One node can monitor several NATS systems. List them under `systems:` and each gets its own DuckDB database (`<data-dir>/systems/<id>.db`), its own stream, its own indexer and check thresholds, and its own `$INS.sys.<id>.*` API. An entry with a `scrape:` block is scraped by this node; an entry without one is filled by a [collector](/insights/1.2/guides/federated-deployment.md#option-2-collector-at-a-site).

```
node:

  id: us-east                # required on a node serving several systems



data-dir: /var/lib/insights



nats:

  server: nats://central:4222

  creds: /etc/insights/insights.creds



sink:

  embed: false



db:

  memory-limit: 4GiB         # per system: each has its own DuckDB instance



systems:

  - id: core-prod

    scrape:

      nats:

        server: nats://core.internal:4222

        creds: /etc/insights/core-prod.creds

      interval: 20s

    retention: 24h           # stream buffer depth; the database keeps history

  - id: edge-west            # collector-fed: no scrape block

    retention: 6h
```

The list is YAML-only and replaces the flat single-system config rather than sitting beside it: `systems:` together with a non-default `system.id`, a `sys.*` connection, the simulator, or the indexer turned off is a config error. A single-system node keeps the flat `system:`/`sys:`/`scraper:`/`sink:` config. `db.memory-limit` is per instance, so three systems at `4GiB` can use `12GiB`.

A system whose database will not open (a corrupt file, a bad path) is logged, skipped, and retried every minute while the other systems serve; a node where no database opens at all exits non-zero. The web UI serves each system under `/systems/<id>/` and lists them on a Systems page. See the `systems` configuration reference for every per-entry key.

### Federated

To monitor NATS systems at several sites, run Insights at each site and connect the sites to a central NATS server over leafnodes. One web UI then shows every site. Each site runs either a full Insights node or a collector that only scrapes. See [Federated Deployment](/insights/1.2/guides/federated-deployment.md).

## NATS authentication

All NATS connections support the same authentication options:

| Method           | Flags                           | Description                                                                                     |
| ---------------- | ------------------------------- | ----------------------------------------------------------------------------------------------- |
| Credentials file | `--*.creds /path/to/file.creds` | NKey + JWT bundled file                                                                         |
| NATS context     | `--*.context my-context`        | Named context from `nats context`                                                               |
| Basic auth       | `--*.user X --*.password Y`     | Username/password                                                                               |
| NKey             | `--*.nkey SEED`                 | User NKey seed (`SU…`); alone it authenticates as an nkey user, with `--*.jwt` it signs the JWT |
| JWT              | `--*.jwt TOKEN`                 | User JWT string                                                                                 |

Replace `*` with the connection prefix: `nats`, `sys`, `indexer.nats`, or `scraper.nats`. In YAML, a multi-system entry takes the same keys under `systems[].scrape.nats`. For the top-level connection, every subcommand (`serve`, `db`, `checks`, `ops`, `config`, `system`, `node`, `web`, `mcp`, `http`, `prometheus`) uses the unprefixed `nats`-CLI-style flags: `--server`, `--creds`, `--context`, `--user`, `--tls-ca-cert`, and so on; the `--nats.*` names remain as hidden aliases.

### TLS

```
--sys.tls-cert /path/to/cert.pem \

--sys.tls-key /path/to/key.pem \

--sys.tls-ca-cert /path/to/ca.pem
```

Use `--*.tls-first` when the server requires TLS handshake before the NATS protocol.

### SOCKS proxy

Route NATS connections through a SOCKS proxy:

```
--sys.socks-proxy socks5://proxy:1080
```

### System account permissions

When you give the `sys` connection a dedicated, restricted user instead of full system-account credentials, grant the subjects below. The `*` in `$SYS.REQ.SERVER.*.<endpoint>` matches both the `PING` broadcast token and a concrete server-id, so it covers the default broadcast scrape **and** the per-server requests Insights issues when a scrape filter excludes part of the deployment.

The endpoint token is enumerated deliberately. Do not collapse it to `$SYS.REQ.SERVER.*.*`. That wildcard would also grant the administrative `RELOAD`, `KICK`, and `LDM` actions (reload server config, disconnect a client, drain a server into lame-duck mode), which Insights does not use and which turn a read-only monitoring credential into one that can disrupt the deployment.

Insights **publishes** to these request subjects (request/reply; replies arrive on `_INBOX`, below):

| Subject                       | Purpose                    |
| ----------------------------- | -------------------------- |
| `$SYS.REQ.SERVER.PING`        | Server discovery           |
| `$SYS.REQ.SERVER.*.VARZ`      | Server metrics             |
| `$SYS.REQ.SERVER.*.CONNZ`     | Connections                |
| `$SYS.REQ.SERVER.*.SUBSZ`     | Subscriptions              |
| `$SYS.REQ.SERVER.*.ROUTEZ`    | Routes                     |
| `$SYS.REQ.SERVER.*.GATEWAYZ`  | Gateways                   |
| `$SYS.REQ.SERVER.*.LEAFZ`     | Leaf nodes                 |
| `$SYS.REQ.SERVER.*.ACCOUNTZ`  | Per-server account details |
| `$SYS.REQ.SERVER.*.JSZ`       | JetStream                  |
| `$SYS.REQ.SERVER.*.HEALTHZ`   | Health                     |
| `$SYS.REQ.SERVER.*.RAFTZ`     | Raft groups                |
| `$SYS.REQ.SERVER.*.IPQUEUESZ` | Internal queues            |
| `$SYS.REQ.ACCOUNT.PING.STATZ` | Account stats              |

Insights **subscribes** to these event subjects:

| Subject                         | Purpose                                                                                                                                                                  |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `$SYS.ACCOUNT.*.DISCONNECT`     | Real-time client disconnect events, including mid-session auth failures and, from nats-server 2.12, clients rejected at connect                                          |
| `$SYS.SERVER.*.CLIENT.AUTH.ERR` | Real-time connect-time authentication rejections; the only event for a rejected leafnode, route or gateway connection, and for a rejected client before nats-server 2.12 |
| `_INBOX.>`                      | Replies to the request subjects above                                                                                                                                    |

The following **publish** subjects are optional. Omitting them only disables on-demand server diagnostics:

| Subject                      | Purpose                            |
| ---------------------------- | ---------------------------------- |
| `$SYS.REQ.SERVER.*.PROFILEZ` | On-demand server profiling         |
| `$SYS.REQ.SERVER.*.EXPVARZ`  | On-demand server runtime variables |

The two subscribe subjects feed the realtime events view. `DISCONNECT` surfaces clients that drop after connecting (including sessions whose credentials expire or fail re-authentication) and, from nats-server 2.12, clients rejected during the connect handshake, as a disconnect from `$G` with reason `Authentication Failure` or `Authentication Timeout`. `CLIENT.AUTH.ERR` reports a rejected client on any version; Insights drops it where the server also sent the `DISCONNECT`, so each failed login counts once. It is also the only event for a leafnode, route or gateway connection rejected at connect. Before 2.12 an auth timeout produces no event. Both are subscribed by Insights directly; neither is consumed by the scraper. Omit `CLIENT.AUTH.ERR` if you don't need auth-failure visibility for leafnode, route and gateway connections, or for clients on servers before 2.12.

`_INBOX.>` is subscribe-only: Insights is the requester, so the server publishes replies to its inbox, so no publish permission on `_INBOX` is needed. No `$SRV.>` or scrape-control permissions are required: the scraper's on-demand control service runs on the Insights connection, not the monitored system.

#### Server config

In a server configuration file, attach the permissions to the system-account user:

```
accounts {

  SYS {

    users = [

      {

        # the monitoring user's public nkey (U…)

        nkey: UAINSIGHTSMONITOREXAMPLEPUBLICUSERNKEY00000000000000000

        permissions {

          publish {

            allow = [

              "$SYS.REQ.SERVER.PING"

              "$SYS.REQ.SERVER.*.VARZ"

              "$SYS.REQ.SERVER.*.CONNZ"

              "$SYS.REQ.SERVER.*.SUBSZ"

              "$SYS.REQ.SERVER.*.ROUTEZ"

              "$SYS.REQ.SERVER.*.GATEWAYZ"

              "$SYS.REQ.SERVER.*.LEAFZ"

              "$SYS.REQ.SERVER.*.ACCOUNTZ"

              "$SYS.REQ.SERVER.*.JSZ"

              "$SYS.REQ.SERVER.*.HEALTHZ"

              "$SYS.REQ.SERVER.*.RAFTZ"

              "$SYS.REQ.SERVER.*.IPQUEUESZ"

              "$SYS.REQ.ACCOUNT.PING.STATZ"

            ]

          }

          subscribe {

            allow = [

              "$SYS.ACCOUNT.*.DISCONNECT"

              "$SYS.SERVER.*.CLIENT.AUTH.ERR"

              "_INBOX.>"

            ]

          }

        }

      }

    ]

  }

}

system_account: SYS
```

#### Decentralized JWT (`nats auth`)

With operator/account JWTs, attach the permissions either to a **scoped signing key** (a reusable template, so every user issued under it inherits the same grants) or **directly to the user**.

Scoped signing key (define the template once on the account, then issue users under it):

```
nats auth account keys add SYS insights-monitor \

  --pub-allow '$SYS.REQ.SERVER.PING' \

  --pub-allow '$SYS.REQ.SERVER.*.VARZ' \

  --pub-allow '$SYS.REQ.SERVER.*.CONNZ' \

  --pub-allow '$SYS.REQ.SERVER.*.SUBSZ' \

  --pub-allow '$SYS.REQ.SERVER.*.ROUTEZ' \

  --pub-allow '$SYS.REQ.SERVER.*.GATEWAYZ' \

  --pub-allow '$SYS.REQ.SERVER.*.LEAFZ' \

  --pub-allow '$SYS.REQ.SERVER.*.ACCOUNTZ' \

  --pub-allow '$SYS.REQ.SERVER.*.JSZ' \

  --pub-allow '$SYS.REQ.SERVER.*.HEALTHZ' \

  --pub-allow '$SYS.REQ.SERVER.*.RAFTZ' \

  --pub-allow '$SYS.REQ.SERVER.*.IPQUEUESZ' \

  --pub-allow '$SYS.REQ.ACCOUNT.PING.STATZ' \

  --sub-allow '$SYS.ACCOUNT.*.DISCONNECT' \

  --sub-allow '$SYS.SERVER.*.CLIENT.AUTH.ERR' \

  --sub-allow '_INBOX.>'



# issue a user under that scoped key (it inherits the template) and write its creds.

# <SIGNING_KEY> is the insights-monitor key from `nats auth account keys ls SYS`.

nats auth user add insights SYS --key <SIGNING_KEY> --defaults

nats auth user credential insights.creds insights SYS
```

Directly on the user (the same `--pub-allow`/`--sub-allow` flags, on `user add`):

```
nats auth user add insights SYS \

  --pub-allow '$SYS.REQ.SERVER.PING' \

  --pub-allow '$SYS.REQ.SERVER.*.VARZ' \

  --pub-allow '$SYS.REQ.SERVER.*.CONNZ' \

  --pub-allow '$SYS.REQ.SERVER.*.SUBSZ' \

  --pub-allow '$SYS.REQ.SERVER.*.ROUTEZ' \

  --pub-allow '$SYS.REQ.SERVER.*.GATEWAYZ' \

  --pub-allow '$SYS.REQ.SERVER.*.LEAFZ' \

  --pub-allow '$SYS.REQ.SERVER.*.ACCOUNTZ' \

  --pub-allow '$SYS.REQ.SERVER.*.JSZ' \

  --pub-allow '$SYS.REQ.SERVER.*.HEALTHZ' \

  --pub-allow '$SYS.REQ.SERVER.*.RAFTZ' \

  --pub-allow '$SYS.REQ.SERVER.*.IPQUEUESZ' \

  --pub-allow '$SYS.REQ.ACCOUNT.PING.STATZ' \

  --sub-allow '$SYS.ACCOUNT.*.DISCONNECT' \

  --sub-allow '$SYS.SERVER.*.CLIENT.AUTH.ERR' \

  --sub-allow '_INBOX.>' \

  --defaults

nats auth user credential insights.creds insights SYS
```

Add `--pub-allow '$SYS.REQ.SERVER.*.PROFILEZ' --pub-allow '$SYS.REQ.SERVER.*.EXPVARZ'` (or the matching `allow` entries in the server config) to enable the optional on-demand diagnostics.

## Scraper tuning

### Interval and timeout

```
--scraper.interval 1m     # Time between scrape cycles (default: 1m)

--scraper.timeout 30s     # Per-request timeout (default: 30s)
```

## Indexer tuning

### Retention

Limit how much historical data is kept in DuckDB:

```
--db.retention.duration 24h  # Keep last 24 hours (default: 768h)

--db.retention.interval 10m  # Minimum time between sweeps (default: 10m)
```

### Stream position

Control where the indexer starts consuming:

```
--indexer.stream-latest      # Start from the latest message

--indexer.stream-epoch 500   # Start from a specific epoch number
```

### Filtering and downsampling

Process only specific endpoint types or reduce resolution:

```
--indexer.stream-filters varz,jsz,connz   # Only process these endpoints

--indexer.stream-interval 60s             # Skip epochs closer than 60s apart
```

## Version checker

Periodically checks GitHub for a newer release and notifies via a log message and a badge in the web UI. Does not download or replace the binary automatically. `insights-collector` and `insights web` take the same settings. See [Version Updates](/insights/1.2/guides/updates.md) for details.

```
--updater \

--updater.interval 1h
```

Official release binaries and container images are stamped with a version string; the checker only runs when a real version is present (binaries reporting `dev` are skipped). See [Version Updates](/insights/1.2/guides/updates.md) for how the check works.

## Web server

```
--web.hostname 0.0.0.0    # Bind to all interfaces (default: 127.0.0.1)

--web.port 8080            # Bind port (default: 8080)
```

## Monitoring with Prometheus & Grafana

Insights exposes check failure metrics via a Prometheus-compatible endpoint, enabled by default on `127.0.0.1:9091/metrics`.

```
--prometheus               # Enable/disable the endpoint (default: true)

--prometheus.hostname      # Bind hostname (default: 127.0.0.1)

--prometheus.port          # Bind port (default: 9091)
```

For full setup instructions including Prometheus scrape configuration, Grafana dashboard import, and alerting rules, see the [Grafana guide](/insights/1.2/guides/grafana.md).

## Environment variables

Every flag has a corresponding environment variable with the `INSIGHTS_` prefix. Nested configs add their prefix:

| Flag                      | Environment Variable             |
| ------------------------- | -------------------------------- |
| `--server`                | `INSIGHTS_NATS_SERVER`           |
| `--sys.creds`             | `INSIGHTS_SYS_CREDS`             |
| `--data-dir`              | `INSIGHTS_DATA_DIR`              |
| `--scraper.interval`      | `INSIGHTS_SCRAPER_INTERVAL`      |
| `--db.retention.duration` | `INSIGHTS_DB_RETENTION_DURATION` |
| `--web.port`              | `INSIGHTS_WEB_PORT`              |
| `--sink.embed`            | `INSIGHTS_SINK_EMBED`            |
| `--simulator`             | `INSIGHTS_SIMULATOR`             |
| `--log-level`             | `INSIGHTS_LOG_LEVEL`             |

## Config file

Pass a YAML config file with `--config`:

```
./insights --config /etc/insights/config.yaml
```

See [config.example.yaml](/assets/files/config.example-87de6145766e1a76f889eaa3d6e150c7.yaml) for all options. Configuration priority (highest wins):

1. CLI flags
2. Environment variables
3. Config file
4. Defaults

Validate a config file before a rollout. `insights config check` loads exactly what `serve` would and reports what it would reject, without opening a database, starting a listener, or connecting to anything:

```
./insights config check --config /etc/insights/config.yaml
```

## Containers and supervisors

### Container images

The Linux images run as UID and GID 1000, not root, and carry a pre-created `/var/lib/insights` owned by that user. Point `--data-dir` there:

```
docker run -d --restart unless-stopped \

  -p 8080:8080 \

  -v insights-data:/var/lib/insights \

  -v /path/to/sys.creds:/etc/insights/sys.creds:ro \

  registry.synadia.io/insights:1.2.2 \

  --sys.server nats://your-nats:4222 \

  --sys.creds /etc/insights/sys.creds \

  --data-dir /var/lib/insights \

  --web.hostname 0.0.0.0
```

A named volume mounted at `/var/lib/insights` inherits that ownership. A named volume mounted anywhere else, a bind mount, and a Kubernetes PersistentVolumeClaim do not, and startup fails with `permission denied` (for example `write identity.json: ... permission denied`, or `create data dir: ... permission denied` for a directory below the mount). Make the directory writable by UID 1000:

```
chown -R 1000:1000 /var/lib/insights   # a bind-mounted host directory
```

In Kubernetes, set `securityContext.fsGroup: 1000` on the pod. The [Helm chart](/insights/1.2/getting-started/installation.md#kubernetes-with-helm) runs as UID/GID 1000 and sets `fsGroup` for you. The images have no shell; attach a debug container rather than `exec` into one. The collector image (the `-collector` tag, entrypoint `insights-collector`) runs as the same user, and needs no data directory: run it as `docker run -v /path/to/collector:/etc/insights:ro registry.synadia.io/insights:1.2.2-collector -c /etc/insights/collector.yaml`.

### Restart policy

Run Insights under a supervisor that restarts it: a Kubernetes Deployment or StatefulSet, a systemd unit with `Restart=`, or `--restart` on a container. If DuckDB recovery after a fatal database error cannot complete within 30 seconds, Insights exits with a non-zero status so the supervisor can start a fresh process, rather than keep serving a database that cannot answer. Startup exits non-zero the same way when no database opens at all.

### Stopping

Insights shuts down cleanly on `SIGTERM` or `SIGINT`: it stops the scrapers and indexers, stops the web and API listeners, and checkpoints each DuckDB database before closing it, so the next start does not replay the write-ahead log. Databases are checkpointed one at a time and each checkpoint is bounded to 5 seconds, so give the supervisor a stop timeout of at least 5 seconds per system; `docker stop` waits 10 seconds by default and Kubernetes 30. A process killed before shutdown completes replays its write-ahead log on the next start.

### Health probe

`GET /healthz` on the web listener returns `200` with the body `ok`, unauthenticated. The listener binds only after the databases are built, so a success means Insights itself is serving. Point container, Kubernetes, and load-balancer probes at it rather than at a page route.

The web UI has no built-in authentication. If an authenticating proxy sits in front of it, exclude `/healthz` from authentication, or the probe gets a login redirect.

## Example: Production deployment

A persistent deployment monitoring a real NATS system with all features enabled:

```
# /etc/insights/config.yaml

log-level: info



data-dir: /data/insights



sys:

  server: nats://nats.internal:4222

  creds: /etc/insights/sys.creds



web:

  hostname: 0.0.0.0

  port: 8080

  session-seed: "your-stable-secret"



db:

  retention:

    duration: 72h



scraper:

  interval: 20s



updater:

  enabled: true

  interval: 1h
```

```
./insights --config /etc/insights/config.yaml
```
