Deployment Guide

Insights is one binary made of parts you can turn on and off. Which parts you turn on, and where, decides how it is deployed. See Architecture for what each part does.

This page shows only the flags each setup needs. Configuration lists every flag, environment variable, and YAML key.

Each part has an on/off flag named after it: --scraper, --indexer, --web, --simulator, --prometheus, --updater, and --telemetry. The environment variables follow the same pattern, such as INSIGHTS_WEB=false. In YAML, use the section's enabled: key. The API starts on any node that scrapes or indexes a system. A node that only serves the web UI uses another node's API.

To try Insights with the simulator, see the Quick Start.

Deployment Options

Single Node

Monitor a NATS system with system account credentials:

./insights \
  --sys.server nats://target:4222 \
  --sys.creds /path/to/sys.creds

Or with a NATS context:

./insights --sys.context my-system

Without --data-dir, the database and stream are written to a temporary directory and lost on restart. Add --data-dir to keep them:

./insights \
  --sys.server nats://target:4222 \
  --sys.creds /path/to/sys.creds \
  --data-dir /data/insights

web.session-seed defaults to "insights", which keeps web sessions valid across restarts. In production, set it to your own secret:

--web.session-seed "your-secret-seed"

For every --sys.* authentication option (creds, context, basic auth, NKey, JWT, TLS, SOCKS proxy), see Configuration › sys.*.

External NATS Server

By default, Insights runs an embedded NATS server to hold the stream between the scraper and the indexer. To use your own NATS server with JetStream instead, turn off the embedded one and connect to yours:

./insights \
  --sys.server nats://target:4222 \
  --sys.creds /path/to/sys.creds \
  --sink.embed=false \
  --server nats://sink-cluster:4222 \
  --creds /path/to/sink.creds \
  --data-dir /data/insights

--server, --creds, --context, --tls-ca-cert, and the other connection flags work like the nats CLI's. They set the connection the API, indexer, and scraper use unless you override it. The older --nats.* names still work, and the YAML section is nats:.

A node that scrapes a system creates its stream on that NATS server, named scrape_<id> with the subjects $INS.sys.<id>.scrape.>, where <id> is the system.id. If the stream already exists, Insights uses it as is, so a stream you created keeps your limits.

The stream only buffers data until the indexer stores it. The database keeps the history. Size the stream with the sink settings:

You can give the indexer and scraper their own NATS connections. The indexer uses indexer.nats, or the top-level nats if that is unset. The scraper uses scraper.nats, then the indexer's connection, then the top-level nats.

nats:
  server: nats://api-cluster:4222

indexer:
  nats:
    server: nats://data-cluster:4222

scraper:
  nats:
    server: nats://data-cluster:4222

Without the Web UI

Turn off the web UI with --web=false. Insights still scrapes, stores, and runs checks, and the API still answers queries over NATS:

./insights \
  --sys.server nats://target:4222 \
  --sys.creds /path/to/sys.creds \
  --web=false

nats req '$INS.sys.default.db.query' '{"sql": "SELECT * FROM hx.servers LIMIT 5"}'

default is the system.id a node uses when none is set. insights db query finds the id for you. To give AI agents access, see AI Agents.

Indexer Only

Store and check data from a stream that another node fills, without scraping:

./insights \
  --scraper=false \
  --sink.embed=false \
  --server nats://data-cluster:4222 \
  --system.id prod \
  --node.id prod-indexer \
  --data-dir /data/insights

This node doesn't create a stream, so scrape_<system.id> must already exist, and system.id must match the node that fills it. The two nodes share the system id but need different node ids. Because node.id defaults to the system id, name at least one of them. Insights checks both at startup and refuses to start if they clash.

Separate Web UI

Run the web UI as its own process with insights web. It has no database, scraper, indexer, or license. It connects over NATS, finds the systems that answer there, and queries them:

./insights web \
  --server nats://data-cluster:4222 \
  --creds /path/to/api.creds

insights web also reads the web:, prometheus:, and updater: sections of a config file, and telemetry.enabled. The same NATS connection works for insights mcp, a read-only MCP server for AI agents, and insights http, a JSON and CSV gateway to the query API.

Multiple Systems on One Node

One node can monitor several NATS systems. List them under systems:. Each system gets its own database (<data-dir>/systems/<id>.db), stream, indexer, check thresholds, and $INS.sys.<id>.* API. An entry with a scrape: block is scraped by this node. An entry without one is filled by a collector.

node:
  id: us-east # required on a node with several systems

data-dir: /var/lib/insights

nats:
  server: nats://central:4222
  creds: /etc/insights/insights.creds

sink:
  embed: false

db:
  memory-limit: 4GiB # per system

systems:
  - id: core-prod
    scrape:
      nats:
        server: nats://core.internal:4222
        creds: /etc/insights/core-prod.creds
      interval: 20s
    retention: 24h # how much the stream keeps; the database keeps the history
  - id: edge-west # filled by a collector, so no scrape block
    retention: 6h

systems: can only be set in YAML, and it replaces the single-system settings. Using it together with a non-default system.id, a sys.* connection, the simulator, or --indexer=false is a config error. db.memory-limit applies to each system, so three systems at 4GiB can use 12GiB in total.

If one system's database won't open, Insights logs it, skips that system, and retries every minute while the others keep running. If no database opens, Insights exits with an error. The web UI shows each system under /systems/<id>/ and lists them on a Systems page. See Configuration for every per-system setting.

Federated

To monitor NATS systems at several sites, run Insights at each site and connect the sites to a central NATS server over leafnodes. One web UI then shows every site. Each site runs either a full Insights node or a collector that only scrapes. See Federated Deployment.

Tuning

Scrape interval and timeout

--scraper.interval 1m     # Time between scrapes (default: 1m)
--scraper.timeout 30s     # Timeout for each request (default: 30s)

Scrape filter

By default Insights scrapes every server it finds. To scrape fewer, allow or deny servers by exact server name, cluster, or tag. Excluded servers receive no requests. Deny wins over allow.

--scraper.filter.allow.clusters us-east,us-west   # Scrape only these clusters
--scraper.filter.deny.tags edge                   # Never scrape servers with this tag

Retention

Limit how much history the database keeps:

--db.retention.duration 24h  # Keep the last 24 hours (default: 768h)
--db.retention.interval 10m  # Minimum time between cleanups (default: 10m)

Web binding

--web.hostname 0.0.0.0    # Listen on all interfaces (default: 127.0.0.1)
--web.port 8080            # Port (default: 8080)

To serve HTTPS, turn on TLS and give a certificate and key. Turning on TLS without both is an error.

--web.tls \
--web.tls-cert /path/to/cert.pem \
--web.tls-key /path/to/key.pem

Health probe

GET /healthz on the web port returns 200 with the body ok. It needs no authentication and does no work, so you can poll it often.

curl -fsS http://localhost:8080/healthz

Point container, Kubernetes, and load balancer probes at /healthz, not at a page. The web port opens only after the databases are ready, so a success means Insights is serving. A proxy in front of Insights can answer a page request while Insights is still starting.

The web UI has no built-in authentication. If an authenticating proxy sits in front of it, exclude /healthz from authentication, or the probe gets a login redirect.

insights http has its own /healthz, described in the API reference. NATS servers also have a healthz monitoring endpoint, which Insights scrapes and which checks such as server-001 and server-014 through server-016 report on.

Notifications

Send check findings to other systems. Endpoints can only be set in the YAML file. Webhook endpoints receive the full payload, and Alertmanager endpoints receive Alertmanager v2 alerts. Each endpoint subscribes to the check codes it wants. See Notifications.

notifications:
  webhook:
    - name: ops-relay
      url: https://hooks.example.com/insights
      subscriptions:
        - server-001
        - stream-002
  alertmanager:
    - name: alertmanager
      url: http://alertmanager:9093/api/v2/alerts

Containers and Supervisors

Container images

The Linux images run as UID and GID 1000 and include a /var/lib/insights directory owned by that user. Log in to registry.synadia.io with your registry credentials first; see Installation. Use /var/lib/insights as the data directory:

docker run -d --restart unless-stopped \
  -p 8080:8080 \
  -v insights-data:/var/lib/insights \
  -v /path/to/sys.creds:/etc/insights/sys.creds:ro \
  registry.synadia.io/insights:1.2.0 \
  --sys.server nats://your-nats:4222 \
  --sys.creds /etc/insights/sys.creds \
  --data-dir /var/lib/insights \
  --web.hostname 0.0.0.0

A named volume at /var/lib/insights gets the right owner. Other volumes, bind mounts, and Kubernetes PersistentVolumeClaims don't, and startup fails with permission denied (for example write identity.json: ... permission denied). Make the directory writable by UID 1000:

chown -R 1000:1000 /var/lib/insights   # a bind-mounted host directory

In Kubernetes, set securityContext.fsGroup: 1000 on the pod. The Helm chart does this for you. The images have no shell, so attach a debug container instead of using exec.

Restart policy

Run Insights under something that restarts it: a Kubernetes Deployment or StatefulSet, a systemd unit with Restart=, or --restart on a container. If the database hits a fatal error and can't recover within 30 seconds, Insights exits so it can be restarted cleanly. It also exits at startup if no database opens.

Stopping

On SIGTERM or SIGINT, Insights stops scraping and indexing, closes its listeners, and saves each database to disk before exiting, so the next start is fast. Saving takes up to 5 seconds per system, one system at a time, so allow a stop timeout of at least 5 seconds per system. docker stop waits 10 seconds by default and Kubernetes waits 30. If the process is killed before it finishes, the next start replays the database's write-ahead log.

Example Config File

A node that monitors a NATS system and keeps its data:

# /etc/insights/config.yaml
log-level: info

data-dir: /data/insights

sys:
  server: nats://nats.internal:4222
  creds: /etc/insights/sys.creds

web:
  hostname: 0.0.0.0
  port: 8080
  session-seed: 'your-stable-secret'

db:
  retention:
    duration: 72h

scraper:
  interval: 20s # faster than the 1m default

./insights --config /etc/insights/config.yaml

Validate a config file before a rollout. insights config check loads the config as serve would and reports any errors, without opening a database, starting a listener, or connecting to anything:

./insights config check --config /etc/insights/config.yaml

See Configuration for every key, its environment variable, and the order in which settings are applied.