Deployment Guide

Insights is a single Go binary with independently toggleable subsystems. How you configure them determines the deployment topology.

Subsystems

Subsystem	Flag	Default	Description
Scraper	`--scraper`	`true`	Scrapes `$SYS` endpoints from a target NATS system
Indexer	`--indexer`	`true`	Processes scrape stream into DuckDB
Web UI	`--web`	`true`	HTTP server for the dashboard
Embedded Sink	`--sink.embed`	`true`	In-process NATS server with JetStream
Simulator	`--simulator`	`false`	In-process NATS clusters with traffic workloads
Prometheus	`--prometheus`	`true`	Prometheus-compatible `/metrics` endpoint
Version checker	`--updater`	`true`	Periodic check for a newer release (notification only)
Telemetry	`--telemetry`	`true`	Product usage pulse, in trial builds only (see Telemetry)

The API service (NATS micro) starts on any node that indexes or scrapes a system. A web-only node is a client of it and serves no API of its own.

Each toggle is a shorthand: --web=false on the command line, INSIGHTS_WEB=false in the environment. Both spell the same thing as the verbose --web.enabled=false / INSIGHTS_WEB_ENABLED=false, which still parse but are hidden from --help. In YAML, use a web: section with enabled:; see the Configuration reference.

Deployment Topologies

The following diagrams show which subsystems run in each topology.

Demo (Simulator)

Zero-config mode for evaluation. Starts an embedded NATS simulator with traffic workloads. All data is ephemeral.

./insights --simulator

Everything runs in-process: simulator, embedded sink, scraper, indexer, web UI. No external NATS required.

Choose a profile to control deployment size:

./insights --simulator --simulator.profile super-medium

Available profiles: core-{small,medium,large}, js-{small,medium,large}, super-{small,medium,large}, leaf-{small,medium,large}, super-leaf-{small,medium,large}.

All-in-One (Ephemeral)

Monitor a real NATS deployment. Requires system account credentials for $SYS access.

./insights \
  --sys.server nats://target:4222 \
  --sys.creds /path/to/sys.creds

Or with a NATS context:

./insights --sys.context my-system

The database and the JetStream store are written to a temporary directory, and the data is lost on restart.

All-in-One (Persistent)

Same as above with durable storage that survives restarts.

./insights \
  --sys.server nats://target:4222 \
  --sys.creds /path/to/sys.creds \
  --data-dir /data/insights

Flag	Default	Purpose
`data-dir`	(temp dir)	Base directory for DuckDB (`insights.db`, or `systems/<id>.db` per system on a node with several systems), the embedded server's JetStream (`nats/`), and the identity snapshot (`identity.json`)

The default web.session-seed is "insights", which preserves sessions across restarts out of the box. For production, set it to a unique secret:

--web.session-seed "your-secret-seed"

External NATS Sink

Disable the embedded NATS server and use an external NATS cluster with JetStream for the scrape stream. This decouples the scraper from the indexer and enables distributed topologies.

./insights \
  --sys.server nats://target:4222 \
  --sys.creds /path/to/sys.creds \
  --sink.embed=false \
  --server nats://sink-cluster:4222 \
  --creds /path/to/sink.creds \
  --data-dir /data/insights

The nats.* config is the top-level connection used by the API, indexer, and scraper sink (unless overridden). A node that scrapes a system creates that system's stream on whichever sink it writes to, embedded or external, named scrape_<id> and capturing $INS.sys.<id>.scrape.>, where <id> is its system.id. A stream that already exists is used as-is and never updated, so one you pre-created keeps its own limits.

Stream shape

The stream is a buffer between scraper and indexer; the database keeps the history. Size it with the sink settings:

Flag	Default	Purpose
`sink.retention`	`24h`	Buffer depth, enforced by pruning whole epochs rather than by the stream's `max_age` (`0` = unlimited)
`sink.replicas`	`1`	Stream replicas. Above 1 requires a clustered external sink; the embedded server is a single node and the config is rejected
`sink.storage`	`file`	`file` or `memory`. Memory storage suits a deployment that can afford to lose the buffer on restart

sink:
  embed: false
  retention: 6h
  replicas: 3
  storage: file

Separate NATS connections

You can override the NATS connection for the indexer or scraper independently:

nats:
  server: nats://api-cluster:4222

indexer:
  nats:
    server: nats://data-cluster:4222

scraper:
  nats:
    server: nats://data-cluster:4222

The indexer uses indexer.nats, or the top-level nats if that is unset. The scraper uses scraper.nats, then the indexer's connection, then the top-level nats.

Sink connection permissions

An unrestricted JetStream user needs no grants. For a least-privilege user, grant the subjects below. The connection creates and fills the scrape stream, consumes it for the indexer, serves the $INS API (NATS micro), and stores webhook delivery history.

Insights publishes to:

Subject	Purpose
`$INS.>`	`$INS` API replies, scrape stream (`$INS.sys.<id>.scrape.>`), and webhook delivery log (`$INS.node.<node_id>.webhook.delivery.*`)
`$JS.API.>`	Create, inspect, and purge the scrape stream and webhook delivery stream; drive the indexer consumer
`$JS.FC.>`	JetStream consumer flow-control replies
`_INBOX.>`	Reply inbox for API, micro, and JetStream requests

Insights subscribes to:

Subject	Purpose
`$INS.>`	The `$INS` API this node serves
`$SRV.>`	NATS micro discovery
`_INBOX.>`	Replies to requests this node makes

Endpoint configuration is held in memory; only the delivery history is stored in JetStream. The webhook delivery stream (webhook-deliveries_<node_id>) is created only when at least one notification endpoint is configured, so its $JS.API.> grant is needed only on a node that delivers notifications; <node_id> is node.id, which on a node serving a system defaults to that system's id. To scope the connection to one system, narrow $INS.> to $INS.sys.<id>.>, $INS.node.<node_id>.>, and $INS.ping; $JS.API.> narrows to INFO, STREAM.CREATE/UPDATE/INFO/PURGE, CONSUMER.>, and STREAM.MSG.GET. INFO is the account info a scraping node reads to report its JetStream domain. Scrape streams are created once and never updated, but the webhook delivery store reconciles its stream with CreateOrUpdateStream, so STREAM.UPDATE is needed on every restart after the first on a node that delivers notifications; drop STREAM.CREATE if the streams are pre-created.

Server config

Attach the permissions to a user on the sink's JetStream-enabled account:

accounts {
  INSIGHTS {
    jetstream: enabled
    users = [
      {
        # the sink user's public nkey (U…)
        nkey: UAINSIGHTSSINKEXAMPLEPUBLICUSERNKEY0000000000000000000
        permissions {
          publish {
            allow = [
              "$INS.>"
              "$JS.API.>"
              "$JS.FC.>"
              "_INBOX.>"
            ]
          }
          subscribe {
            allow = [
              "$INS.>"
              "$SRV.>"
              "_INBOX.>"
            ]
          }
        }
      }
    ]
  }
}

Decentralized JWT (`nats auth`)

Enable JetStream on the account, then attach the permissions to a scoped signing key (a reusable template) or directly to the user:

nats auth account edit INSIGHTS --jetstream --js-memory 1G --js-disk 10G

nats auth account keys add INSIGHTS insights-sink \
  --pub-allow '$INS.>' \
  --pub-allow '$JS.API.>' \
  --pub-allow '$JS.FC.>' \
  --pub-allow '_INBOX.>' \
  --sub-allow '$INS.>' \
  --sub-allow '$SRV.>' \
  --sub-allow '_INBOX.>'

# issue a user under that scoped key (it inherits the template) and write its creds
nats auth user add insights-sink INSIGHTS --key <SIGNING_KEY> --defaults
nats auth user credential sink.creds insights-sink INSIGHTS

Attach the same --pub-allow/--sub-allow flags directly to nats auth user add … --defaults to skip the signing-key template.

Without the Web UI

Run without the web UI. The NATS micro API remains available for direct nats req queries.

./insights \
  --sys.server nats://target:4222 \
  --sys.creds /path/to/sys.creds \
  --web=false

Query via the NATS CLI:

nats req '$INS.sys.default.db.query' '{"sql": "SELECT * FROM hx.servers LIMIT 5"}'

default is the system.id a node uses when none is set; every system-scoped subject carries it. insights db query resolves the id for you.

Indexer-Only

Consume from an existing scrape stream without running the scraper. Useful when a separate instance or process handles scraping.

./insights \
  --scraper=false \
  --sink.embed=false \
  --server nats://data-cluster:4222 \
  --system.id prod \
  --node.id prod-indexer \
  --data-dir /data/insights

This node neither scrapes nor mirrors the system, so it creates no stream: scrape_<system.id> must already exist, and system.id must match the node that fills it. The two nodes share a system id in different roles (scrape and catalog), which is allowed. They must not share a node id, and node.id defaults to the system id on both, so name at least one of them. Insights checks both rules over $INS.ping at startup and refuses to start on a clash.

Split Web Tier

Run the web UI on its own node as a pure client of a remote indexer/scraper node over NATS. The web tier owns no database, runs no scraper or indexer, and validates no license. It discovers the systems on its NATS over $INS.ping and each backend's capabilities (geo enrichment, realtime feed, scrape interval, retention, license) over $INS.sys.<id>.ops.info. Useful for placing the UI close to operators, scaling it independently, or exposing it in a different network zone from the data plane.

insights web \
  --server nats://data-cluster:4222 \
  --web.hostname 0.0.0.0 --web.port 8080

insights web reads the web:, prometheus: and updater: sections of a config file, and telemetry.enabled, as well as their flags.

The same read-only NATS query API also backs insights mcp (a read-only MCP server for AI assistants) and insights http (an HTTP gateway for HTTP-only clients such as Grafana's Infinity data source). Each runs as an independent pure client. See the CLI reference.

Multiple Systems on One Node

One node can monitor several NATS systems. List them under systems: and each gets its own DuckDB database (<data-dir>/systems/<id>.db), its own stream, its own indexer and check thresholds, and its own $INS.sys.<id>.* API. An entry with a scrape: block is scraped by this node; an entry without one is filled by a collector.

node:
  id: us-east                # required on a node serving several systems

data-dir: /var/lib/insights

nats:
  server: nats://central:4222
  creds: /etc/insights/insights.creds

sink:
  embed: false

db:
  memory-limit: 4GiB         # per system: each has its own DuckDB instance

systems:
  - id: core-prod
    scrape:
      nats:
        server: nats://core.internal:4222
        creds: /etc/insights/core-prod.creds
      interval: 20s
    retention: 24h           # stream buffer depth; the database keeps history
  - id: edge-west            # collector-fed: no scrape block
    retention: 6h

The list is YAML-only and replaces the flat single-system config rather than sitting beside it: systems: together with a non-default system.id, a sys.* connection, the simulator, or the indexer turned off is a config error. A single-system node keeps the flat system:/sys:/scraper:/sink: config. db.memory-limit is per instance, so three systems at 4GiB can use 12GiB.

A system whose database will not open (a corrupt file, a bad path) is logged, skipped, and retried every minute while the other systems serve; a node where no database opens at all exits non-zero. The web UI serves each system under /systems/<id>/ and lists them on a Systems page. See the systems configuration reference for every per-entry key.

Federated

To monitor NATS systems at several sites, run Insights at each site and connect the sites to a central NATS server over leafnodes. One web UI then shows every site. Each site runs either a full Insights node or a collector that only scrapes. See Federated Deployment.

NATS Authentication

All NATS connections support the same authentication options:

Method	Flags	Description
Credentials file	`--*.creds /path/to/file.creds`	NKey + JWT bundled file
NATS context	`--*.context my-context`	Named context from `nats context`
Basic auth	`--.user X --.password Y`	Username/password
NKey	`--*.nkey SEED`	NKey seed string
JWT	`--*.jwt TOKEN`	User JWT string

Replace * with the connection prefix: nats, sys, indexer.nats, or scraper.nats. In YAML, a multi-system entry takes the same keys under systems[].scrape.nats. For the top-level connection, every subcommand (serve, db, checks, ops, config, system, node, web, mcp, http, prometheus) uses the unprefixed nats-CLI-style flags: --server, --creds, --context, --user, --tls-ca-cert, and so on; the --nats.* names remain as hidden aliases.

TLS

--sys.tls-cert /path/to/cert.pem \
--sys.tls-key /path/to/key.pem \
--sys.tls-ca-cert /path/to/ca.pem

Use --*.tls-first when the server requires TLS handshake before the NATS protocol.

SOCKS Proxy

Route NATS connections through a SOCKS proxy:

--sys.socks-proxy socks5://proxy:1080

System Account Permissions

When you give the sys connection a dedicated, restricted user instead of full system-account credentials, grant the subjects below. The * in $SYS.REQ.SERVER.*.<endpoint> matches both the PING broadcast token and a concrete server-id, so it covers the default broadcast scrape and the per-server requests Insights issues when a scrape filter excludes part of the deployment.

The endpoint token is enumerated deliberately. Do not collapse it to $SYS.REQ.SERVER.*.*. That wildcard would also grant the administrative RELOAD, KICK, and LDM actions (reload server config, disconnect a client, drain a server into lame-duck mode), which Insights does not use and which turn a read-only monitoring credential into one that can disrupt the deployment.

Insights publishes to these request subjects (request/reply; replies arrive on _INBOX, below):

Subject	Purpose
`$SYS.REQ.SERVER.PING`	Server discovery
`$SYS.REQ.SERVER.*.VARZ`	Server metrics
`$SYS.REQ.SERVER.*.CONNZ`	Connections
`$SYS.REQ.SERVER.*.SUBSZ`	Subscriptions
`$SYS.REQ.SERVER.*.ROUTEZ`	Routes
`$SYS.REQ.SERVER.*.GATEWAYZ`	Gateways
`$SYS.REQ.SERVER.*.LEAFZ`	Leaf nodes
`$SYS.REQ.SERVER.*.ACCOUNTZ`	Per-server account details
`$SYS.REQ.SERVER.*.JSZ`	JetStream
`$SYS.REQ.SERVER.*.HEALTHZ`	Health
`$SYS.REQ.SERVER.*.RAFTZ`	Raft groups
`$SYS.REQ.SERVER.*.IPQUEUESZ`	Internal queues
`$SYS.REQ.ACCOUNT.PING.STATZ`	Account stats

Insights subscribes to these event subjects:

Subject	Purpose
`$SYS.ACCOUNT.*.DISCONNECT`	Real-time client disconnect events (incl. mid-session auth failures/timeouts)
`$SYS.SERVER.*.CLIENT.AUTH.ERR`	Real-time connect-time authentication rejections (not captured by `DISCONNECT`)
`_INBOX.>`	Replies to the request subjects above

The following publish subjects are optional. Omitting them only disables on-demand server diagnostics:

Subject	Purpose
`$SYS.REQ.SERVER.*.PROFILEZ`	On-demand server profiling
`$SYS.REQ.SERVER.*.EXPVARZ`	On-demand server runtime variables

The two subscribe subjects feed the realtime events view: DISCONNECT surfaces clients that drop after connecting (including sessions whose credentials expire or fail re-authentication), while CLIENT.AUTH.ERR surfaces clients rejected during the connect handshake, which never reach a connected state and so produce no DISCONNECT. Both are subscribed by Insights directly; neither is consumed by the scraper. Omit CLIENT.AUTH.ERR if you don't need connect-time auth-failure visibility.

_INBOX.> is subscribe-only: Insights is the requester, so the server publishes replies to its inbox, so no publish permission on _INBOX is needed. No $SRV.> or scrape-control permissions are required: the scraper's on-demand control service runs on the Insights connection, not the monitored system.

Server config

In a server configuration file, attach the permissions to the system-account user:

accounts {
  SYS {
    users = [
      {
        # the monitoring user's public nkey (U…)
        nkey: UAINSIGHTSMONITOREXAMPLEPUBLICUSERNKEY00000000000000000
        permissions {
          publish {
            allow = [
              "$SYS.REQ.SERVER.PING"
              "$SYS.REQ.SERVER.*.VARZ"
              "$SYS.REQ.SERVER.*.CONNZ"
              "$SYS.REQ.SERVER.*.SUBSZ"
              "$SYS.REQ.SERVER.*.ROUTEZ"
              "$SYS.REQ.SERVER.*.GATEWAYZ"
              "$SYS.REQ.SERVER.*.LEAFZ"
              "$SYS.REQ.SERVER.*.ACCOUNTZ"
              "$SYS.REQ.SERVER.*.JSZ"
              "$SYS.REQ.SERVER.*.HEALTHZ"
              "$SYS.REQ.SERVER.*.RAFTZ"
              "$SYS.REQ.SERVER.*.IPQUEUESZ"
              "$SYS.REQ.ACCOUNT.PING.STATZ"
            ]
          }
          subscribe {
            allow = [
              "$SYS.ACCOUNT.*.DISCONNECT"
              "$SYS.SERVER.*.CLIENT.AUTH.ERR"
              "_INBOX.>"
            ]
          }
        }
      }
    ]
  }
}
system_account: SYS

Decentralized JWT (`nats auth`)

With operator/account JWTs, attach the permissions either to a scoped signing key (a reusable template, so every user issued under it inherits the same grants) or directly to the user.

Scoped signing key (define the template once on the account, then issue users under it):

nats auth account keys add SYS insights-monitor \
  --pub-allow '$SYS.REQ.SERVER.PING' \
  --pub-allow '$SYS.REQ.SERVER.*.VARZ' \
  --pub-allow '$SYS.REQ.SERVER.*.CONNZ' \
  --pub-allow '$SYS.REQ.SERVER.*.SUBSZ' \
  --pub-allow '$SYS.REQ.SERVER.*.ROUTEZ' \
  --pub-allow '$SYS.REQ.SERVER.*.GATEWAYZ' \
  --pub-allow '$SYS.REQ.SERVER.*.LEAFZ' \
  --pub-allow '$SYS.REQ.SERVER.*.ACCOUNTZ' \
  --pub-allow '$SYS.REQ.SERVER.*.JSZ' \
  --pub-allow '$SYS.REQ.SERVER.*.HEALTHZ' \
  --pub-allow '$SYS.REQ.SERVER.*.RAFTZ' \
  --pub-allow '$SYS.REQ.SERVER.*.IPQUEUESZ' \
  --pub-allow '$SYS.REQ.ACCOUNT.PING.STATZ' \
  --sub-allow '$SYS.ACCOUNT.*.DISCONNECT' \
  --sub-allow '$SYS.SERVER.*.CLIENT.AUTH.ERR' \
  --sub-allow '_INBOX.>'

# issue a user under that scoped key (it inherits the template) and write its creds.
# <SIGNING_KEY> is the insights-monitor key from `nats auth account keys ls SYS`.
nats auth user add insights SYS --key <SIGNING_KEY> --defaults
nats auth user credential insights.creds insights SYS

Directly on the user (the same --pub-allow/--sub-allow flags, on user add):

nats auth user add insights SYS \
  --pub-allow '$SYS.REQ.SERVER.PING' \
  --pub-allow '$SYS.REQ.SERVER.*.VARZ' \
  --pub-allow '$SYS.REQ.SERVER.*.CONNZ' \
  --pub-allow '$SYS.REQ.SERVER.*.SUBSZ' \
  --pub-allow '$SYS.REQ.SERVER.*.ROUTEZ' \
  --pub-allow '$SYS.REQ.SERVER.*.GATEWAYZ' \
  --pub-allow '$SYS.REQ.SERVER.*.LEAFZ' \
  --pub-allow '$SYS.REQ.SERVER.*.ACCOUNTZ' \
  --pub-allow '$SYS.REQ.SERVER.*.JSZ' \
  --pub-allow '$SYS.REQ.SERVER.*.HEALTHZ' \
  --pub-allow '$SYS.REQ.SERVER.*.RAFTZ' \
  --pub-allow '$SYS.REQ.SERVER.*.IPQUEUESZ' \
  --pub-allow '$SYS.REQ.ACCOUNT.PING.STATZ' \
  --sub-allow '$SYS.ACCOUNT.*.DISCONNECT' \
  --sub-allow '$SYS.SERVER.*.CLIENT.AUTH.ERR' \
  --sub-allow '_INBOX.>' \
  --defaults
nats auth user credential insights.creds insights SYS

Add --pub-allow '$SYS.REQ.SERVER.*.PROFILEZ' --pub-allow '$SYS.REQ.SERVER.*.EXPVARZ' (or the matching allow entries in the server config) to enable the optional on-demand diagnostics.

Scraper Tuning

Interval and Timeout

--scraper.interval 1m     # Time between scrape cycles (default: 1m)
--scraper.timeout 30s     # Per-request timeout (default: 30s)

Indexer Tuning

Retention

Limit how much historical data is kept in DuckDB:

--db.retention.duration 24h  # Keep last 24 hours (default: 768h)
--db.retention.interval 10m  # Minimum time between sweeps (default: 10m)

Stream Position

Control where the indexer starts consuming:

--indexer.stream-latest      # Start from the latest message
--indexer.stream-epoch 500   # Start from a specific epoch number

Filtering and Downsampling

Process only specific endpoint types or reduce resolution:

--indexer.stream-filters varz,jsz,connz   # Only process these endpoints
--indexer.stream-interval 60s             # Skip epochs closer than 60s apart

Version Checker

Periodically checks GitHub for a newer release and notifies via a log message and a badge in the web UI. Does not download or replace the binary automatically. insights-collector and insights web take the same settings. See Version Updates for details.

--updater \
--updater.interval 1h

Official release binaries and container images are stamped with a version string; the checker only runs when a real version is present (binaries reporting dev are skipped). See Version Updates for how the check works.

Web Server

--web.hostname 0.0.0.0    # Bind to all interfaces (default: 127.0.0.1)
--web.port 8080            # Bind port (default: 8080)

Monitoring with Prometheus & Grafana

Insights exposes check failure metrics via a Prometheus-compatible endpoint, enabled by default on 127.0.0.1:9091/metrics.

--prometheus               # Enable/disable the endpoint (default: true)
--prometheus.hostname      # Bind hostname (default: 127.0.0.1)
--prometheus.port          # Bind port (default: 9091)

For full setup instructions including Prometheus scrape configuration, Grafana dashboard import, and alerting rules, see the Grafana guide.

Environment Variables

Every flag has a corresponding environment variable with the INSIGHTS_ prefix. Nested configs add their prefix:

Flag	Environment Variable
`--server`	`INSIGHTS_NATS_SERVER`
`--sys.creds`	`INSIGHTS_SYS_CREDS`
`--data-dir`	`INSIGHTS_DATA_DIR`
`--scraper.interval`	`INSIGHTS_SCRAPER_INTERVAL`
`--db.retention.duration`	`INSIGHTS_DB_RETENTION_DURATION`
`--web.port`	`INSIGHTS_WEB_PORT`
`--sink.embed`	`INSIGHTS_SINK_EMBED`
`--simulator`	`INSIGHTS_SIMULATOR`
`--log-level`	`INSIGHTS_LOG_LEVEL`

Config File

Pass a YAML config file with --config:

./insights --config /etc/insights/config.yaml

See config.example.yaml for all options. Configuration priority (highest wins):

CLI flags
Environment variables
Config file
Defaults

Validate a config file before a rollout. insights config check loads exactly what serve would and reports what it would reject, without opening a database, starting a listener, or connecting to anything:

./insights config check --config /etc/insights/config.yaml

Containers and Supervisors

Container images

The Linux images run as UID and GID 1000, not root, and carry a pre-created /var/lib/insights owned by that user. Point --data-dir there:

docker run -d --restart unless-stopped \
  -p 8080:8080 \
  -v insights-data:/var/lib/insights \
  -v /path/to/sys.creds:/etc/insights/sys.creds:ro \
  registry.synadia.io/insights:1.2.0 \
  --sys.server nats://your-nats:4222 \
  --sys.creds /etc/insights/sys.creds \
  --data-dir /var/lib/insights \
  --web.hostname 0.0.0.0

A named volume mounted at /var/lib/insights inherits that ownership. A named volume mounted anywhere else, a bind mount, and a Kubernetes PersistentVolumeClaim do not, and startup fails with permission denied (for example write identity.json: ... permission denied, or create data dir: ... permission denied for a directory below the mount). Make the directory writable by UID 1000:

chown -R 1000:1000 /var/lib/insights   # a bind-mounted host directory

In Kubernetes, set securityContext.fsGroup: 1000 on the pod. The Helm chart runs as UID/GID 1000 and sets fsGroup for you. The images have no shell; attach a debug container rather than exec into one. The collector image (the -collector tag, entrypoint insights-collector) runs as the same user, and needs no data directory: run it as docker run -v /path/to/collector:/etc/insights:ro registry.synadia.io/insights:1.2.0-collector -c /etc/insights/collector.yaml.

Restart policy

Run Insights under a supervisor that restarts it: a Kubernetes Deployment or StatefulSet, a systemd unit with Restart=, or --restart on a container. If DuckDB recovery after a fatal database error cannot complete within 30 seconds, Insights exits with a non-zero status so the supervisor can start a fresh process, rather than keep serving a database that cannot answer. Startup exits non-zero the same way when no database opens at all.

Stopping

Insights shuts down cleanly on SIGTERM or SIGINT: it stops the scrapers and indexers, stops the web and API listeners, and checkpoints each DuckDB database before closing it, so the next start does not replay the write-ahead log. Databases are checkpointed one at a time and each checkpoint is bounded to 5 seconds, so give the supervisor a stop timeout of at least 5 seconds per system; docker stop waits 10 seconds by default and Kubernetes 30. A process killed before shutdown completes replays its write-ahead log on the next start.

Health probe

GET /healthz on the web listener returns 200 with the body ok, unauthenticated. The listener binds only after the databases are built, so a success means Insights itself is serving. Point container, Kubernetes, and load-balancer probes at it rather than at a page route.

The web UI has no built-in authentication. If an authenticating proxy sits in front of it, exclude /healthz from authentication, or the probe gets a login redirect.

Example: Production Deployment

A persistent deployment monitoring a real NATS system with all features enabled:

# /etc/insights/config.yaml
log-level: info

data-dir: /data/insights

sys:
  server: nats://nats.internal:4222
  creds: /etc/insights/sys.creds

web:
  hostname: 0.0.0.0
  port: 8080
  session-seed: "your-stable-secret"

db:
  retention:
    duration: 72h

scraper:
  interval: 20s

updater:
  enabled: true
  interval: 1h

./insights --config /etc/insights/config.yaml

Subsystems​

Deployment Topologies​

Demo (Simulator)​

All-in-One (Ephemeral)​

All-in-One (Persistent)​

External NATS Sink​

Stream shape​

Separate NATS connections​

Sink connection permissions​

Server config​

Decentralized JWT (nats auth)​

Without the Web UI​

Indexer-Only​

Split Web Tier​

Multiple Systems on One Node​

Federated​

NATS Authentication​

TLS​

SOCKS Proxy​

System Account Permissions​

Server config​

Decentralized JWT (nats auth)​

Scraper Tuning​

Interval and Timeout​

Indexer Tuning​

Retention​

Stream Position​

Filtering and Downsampling​

Version Checker​

Web Server​

Monitoring with Prometheus & Grafana​

Environment Variables​

Config File​

Containers and Supervisors​

Container images​

Restart policy​

Stopping​

Health probe​

Example: Production Deployment​

Subsystems

Deployment Topologies

Demo (Simulator)

All-in-One (Ephemeral)

All-in-One (Persistent)

External NATS Sink

Stream shape

Separate NATS connections

Sink connection permissions

Server config

Decentralized JWT (`nats auth`)

Without the Web UI

Indexer-Only

Split Web Tier

Multiple Systems on One Node

Federated

NATS Authentication

TLS

SOCKS Proxy

System Account Permissions

Server config

Decentralized JWT (`nats auth`)

Scraper Tuning

Interval and Timeout

Indexer Tuning

Retention

Stream Position

Filtering and Downsampling

Version Checker

Web Server

Monitoring with Prometheus & Grafana

Environment Variables

Config File

Containers and Supervisors

Container images

Restart policy

Stopping

Health probe

Example: Production Deployment