Troubleshooting

Common problems and how to diagnose them. When investigating an issue, check the server logs first. Insights logs every significant event and lifecycle decision at info or warn. Start the server with --log-level debug for finer-grained output.

Scrape Failures

The scraper connects to the target NATS system using the --sys.* credentials and publishes collected monitoring data to the indexer over NATS. A failure anywhere in that pipeline surfaces in the logs with the word scraper or scrape.

Symptoms

• Log line scraper setup failed, retrying repeats at growing backoff intervals.
• Web UI shows a stale "Last epoch" timestamp.
• insights_check_epoch_timestamp (Prometheus) stops advancing.

Common causes

Cause	Diagnostic	Fix
Target NATS unreachable	connect to scrape target in logs	Verify --sys.server, firewall rules, TLS cert chain. Test with nats --context=<ctx> server check from the same host.
Invalid system credentials	authorization violation or nkey unknown in logs	Ensure --sys.creds points at a system-account credentials file and that the account has $SYS access on the target.
Scraper already running	scrape already in progress	Previous scrape is still in flight. Either the previous cycle is hung (check target NATS health) or the scrape interval is too aggressive for the system size.
Trial license expired	scraper not reconnecting: license expired	Refresh the trial license JWT. See Licensing & Trial.

What the retry loop does

When setup fails, the scraper retries with exponential backoff up to a maximum interval. Failures do not crash the process. Insights keeps the web UI, API, and indexer running so queries against historical data still work.

License Validation Errors

Trial builds validate the license JWT at startup (skipped in simulator mode and on a node that runs neither the indexer nor the scraper). Other builds require no license.

no license token provided: neither --license.token nor --license.file was supplied. Set one of them. For evaluation, use --simulator instead.

invalid license: token is malformed: the string you provided is not a valid JWT. Copy-paste the token again, ensuring no stray whitespace or line breaks.

license does not include an Insights entitlement: the JWT is signed correctly but it wasn't issued for Insights. Confirm the license was provisioned by Synadia specifically for Insights (the entitlement is called insights).

scraper disabled: license expired: the license has passed its expiry. Insights still starts and serves the data already collected, but does not scrape. Check expires_at logged at startup, contact Synadia for a renewal, and restart with the new license. See Licensing & Trial.

Clock skew: JWT validation rejects tokens issued in the future. If the host clock is wrong, NTP sync and try again.

Disk and Memory Issues

Insights stores historical data in DuckDB (<data-dir>/insights.db for a single system, one <data-dir>/systems/<id>.db per system on a node with several systems) and, with the embedded sink, uses JetStream streams under <data-dir>/nats/ as the scrape message bus.

Disk growth

Historical data grows roughly linearly with the scrape interval and the size of the monitored system. The retention sweeper deletes epochs older than --db.retention.duration on each sweep, but the database file is not physically compacted: DuckDB reuses freed blocks for new writes, so the file size is a high-water mark rather than the current footprint.

• Set --db.retention.duration (default 768h, 32 days; 0 disables retention). For example, --db.retention.duration 168h keeps a week of history.
• --db.retention.interval is the minimum time between sweeps. A sweep runs on the indexer right after an epoch commit, never alongside one; its cost is dominated by scanning the retained data rather than by how much it deletes, so a larger interval is cheaper.
• To reclaim disk space immediately, take a backup with insights ops backup save (it writes a freshly built file on the machine running the command), stop Insights, put that file in place of the system's database file, and start it again.

retention prune statement warnings identify individual statements taking at least 10 seconds. Subscription and IP queue identity pruning first build an orphan-key temp table from retained stats, then delete those identities in batches. The build can scan a large history even when little data expires. For find orphan idents, rows=0 is the driver's affected-row count for temp-table creation, not a count of orphan identities or rows scanned. UI queries and the periodic checkpoint can still compete for database resources during a sweep.

Memory pressure

• --db.memory-limit caps DuckDB's in-memory buffer (default 4GiB). On a shared host, lower it to prevent the OS OOM-killer from targeting Insights. On a node with several systems, each system has its own DuckDB pool (systems[].db.memory-limit), so the process can use up to the sum of them.
• Large ad-hoc queries (especially unbounded SELECT * FROM against high-cardinality views like hx.conns) can exceed the limit. Add a WHERE epoch > ... filter to narrow the scan.
• Backups consume no server disk or memory beyond a table scan: insights ops backup streams each table to the client, which builds the file locally. Plan for the database's size on whichever host runs the command, not on the server.

Missing Data

If the web UI reports fewer epochs than expected, or gaps appear in sparklines:

• Read the chart first. A solid line joins consecutive measurements. A dashed span crosses epochs with no measurement and is interpolated between the samples on either side, so it is not data and the hover readout omits it. A lone dot is a sample with no measured neighbour on either side. A rate cell reading as unavailable means there was no sample in the previous epoch to take a rate from, which is what a just-restarted server looks like.
• Check retention: epochs older than --db.retention.duration are intentionally deleted.
• Check for scraper downtime in the logs. Each gap in the insights_check_epoch_timestamp series corresponds to a stretch where no scrape cycle completed.
• Verify the scraper is still advancing epochs: time() - insights_check_epoch_timestamp on the /metrics endpoint should stay close to the scrape interval. A growing gap means no new epoch has been processed.
• If the scraper recently reconnected, it resumes from the latest available epoch; historical data during the outage is not backfilled.

A Server Stops Reporting

An epoch can commit without every server in it. A server that misses the scraper's window for an endpoint contributes nothing for that endpoint, and system, cluster and account totals for that epoch are summed without it, so they read low rather than blank.

Symptoms

• A server-065 (Server Not Reporting) finding names a server that answered no VARZ while its route peers still reported a route to it.
• The server's detail page shows a Not reporting at this epoch banner.
• insights_epoch_servers_expected - insights_epoch_servers_observed > 0, or insights_epoch_partial_endpoints > 0, for that system.
• scrape/incomplete warnings in the scraper logs name the partial endpoints, and the indexer logs epoch/servers missing with the servers that disappeared from the previous epoch.

What to check

• The scraper's server filter (scraper.filter): a server excluded after it had been scraped also stops reporting while its peers still route to it.
• System-account permissions and request errors on the scrape connection.
• The server itself: if it still answers STATSZ but its other monitoring endpoints stall, capture a goroutine profile from it and look for blocked monitoring handlers.

server-065 needs route evidence, so it never fires for a solo server, a single-member cluster, or a server reachable only over gateways; watch the epoch-completeness metrics for those. See the Checks Reference and Metrics reference.

A System Is Degraded

On a node with several systems, a system whose database will not open (a corrupt file, a bad path, a full disk) does not stop the node. It is logged at error as system catalog unavailable, continuing without it, naming the system, the file, and the reason, and the other systems keep indexing and serving. The degraded system keeps its place in the system list: its queries answer with the open error rather than timing out, and it indexes nothing. If it is scraped by this node, the scraper keeps filling its stream, bounded by that stream's retention.

Repair the file (or free the disk, or fix the mount). The node retries the open once a minute and logs catalog reopened when it succeeds; the system then indexes the epochs still in its stream. Its finding feed and notification deliveries stay off until the node restarts, so restart once the catalog is back if alerting depends on it. If no system's database opens at all, startup fails with a non-zero exit instead.

Inspecting a Backup

To look at a backup file, serve it with insights ops backup view <file>. It opens the file in DuckDB's read-only mode and starts the web UI over it (no scraper, no indexer, no retention sweep, no connection to your NATS system), so browsing a backup cannot change it.

Do not move a backup into a data directory and start the server over it (mv insights-backup-*.db <dir>/insights.db && insights serve --data-dir <dir>). That opens your only copy read-write: it migrates the file to the running binary's schema and leaves the retention sweep live enough to delete the oldest epochs in it.

The viewer refuses a file whose recorded migrations differ from the binary's, naming what differs. A backup carries its own views, macros, and comments, and a read-only handle cannot refresh them, so a mismatched binary would render pages against definitions that no longer mean what it assumes. Open the file with the Insights version that wrote it.

Pages describing the serving process rather than the backed-up system (the Operations page's NATS panel, the node inventory) report the viewer itself. The Operations page offers no scraper controls, since nothing scrapes into a backup, and its Process panel lists the scraper as None. The time controls have no live end to follow: the toolbar reads OFFLINE and pins to the backup's last epoch. Their presets and the charts' resolution follow the cadence the data was scraped at, read from the spacing of the file's epochs since a backup records no interval.

Web UI Not Loading

Symptom	Likely cause	Resolution
Connection refused	--web.hostname binds to 127.0.0.1 by default	For remote access, set --web.hostname 0.0.0.0 or a specific interface.
Port already in use	Another process on --web.port	Change the port or stop the conflicting process.
Blank page, console errors	Stale browser cache after an upgrade	Hard-reload (Shift-click the reload button) or clear cache.
Startup fails with a TLS certificate error	--web.tls enabled without a certificate	Provide --web.tls-cert and --web.tls-key (Insights does not generate a self-signed cert), or disable --web.tls.
401 / redirect loop	Authentication configured but session seed changed	Clear browser cookies for the Insights host. Ensure --web.session-seed is stable across restarts in production.
insights web won't start or serves empty pages	Backend NATS unreachable, or system discovery over $INS.ping found no queryable system	The web tier is a pure client. Verify --server/--context reach a running node that holds a catalog. Run insights system list with the same connection flags from the same host; an empty result means nothing on that connection answers $INS.ping.
A bookmarked link opens the systems list instead of a page	Pages are scoped under /systems/<id>/…; a flat path redirects into the system only when the node serves exactly one	Pick the system from the list, and update the bookmark to its /systems/<id>/… URL.
Startup fails with an invalid notification config	A notifications endpoint has a bad or missing URL, a duplicate or missing name, an unknown auth-type, a subscription pattern that matches no check code, or a systems entry this node does not own	Fix the reported endpoint. Invalid endpoints fail startup by design so alerting is never silently off. insights config check runs the same validation without starting. See Notifications.

Slow Queries

If SQL queries via insights db query or the web UI feel slow:

• Add an epoch predicate. Queries that scan all of history are rarely what you want.
• For aggregates, use the pre-joined hx views (hx.servers, hx.conns, hx.streams, …) rather than raw _stats tables unless you need performance-sensitive access.
• Check insights_epoch_close_duration_seconds and insights_ingest_duration_seconds. If the indexer is already saturated, new queries compete with ingest for CPU and memory.
• Increase --db.memory-limit if the query plan is spilling to disk (visible in DuckDB's query log).

If a query is rejected with a 413, its result exceeded --db.query-max-rows (default 100000) on the non-streaming $INS.db.query endpoint. Add a LIMIT or an epoch predicate, or use the streaming path: insights db query streams automatically and is not subject to that cap (though very large unscoped scans can still hit the streaming --db.query-stream.max-rows backstop).

Notifications Not Arriving

• Open the endpoint on the system's Notifications page. The delivery log records each delivery and its error, and the Test button sends a synthetic alert and reports the result inline.
• webhook delivery failed warnings in the logs carry the endpoint id, URL, and error. A failed batch is retried once a minute until the receiver accepts it, 4xx responses included, and newer transitions wait behind it.
• Check the endpoint's subscriptions: an empty list delivers nothing, and a subscription to a check listed in disabled-checks is marked disabled and never delivers.
• Check the endpoint's systems: a scoped endpoint delivers only for the systems it names, and appears only on those systems' pages.
• Only transitions are delivered to webhook endpoints. A finding that was already firing when the endpoint was added is not sent until it changes state; Alertmanager endpoints receive the full active set every minute.

Updater Not Finding New Versions

The updater compares the running binary's version against the latest release published on GitHub (https://api.github.com/repos/{repo}/releases/latest, with the repository baked into the binary at build time). See Version Updates for the full mechanism.

• Confirm the updater is enabled (--updater). It is enabled by default, but check it wasn't turned off in config.
• Binaries that report their version as dev are skipped by the checker. Use an official release binary or container image to receive update notifications.
• Check the logs for updater entries. A running updater logs updater started with the version it compares, one that cannot run (no release version or repository compiled in) logs the reason at startup, and fetch errors are logged at warn.
• Corporate egress proxies: ensure the host running Insights can reach api.github.com over HTTPS. The updater uses the default Go HTTP client, which honours HTTPS_PROXY.

Getting Support

When filing a support request or GitHub issue, include:

1. Version: insights --version.
2. Identity: the node identity claimed line from the startup logs, which names the node and the systems it serves.
3. Config: relevant portions of your config file with secrets redacted (nats.creds, sys.creds, license.token, license.file, and each notification endpoint's auth-token).
4. Logs: the last few minutes of logs at --log-level debug if the problem is reproducible.
5. Metrics: a snapshot of /metrics if a latency or ingest-rate issue is suspected.
6. Topology: whether you're running against the simulator or a real NATS system, and the deployment topology (all-in-one, without the web UI, external sink, several systems on one node, federated with full nodes or collectors).

For licensing, billing or product bug reports, contact Synadia.