Troubleshooting

Common problems and how to diagnose them. When you're investigating an issue, check the server logs first. Insights logs every significant event and lifecycle decision at info or warn. Start the server with --log-level debug for finer-grained output.

Scrape Failures

The scraper connects to the target NATS system using the --sys.* credentials and publishes collected monitoring data to the indexer over NATS. A failure anywhere in that pipeline surfaces in the logs with the word scraper or scrape.

Symptoms

• Log line scraper setup failed, retrying repeats at growing backoff intervals.
• Web UI shows a stale "Last epoch" timestamp.

Common causes

What the retry loop does

When setup fails, the scraper retries with exponential backoff up to a maximum interval. Failures don't crash the process. Insights keeps the web UI, API, and indexer running so queries against historical data still work.

License Validation Errors

Trial builds validate the license JWT at startup (skipped in simulator mode and on a node that runs neither the indexer nor the scraper). Other builds require no license.

no license token provided. Neither --license.token nor --license.file was supplied. Set one of them. For evaluation, use --simulator instead.

invalid license: token is malformed. The string you provided isn't a valid JWT. Copy-paste the token again, with no stray whitespace or line breaks.

license does not include an Insights entitlement. The JWT is signed correctly but it wasn't issued for Insights. Confirm the license was provisioned by Synadia specifically for Insights (the entitlement is called insights).

scraper disabled: license expired. The license is past its expiry. Insights still starts and serves the data already collected, but does not scrape. Check expires_at logged at startup, renew by contacting Synadia at https://www.synadia.com/contact, and restart with the new token.

Clock skew. JWT validation rejects tokens issued in the future. If the host clock is wrong, run NTP sync and try again.

Notification Endpoint Errors

Insights validates every configured notification endpoint at startup. An endpoint with a missing or malformed URL, a missing or duplicate name, an unknown auth type, a subscription or ignore pattern that matches no check code, or a systems entry naming a system the node doesn't own fails startup by design: a misconfigured endpoint that would silently never deliver is caught up front, not at the first alert. A pattern in the uppercase, underscored form (SERVER_001) fails with its current spelling in the error. Correct or remove the offending entry under notifications: in your config and restart; insights config check runs the same validation without starting the node.

Notifications Not Arriving

• Open the endpoint on the system's Notifications page. The delivery log records each delivery and its error, and the Test button sends a synthetic alert and reports the result inline.
• webhook delivery failed warnings in the logs carry the endpoint id, URL, and error. A failed batch is retried once a minute until the receiver accepts it, 4xx responses included, and newer transitions wait behind it.
• Check the endpoint's subscriptions: an empty list delivers nothing, and a subscription to a check listed in disabled-checks is marked disabled and never delivers.
• Check the endpoint's systems: a scoped endpoint delivers only for the systems it names, and appears only on those systems' pages.
• Webhook endpoints receive transitions only. A finding that was already firing when the endpoint was added isn't sent until it changes state; Alertmanager endpoints receive the full active set every minute.

See Notifications for how delivery and retries work.

Disk and Memory Issues

Insights stores historical data in DuckDB (<data-dir>/insights.db for a single system, one <data-dir>/systems/<id>.db per system on a multi-system node) and, with the embedded sink, uses JetStream streams under <data-dir>/nats/ as the scrape message bus.

Disk growth

Historical data grows roughly linearly with the scrape interval and the size of the monitored system. The retention sweeper deletes epochs older than --db.retention.duration on each sweep, but the database file isn't physically compacted. DuckDB reuses freed blocks for new writes, so the file size is a high-water mark, not the current footprint.

• Set --db.retention.duration (default 768h, 32 days; 0 disables retention). For example, --db.retention.duration 168h keeps a week of history.
• --db.retention.interval is the minimum time between sweeps. A sweep runs on the indexer right after an epoch commit, never alongside one; its cost is dominated by scanning the retained data rather than by how much it deletes, so a larger interval is cheaper.
• To cap growth, stop the server and restart with a lower --db.retention.duration; the next sweep prunes the older epochs. Pruning frees blocks for reuse but does not shrink the file on disk. DuckDB holds the freed space for new writes rather than returning it to the OS.
• To reclaim disk space immediately, take a backup with insights ops backup save (it writes a freshly built file on the machine running the command), stop Insights, put that file in place of the system's database file, and start it again.

Memory pressure

• --db.memory-limit caps DuckDB's in-memory buffer (default 4GiB). On a shared host, lower it so the OS OOM-killer doesn't target Insights. On a multi-system node each system has its own DuckDB pool (systems[].db.memory-limit), so the process can use up to the sum of them.
• Large ad-hoc queries (especially unbounded SELECT * FROM against high-cardinality views like hx.conns) can blow past the limit. Add a WHERE epoch > ... filter to narrow the scan.
• Backups use no server disk or memory beyond a table scan: insights ops backup save streams each table to the client, which builds the file locally. Plan for the database's size on whichever host runs the command, not on the server.

Missing Data

If the web UI reports fewer epochs than expected, or gaps appear in sparklines:

• Read the chart first. A solid line joins consecutive measurements. A dashed span crosses epochs with no measurement and is interpolated between the samples on either side, so it is not data and the hover readout omits it. A lone dot is a sample with no measured neighbour on either side. A rate cell reading as unavailable means there was no sample in the previous epoch to take a rate from, which is what a just-restarted server looks like.
• Check retention: epochs older than --db.retention.duration are intentionally deleted.
• Check for scraper downtime in the logs. Each gap in the timeline corresponds to a stretch where no scrape cycle completed.
• Verify the scraper is still advancing epochs: the "Last epoch" timestamp in the web UI should stay close to the scrape interval. A growing gap means no new epoch has been processed.
• If the scraper recently reconnected, it resumes from the latest available epoch. Historical data during the outage is not backfilled.

A Server Stops Reporting

An epoch can commit without every server in it. A server that misses the scraper's window for an endpoint contributes nothing for that endpoint, and system, cluster, and account totals for that epoch are summed without it, so they read low rather than blank.

Symptoms

• A server-065 (Server Not Reporting) finding names a server that answered no VARZ while its route peers still reported a route to it.
• The server's detail page shows a Not reporting at this epoch banner.
• On the Prometheus endpoint, insights_epoch_servers_expected - insights_epoch_servers_observed > 0 or insights_epoch_partial_endpoints > 0 for that system.
• scrape/incomplete warnings in the logs name the partial endpoints, and the indexer logs epoch/servers missing with the servers that disappeared from the previous epoch.

What to check

• The scraper's server filter: a server excluded after it had been scraped also stops reporting while its peers still route to it.
• System-account permissions and request errors on the scrape connection.
• The server itself: if it still answers STATSZ but its other monitoring endpoints stall, capture a goroutine profile from it and look for blocked monitoring handlers.

server-065 needs route evidence, so it never fires for a solo server, a single-member cluster, or a server reachable only over gateways; watch the epoch-completeness metrics for those. See the Checks Reference.

A System Is Degraded

On a node serving several systems, a system whose database won't open (a corrupt file, a bad path, a full disk) doesn't stop the node. It is logged at error as system catalog unavailable, continuing without it, naming the system, the file, and the reason, and the other systems keep indexing and serving. The degraded system keeps its place in the system list: its queries answer with the open error rather than timing out, and it indexes nothing. If this node scrapes it, the scraper keeps filling its stream, bounded by that stream's retention.

Repair the file (or free the disk, or fix the mount). The node retries the open once a minute and logs catalog reopened when it succeeds; the system then indexes the epochs still in its stream. Its finding feed and notification deliveries stay off until the node restarts, so restart once the catalog is back if alerting depends on it. If no system's database opens at all, startup fails with a non-zero exit instead.

Inspecting a Backup

To look at a backup file, serve it with insights ops backup view <file>. It opens the file in DuckDB's read-only mode and starts the web UI over it with no scraper, no indexer, no retention sweep, and no connection to your NATS system, so browsing a backup can't change it.

Don't move a backup into a data directory and start the server over it to browse it. That opens your only copy read-write: it migrates the file to the running binary's schema and leaves the retention sweep live enough to delete the oldest epochs in it.

The viewer refuses a file whose recorded migrations differ from the binary's, naming what differs. Open the file with the Insights version that wrote it.

Web UI Not Loading

A bookmarked link opens the systems list. Pages live under /systems/<id>/…. A path without the system redirects into it only when the node serves exactly one system; otherwise it lands on the systems list. Pick the system and update the bookmark.

Standalone web tier serving empty pages. When you run the web UI as its own tier (insights web), it holds no local state and reaches the backend entirely over NATS. Empty pages or a failure to start almost always mean the backend NATS is unreachable, or system discovery over $INS.ping found no system with a database to query. Verify --server (or --context) points at the NATS a running Insights node serves its API on, and run insights system list with the same flags from the same host; an empty result means nothing on that connection answers.

Slow Queries

If SQL queries via insights db query or the web UI feel slow:

• Add an epoch predicate. Queries that scan all of history are rarely what you want.
• For aggregates, use the pre-joined hx views (hx.servers, hx.conns, hx.streams, and so on) rather than raw _stats tables, unless you need performance-sensitive access.
• If the indexer is saturated, new queries compete with ingest for CPU and memory. Watch for slow ingest cycles in the logs.
• Increase --db.memory-limit if the query plan is spilling to disk (you'll see it in DuckDB's query log).

Result too large (413). A query whose result exceeds --db.query-max-rows (default 100000) is rejected rather than buffered: the request/reply $INS.db.query endpoint returns a 413 telling you to bound the result. Add a LIMIT or an epoch predicate to narrow it, or run the query through insights db query, which uses the streaming path and returns the result incrementally instead of buffering it in one reply (subject to the streaming --db.query-stream.max-rows backstop).