Troubleshooting

Common problems and how to diagnose them. When you're investigating an issue, check the server logs first. Insights logs every significant event and lifecycle decision at info or warn. Start the server with --log-level debug for finer-grained output.

Scrape Failures

The scraper connects to the target NATS system using the --sys.* credentials and publishes collected monitoring data to the indexer over NATS. A failure anywhere in that pipeline surfaces in the logs with the word scraper or scrape.

Symptoms

• Log line scraper setup failed, retrying repeats at growing backoff intervals.
• Web UI shows a stale "Last epoch" timestamp.

Common causes

What the retry loop does

When setup fails, the scraper retries with exponential backoff up to a maximum interval. Failures don't crash the process. Insights keeps the web UI, API, and indexer running so queries against historical data still work.

License Validation Errors

Insights validates the license JWT at startup (skipped in simulator mode).

no license token provided. Neither --license.token nor --license.file was supplied. Set one of them. For evaluation, use --simulator.enabled instead.

invalid license: token is malformed. The string you provided isn't a valid JWT. Copy-paste the token again, with no stray whitespace or line breaks.

license does not include an Insights entitlement. The JWT is signed correctly but it wasn't issued for Insights. Confirm the license was provisioned by Synadia specifically for Insights (the entitlement is called insights).

license check failed: invalid license: token has invalid claims: token is expired. The license is past its expiry. Check expires_at logged at startup, renew by contacting Synadia at https://www.synadia.com/contact, and restart with the new token.

Clock skew. JWT validation rejects tokens issued in the future. If the host clock is wrong, run NTP sync and try again.

Notification Endpoint Errors

Insights validates every configured notification endpoint at startup. An endpoint with a missing or malformed URL, an unknown auth type, or an out-of-range timeout or retry count fails startup by design — a misconfigured endpoint that would silently never deliver is caught up front rather than at the first alert. Correct or remove the offending entry under notifications: in your config and restart.

Disk and Memory Issues

Insights stores historical data in DuckDB (<data-dir>/insights.db) and uses JetStream streams under <data-dir>/nats/ as the scrape message bus.

Disk growth

Historical data grows roughly linearly with the scrape interval and the size of the monitored system. The retention sweeper deletes epochs older than --db.retention.duration on each sweep, but the database file isn't physically compacted. DuckDB reuses freed blocks for new writes, so the file size is a high-water mark, not the current footprint.

• Set --db.retention.duration (default 768h, 32 days; 0 disables retention). For example, --db.retention.duration 168h keeps a week of history.
• --db.retention.interval controls how often the sweep runs. A larger interval smooths out sweep cost.
• To cap growth, stop the server and restart with a lower --db.retention.duration; the next sweep prunes the older epochs. Pruning frees blocks for reuse but does not shrink the file on disk — DuckDB holds the freed space for new writes rather than returning it to the OS.

Memory pressure

• --db.memory-limit caps DuckDB's in-memory buffer (default 4GB). On a shared host, lower it so the OS OOM-killer doesn't target Insights.
• Large ad-hoc queries (especially unbounded SELECT * FROM against high-cardinality tables like hx.connections) can blow past the limit. Add a WHERE epoch > ... filter to narrow the scan.
• Backups temporarily bump up disk use while the snapshot is being prepared. Plan for roughly 2x the current database size during a backup window.

Missing Data

If the web UI reports fewer epochs than expected, or gaps appear in sparklines:

• Check retention: epochs older than --db.retention.duration are intentionally deleted.
• Check for scraper downtime in the logs. Each gap in the timeline corresponds to a stretch where no scrape cycle completed.
• Verify the scraper is still advancing epochs: the "Last epoch" timestamp in the web UI should stay close to the scrape interval. A growing gap means no new epoch has been processed.
• If the scraper recently reconnected, it resumes from the latest available epoch. Historical data during the outage is not backfilled.

Web UI Not Loading

Standalone web tier serving empty pages. When you run the web UI as its own tier (insights web), it holds no local state and reaches the backend entirely over NATS. Empty pages or a failure to start almost always mean the backend NATS is unreachable, or capability discovery over $INS.ops.info failed. Verify --server (or --context) points at a reachable indexer/scraper node.

Slow Queries

If SQL queries via insights query or the web UI feel slow:

• Add an epoch predicate. Queries that scan all of history are rarely what you want.
• For aggregates, use the pre-joined hx views (hx.servers, hx.connections, hx.streams, and so on) rather than raw _stats tables, unless you need performance-sensitive access.
• If the indexer is saturated, new queries compete with ingest for CPU and memory. Watch for slow ingest cycles in the logs.
• Increase --db.memory-limit if the query plan is spilling to disk (you'll see it in DuckDB's query log).

Result too large (413). A query whose result exceeds --db.query-max-rows (default 100000) is rejected rather than buffered: the request/reply $INS.db.query endpoint returns a 413 telling you to bound the result. Add a LIMIT or an epoch predicate to narrow it, or run the query through insights query, which uses the streaming path and returns the result incrementally instead of buffering it in one reply.