Skip to main content

Insights v0.1.9

A substantial release. The headline additions are outbound notifications (webhook and Alertmanager), streaming query results so a large query can no longer run an instance out of memory, and three new ways to reach the data as a pure client of a remote instance — a standalone web tier, a read-only MCP server, and an HTTP query gateway. Alongside those: targeted scrape filters, a broad accuracy pass over the audit checks, and detail pages that now fail one section at a time instead of silently. Three changes need attention before you upgrade — the TLS CA certificate flag was renamed, the experimental --db.in-memory-staging flag was removed (it's now the only mode), and enabling web TLS now requires an explicit certificate instead of silently self-signing.

Features​

  • Webhook & Alertmanager notifications: Insights can now push alerts when an audit check changes state. Each epoch it diffs check results and delivers the firing/resolved transitions to your endpoints — a webhook type gets the full Alertmanager-compatible payload, an alertmanager type gets a bare alert array for the Alertmanager v2 /api/v2/alerts API. Endpoints support Basic, Bearer, and HMAC (X-Hub-Signature-256) auth, retry with backoff, and keep a delivery log. A new web UI lets you create, edit, subscribe, and send a test delivery; endpoints can also be declared in config. A malformed endpoint now fails startup with a clear error rather than being silently dropped, so you can't believe alerting is on while it never delivers.

    Configure one or both endpoint types — webhook receives the full envelope, alertmanager the bare alert array:

    # config.yaml
    notifications:
      webhook:
        - name: "PagerDuty Relay"
          url: "https://example.com/webhook"
          enabled: true
          auth-type: "bearer"      # "", "basic", "bearer", "hmac"
          auth-token: "…"
          subscriptions: [server-001, JETSTREAM_001]
      alertmanager:
        - name: "local-am"
          url: "http://localhost:9093/api/v2/alerts"
          subscriptions: [server-001]

    You can see a delivery end to end without waiting for a check to change state — run a receiver, start Insights with the config above, and use the Send test button on the endpoint in the Notifications UI (it delivers a synthetic firing alert):

    docker run --rm -p 9093:9093 prom/alertmanager     # a local receiver
    insights serve -c config.yaml                       # start Insights
    # → Notifications UI → "local-am" → Send test
    #   a firing alert appears in Alertmanager at http://localhost:9093

  • Streaming query results — large queries no longer run the instance out of memory: A big SELECT used to buffer every row in memory before replying, so an unbounded query could exhaust memory and kill a running instance regardless of the configured db.memory-limit (which caps DuckDB's query engine, not the buffered reply). This was not hypothetical — a SELECT * FROM check_findings returning roughly 936 million rows took down a production instance. Results are now streamed to the client as windowed Apache Arrow chunks and rendered locally, so neither side ever holds the whole result. Flow control keeps at most db.query-stream.window chunks (default 16) of db.query-stream.chunk-size (default 512 KiB) in flight, so only a bounded amount is buffered on either side regardless of result size, and the scan aborts if the client disappears. A safety limit, db.query-stream.max-rows (default 1,000,000; set 0 to disable), aborts a runaway unscoped scan — scope the query by epoch or add a LIMIT rather than raising it. Chunks can optionally be compressed with db.query-stream.compression (lz4/zstd; none by default, since compression uses additional CPU). The path is efficient enough to be the default, not just a big-result escape hatch.

    insights query uses the streaming path automatically (falling back to the legacy endpoint against an older server). The legacy $INS.db.query endpoint is now capped (db.query-max-rows, default 100000) and rejects an oversized result with a 413 pointing at the streaming endpoint instead of trying to buffer it.

    insights query "SELECT * FROM hx.streams"    # streams; never buffers the full result

  • Run the web tier on its own (insights web): A new subcommand runs only the web UI as a pure client of a remote indexer/scraper node over NATS — no local database, no embedded NATS server, no license to validate. It discovers the backend's capabilities (geo enrichment, realtime feed, scrape interval, retention, license claims) from the instance over a new $INS.ops.info endpoint, so a web-only node needs no local copy of the server config. It exposes only the flags a web tier uses (--web.*, --prometheus.*, --updater.*, and the connection flags).

    insights web --server=nats://node-a:4222 --web.hostname 0.0.0.0 --web.port 8080

  • Read-only MCP server (insights mcp): A new subcommand runs a Model Context Protocol server so an AI assistant can query your instance directly. It exposes ten read-only tools over the database and checks endpoints — query, explain, schemas, tables, columns, macros, info, checks_list, checks_info, and checks_findings — and, like insights web, owns no database: every tool is a thin gateway to the existing $INS.* request/reply API and reuses the server-side read-only guard. It speaks stdio (for a client-spawned subprocess) or a long-running Streamable HTTP service. See the MCP guide.

    insights mcp --server=nats://node-a:4222              # stdio, for an MCP client
    insights mcp --transport=http --http 127.0.0.1:8090   # long-running HTTP service

  • HTTP query gateway (insights http): A new subcommand runs a stateless HTTP→NATS bridge that fronts the read-only query API for clients that speak HTTP rather than NATS — for example Grafana's Infinity data source. It maps POST /db/query, /db/explain, /db/schemas, /db/tables, /db/columns, and /db/macros onto the corresponding $INS.db.* requests and adds GET /healthz; it owns no database, and the read-only guard, row caps, and per-account NATS auth all stay enforced in the underlying handlers. It refuses to start without a bearer token unless you explicitly pass --allow-unauthenticated. See the HTTP gateway reference.

    insights http --server=nats://node-a:4222 --auth-token "$TOKEN" --listen 0.0.0.0:8080

  • Targeted scrape filters: You can now restrict which servers are scraped by exact server name, cluster, or tag, with allow and deny lists (deny wins; a non-empty allow list restricts the set). Excluded servers receive no monitoring requests at all — useful for carving a large system down to the parts you want to observe.

    scraper:
      filter:
        allow:
          clusters: [us-east, us-west]
        deny:
          tags: [edge]

  • Profile a query, not just plan it (db explain --analyze): insights db explain could plan a read-only query with EXPLAIN; --analyze now runs EXPLAIN ANALYZE and reports per-operator runtime timing, so you can see where a slow query spends its time. The read-only guard still applies — the statement is proven read-only before it executes.

    insights db explain --analyze "SELECT name, cpu FROM hx.servers WHERE epoch = (SELECT max(epoch) FROM hx.server_stats)"

Improvements​

  • More accurate audit checks: A broad hardening pass corrected check logic that produced false or duplicate findings. Among the fixes: server version drift is now judged per cluster rather than across the whole system (server-002); gateway-loss detection keys on the (server, remote cluster) link so a gateway reconnect no longer false-fires as a disconnection (CLUSTER_007); several consumer checks were aligned to the upstream NATS monitor contract (fire at >= the operator threshold, treat 0 as disabled), so Insights agrees with nats server check consumer; and duplicate storage-pressure checks that raised two critical findings for the same breach were retired. Findings that key one row per raft group no longer collapse several groups into one. As part of the same pass the checks were re-leveled onto a consistent severity ladder — many that were critical are now warning, and several warning checks are now info — so a check's health-dot color and triage priority may differ from v0.1.8. The checks reference lists the current severity of every check.

  • Detail pages fail one section at a time: Detail pages compose several independently-queried sections. Previously a failed query on initial load was logged and swallowed, leaving that section as a perpetual skeleton or zeroed data as if the page had loaded fine. Each section is now explicitly in one of three states — loaded, still-loading (skeleton), or failed — and a failed section shows a scoped "couldn't load — retry" inline without affecting the rest of the page.

  • Steadier realtime charts: The realtime activity chart no longer visibly jumps as its time window advances. It was switched to a linear value axis (raw epoch milliseconds) so the data-to-pixel mapping stops re-snapping, and the series render as discrete per-second stacked bars, so a quiet second reads as zero instead of a misleading diagonal bridging the gap.

  • Rate cards show a pending hint on a fresh instance: Per-second rate cards (Bytes/s, Msgs/s, Delivered/s) can't compute a rate from the first scrape, so they used to show "0 B/s" — indistinguishable from genuine zero throughput. They now show a "rate begins after next scrape" hint until the first real sample arrives, then switch to the live value.

  • Client auth failures now show in the realtime feed: Connect-time authentication failures (bad credentials, auth timeout) never join an account and so emitted no disconnect event, leaving the realtime "Auth Errors" series empty. Insights now also subscribes to the server auth-error events, so those failures appear in the feed.

  • nats-CLI-style connection flags: The client subcommands (query, db, checks, ops, and the new web, mcp, and http) now take the same connection flags the nats CLI uses — --context, --server, --creds, --user, --tls-ca-cert, and so on — and any subcommand can be pointed at a single -c config.yaml. The old --nats.* flag names keep working as hidden aliases. When -c isn't given, config is auto-discovered from <user-config-dir>/insights/config.yaml, ./insights.yaml, or the $INSIGHTS_CONFIG path.

  • DuckDB 1.5.4: The embedded query engine was updated to DuckDB 1.5.4.

Bug fixes​

  • Crash on certain glibc versions (self-hosted): A bug in a particular glibc version could crash Insights during hostname resolution. The binary now resolves names with Go's built-in resolver instead of the system C library, avoiding the faulty path. (One consequence: hostnames that rely on the system resolver — for example mDNS .local names — no longer resolve; use IPs or DNS.)

  • An account lingering on a former server's page: An account that moved to a different server still appeared on its old server's detail page, because the "accounts on this server" list matched any epoch in the retained window rather than the current one. It's now scoped to the current epoch, consistent with the streams/connections/leafnodes lists on the same page.

  • Services wrongly shown as gone, and swallowed query errors: The services gone-status query was corrected, and an error in that query is now surfaced instead of being hidden — so a service isn't misreported as gone and a failure doesn't pass silently.

  • Column Summary values under the wrong header: A tooltip on the Min/Max summary cells forced them out of the table's column grid, so the values rendered beneath the wrong column headers. Truncation moved to an inner element with a native tooltip, keeping the cells aligned to their columns.

  • Overview search no longer crashes, and unparseable filters do something useful: Typing free text or an option-style filter into an overview page's search box could crash the page; those inputs are now handled. And a filter expression the parser can't understand (for example an unbalanced orders.>) now falls back to a plain substring match instead of silently returning every row as if no filter were set.

Upgrade considerations​

  • Breaking — the TLS CA certificate flag was renamed to --*.tls-ca-cert. --nats.tlsca, --sys.tlsca, and --web.tlsca-cert are now --nats.tls-ca-cert, --sys.tls-ca-cert, --web.tls-ca-cert (and the same on the other NATS surfaces). The NATS_TLS_CA environment variable becomes NATS_TLS_CA_CERT, and the YAML key nats.tlsca becomes nats.tls-ca-cert. There is no alias — update any command line, env var, or config that sets a CA cert.

  • Breaking — enabling web TLS now requires an explicit certificate. Insights no longer generates a self-signed certificate when web TLS is turned on. If you set --web.tls (or web.tls: true), you must also provide --web.tls-cert and --web.tls-key, or startup fails with a clear error instead of serving a self-signed cert no client would trust. TLS is still off by default, so this only affects deployments that had enabled it and relied on the self-signed fallback — terminate TLS at an ingress, or supply a real cert and key.

  • Restricted system-account creds need one more permission. To surface connect-time client auth failures in the realtime feed, Insights now also subscribes to $SYS.SERVER.*.CLIENT.AUTH.ERR. If you scoped the system-account credentials to a permission allow-list, add subscribe on that subject or the Auth Errors series stays empty.

  • --db.in-memory-staging was removed. In-memory epoch staging is now the only mode, so the experimental flag is gone — remove it from any config or command line. Existing on-disk databases drop their stale staging schema automatically on first start after upgrade.

  • The run command is now serve. insights run still works as an alias (and a bare insights still starts the server), but serve is the canonical name going forward.

  • The legacy query endpoint is capped at 100,000 rows. $INS.db.query now rejects a result larger than db.query-max-rows (default 100000) with a 413 rather than buffering it. The insights query CLI streams automatically and is unaffected; if you built tooling directly on $INS.db.query, either keep results under the cap or move to the streaming endpoint $INS.db.query.stream.