Skip to main content

Architecture

Insights is a single Go binary composed of several subsystems that can be enabled or disabled independently. This modular design supports topologies ranging from a single all-in-one process to distributed deployments where scraping, indexing, and serving are handled by separate instances, and one Insights process can monitor several NATS systems at once.

System Architecture​

Systems and Nodes​

Two identifiers organize every deployment:

  • A system is one monitored NATS deployment, named by an operator-assigned id (system.id, default default). Everything that belongs to it (its database, its checks, its scraper controls, its feeds) is addressed under $INS.sys.<id>.*.
  • A node is one Insights process, named by node.id. A node serving one system defaults to that system's id; a node serving several must be named. Process diagnostics (ops.pprof, ops.version, ops.nats.info) are addressed under $INS.node.<id>.*.

For each system it participates in, a node holds one or both sides:

SideWhat it holds
ScrapeThe connection to the monitored system, the scraper, and the JetStream stream it fills
CatalogThe DuckDB database, the indexer, the checks, and the db.* / checks.* query API

Each side binds only the part of $INS.sys.<id>.* it can serve, so a client addresses subjects and never needs to know which process holds what. Every node answers the $INS.ping broadcast with its node id and its roles (scrape, catalog) per system; insights system list and insights node list show what is reachable.

A node configured with the flat system: / sys: / scraper: / sink: sections owns one system. A node configured with a systems: list owns every listed system, and each entry gets its own DuckDB database (<data-dir>/systems/<id>.db), its own JetStream stream, its own indexer, its own check thresholds and disabled checks, and its own API subtree. Nothing is shared between systems: a system whose database fails to open is reported as degraded and retried while the node's other systems keep serving. A systems: node always runs the indexer and cannot run the simulator.

The deployment models differ only in which process fills each system's stream:

ModelScrape sideCatalog side
Outbound scraperThis nodeThis node
Federated with collectorsAn insights-collector process at the siteA central node, whose stream mirrors the collector's
FederatedEach node, for its own systemsEach node, for its own systems

A collector is part of a federated deployment: it runs at a site that can't spare the resources for a database, or where monitoring data should not be kept, and a central node stores and checks what it scrapes. It is a separate DuckDB-free build (insights-collector) that carries no database, web UI, embedded NATS server, or Prometheus client. It publishes to an external NATS and serves the scrape side of the API. The central node creates the mirror itself, asking the collector over $INS.sys.<id>.ops.sink.info for the origin stream's name, subject prefix, JetStream domain, and scrape cadence. See Federated Deployment for the leafnode setup, JetStream domains, and permissions this involves.

Data Pipeline​

1. Scraper​

The scraper connects to a target NATS system using system account credentials and collects data from the $SYS monitoring endpoints at a configurable interval (default: 1 minute). Each collection cycle is called an epoch.

Each epoch starts by discovering servers with $SYS.REQ.SERVER.PING, then issues requests to every server in the deployment and collects responses from multiple monitoring endpoints: server metrics (varz), connections (connz), subscriptions (subsz), routes (routez), JetStream state (jsz), gateways (gatewayz), leaf nodes (leafz), accounts (accountz, accstatz), health (healthz), Raft state (raftz), and internal queues (ipqueuesz). How many servers an epoch expected and how many it reached are recorded on hx.epochs, so an epoch that missed a server does not read as complete.

A scrape filter (scraper.filter) limits which discovered servers receive requests.

2. Message Transport​

Scrape results flow through a NATS connection with JetStream for durable delivery. Each system has its own stream: scrape_<id> for a system this node scrapes, mirror_<id> for one a collector fills. Scrapes are published under $INS.sys.<id>.scrape, so two systems sharing a NATS account never capture each other's data. By default, Insights runs an embedded NATS server in-process, so no external NATS infrastructure is needed for the data pipeline.

For distributed deployments, you can disable the embedded server and point Insights at an external NATS cluster. This decouples the scraper from the indexer and allows multiple instances to share a common data stream.

3. Indexer​

The indexer consumes scrape data from each system's JetStream stream and processes it into that system's DuckDB database, an embedded analytical database. Each epoch produces a time-series snapshot of the entire NATS deployment: every entity's state, configuration, metrics, and relationships.

DuckDB is well-suited for this workload because it handles analytical queries over time-series data efficiently without requiring an external database server.

The indexer supports configurable retention by duration (db.retention.duration, default 768h). It also supports filtering to process only specific endpoint types, and downsampling to reduce storage at the cost of time resolution.

4. Checks​

When an epoch commits, Insights evaluates every check (around 140) and stores what fails in hx.check_findings, in the same transaction as the epoch's data. Operational checks evaluate the state at that epoch; optimization checks evaluate a trailing window ending at it (six hours by default). Together these checks evaluate health, performance, error patterns, resource saturation, configuration consistency, and recent changes.

A findings tracker per system then diffs the epoch's findings against the previous epoch's and publishes each transition (firing, changed, resolved) on $INS.sys.<id>.feed.finding.>, from which webhook and Alertmanager notifications are delivered. See Checks for details on the check framework.

5. Web UI​

The web UI is an HTTP server that presents the indexed data through an interactive dashboard using server-side rendering with real-time updates. It does not read DuckDB directly: every query goes over the NATS query API ($INS.sys.<id>.db.query.stream), even when the web tier runs in the same process as the indexer.

Every page is scoped to one system under /systems/<id>. A node serving one system lands on that system's pages; a node serving several lands on a Systems list, with a Nodes inventory beside it. The web tier discovers the systems on its NATS over $INS.ping and reads each one's capabilities from $INS.sys.<id>.ops.info, so it also lists systems whose catalogs other nodes hold.

Because the UI is just a NATS client, it can also run standalone as a pure client of remote indexer/scraper nodes; see insights web. The same NATS query API backs the insights mcp server and the insights http gateway, so any client that speaks the protocol can supplement or replace the built-in UI.

6. API Layer​

The NATS micro API starts automatically whenever a sink connection is available. It exposes query capabilities over NATS request/reply under each system's $INS.sys.<id>.* subtree and the node's $INS.node.<id>.* subtree, enabling programmatic access to the indexed data from any NATS client. See the API Reference for endpoint details.

Subsystem Independence​

Each subsystem can be toggled independently via configuration flags:

SubsystemFlagDefault
Scraper--scrapertrue
Indexer--indexertrue
Web UI--webtrue
Embedded NATS--sink.embedtrue
Simulator--simulatorfalse

This enables flexible deployment topologies:

  • All-in-one: every subsystem in a single process (the default)
  • Without the web UI: disable the web UI and query via the NATS API
  • Indexer-only: disable the scraper and consume from an existing scrape stream
  • Distributed: use an external NATS cluster to decouple scraping from indexing
  • Federated: run Insights at each site, either a full node or an insights-collector whose data a central node stores, and see every site in one web UI
  • Multi-system: list several systems under systems: on one node

See the Deployment Guide for detailed topology configurations.

Data Storage​

Without a data-dir, Insights creates a temporary directory at startup and logs a warning; its data is lost on restart. Setting the data-dir flag enables persistent storage that survives process restarts. A web-only node and a collector keep no local storage.

The data directory contains:

  • The DuckDB database: insights.db for a single-system node, systems/<id>.db for each entry of a systems: list
  • The embedded NATS server's JetStream storage (nats/), including each system's scrape stream. With an external NATS server, the stream lives there instead
  • identity.json, which records the node and system ids the directory was last run under and the names they resolved to

Because node and system ids name persistent state, insights config check reports state a changed configuration no longer claims, and insights config migrate moves it onto the new ids.

Simulator​

The built-in simulator creates an in-process NATS deployment with multiple clusters, servers, accounts, streams, consumers, and connections generating traffic. It is useful for evaluation, development, and testing. The simulator replaces the need for a target NATS system: the scraper collects from the simulated servers just as it would from a real deployment. It runs on a single-system node only.

Next Steps​