API

Insights exposes a programmatic API over NATS request/reply using NATS micro. The API starts automatically whenever a sink connection is available and provides the same query capabilities that power the web UI.

Any NATS client connected to the same server (or cluster) can issue requests to these endpoints. This enables automation, integration with external tooling, and running without the web UI.

Service Info

The API registers as a NATS micro service named insights with version 0.0.1. You can discover it using the standard micro service discovery subjects ($SRV.INFO, $SRV.PING, etc.).

Subject Hierarchy

All API subjects use the $INS prefix and follow a $INS.{domain}.{action} convention. The table names that domain and action half; on the wire every subject also carries the id of the system or node it addresses, shown in the Space column and described in Canonical Subject Spaces:

Domain	Subject	Space	Description
db	$INS.db.query	system	SQL query execution
db	$INS.db.query.stream	system	Streaming SQL query (windowed Arrow chunks)
db	$INS.db.explain	system	Validate SQL without executing it
db	$INS.db.schemas	system	List schemas with descriptions + counts
db	$INS.db.tables	system	List tables and views in a schema
db	$INS.db.columns	system	List columns of a table or view
db	$INS.db.macros	system	List macros in a schema with signatures
db	$INS.db.backup.stream	system	Stream one table for a client-built backup
ops	$INS.ops.info	system	Instance capability + config discovery
ops	$INS.ops.sizing	system	DuckDB memory_limit / threads sizing
ops	$INS.ops.memory	system	Live DuckDB memory profile (by tag)
ops	$INS.ops.storage	system	Database file, WAL and connection-pool statistics
ops	$INS.ops.sink.info	system	Layout of the stream the system's scrapes are written to
ops	$INS.ops.scraper.status / .toggle / .trigger	system	Scraper state, pause/resume, and an on-demand scrape
ops	$INS.ops.diagnostics.profilez / .expvarz	system	Profile or expvar capture proxied to the monitored NATS servers
ops	$INS.ops.disconnect.history	system	Recent disconnect-event buckets, for replay
ops	$INS.ops.pprof	node	Go runtime profile (goroutine, heap, ...)
ops	$INS.ops.version / .version.check	node	Update availability, and an immediate update check
ops	$INS.ops.nats.info	node	The embedded NATS server's varz
ops	$INS.ops.metrics	node	This node's Prometheus metrics, for federation
checks	$INS.checks.list	system	List checks
checks	$INS.checks.findings	system	Execute a check and return findings
checks	$INS.checks.info	system	Full metadata for a single check
feed	$INS.feed.epoch	system	Broadcast: an epoch's data committed
feed	$INS.feed.finding.…	system	Broadcast: a check started, stopped, or changed severity
feed	$INS.feed.disconnect	system	Broadcast: aggregated disconnect buckets
feed	$INS.feed.memory	system	Broadcast: a memory profile per tick
usage	$INS.usage.record	system	A page view forwarded by the web tier that served it; no reply

An endpoint is registered only on a node that holds the matching side of the system: the db, checks and usage subjects and ops.info, ops.sizing, ops.memory and ops.storage on the node holding the catalog; the scraper, diagnostics, disconnect-history and sink subjects on the node holding the scrape side.

The first token after the id identifies the domain, the second the action. This makes subjects predictable and easy to authorize via NATS subject permissions (e.g., grant $INS.sys.*.db.> for all database operations).

feed is the exception to request/reply: every subject under it is a fire-and-forget broadcast (a complete frame, no reply, no queue group, no delivery guarantee). It is a domain of its own so that a live-updating client can be granted feed.> and nothing else.

feed.finding appends three tokens naming what changed:

$INS.sys.<id>.feed.finding.<status>.<severity>.<code>
$INS.sys.prod.feed.finding.firing.critical.server-001

They are in the subject, not only the payload, so a subscriber filters server-side rather than receiving everything and discarding most of it: feed.finding.firing.> for everything firing, feed.finding.*.critical.> for every critical transition, feed.finding.*.*.server-001 for one check in both directions.

status is firing, resolved, or changed. The third is a check whose severity moved while it stayed up, which happens for checks with a dynamic severity; without it a subscriber would keep showing whichever severity it first saw. severity is the word (a subject reading .2. tells a human nothing), and on a resolved advisory it is the severity the finding held when it went away.

The payload is one finding's transition:

{
  "code": "server-003",
  "entity_type": "server",
  "entity_pk": 42,
  "entity_name": "nats-1",
  "status": "resolved",
  "severity": "warning",
  "starts_at": "2024-01-15T09:12:00Z",
  "ends_at": "2024-01-15T09:30:00Z"
}

prev_severity appears only on a changed advisory, naming the severity left behind; ends_at only on a resolved one. entity_key carries the entity's key for a finding that has one, such as a service name.

starts_at is when the finding first appeared, carried across every epoch it stays up, so a resolve reports how long the condition actually lasted rather than how long ago the last epoch was. Entity names are resolved at transition time and cached, so a resolve still names an entity that has since been deleted.

On start, a node seeds its baseline from the latest epoch already in its catalog, so the first epoch it indexes publishes only what changed since. A catalog with no epoch yet has nothing to seed from: its first epoch establishes the baseline and publishes nothing, otherwise every standing finding would arrive as new. A check turned off with disabled-checks publishes nothing on the feed.

One more subtree is not an endpoint of this service: scrape.>, the namespace the scraper publishes epochs under and the sink stream captures. It lives under $INS so that one system's whole surface is $INS.sys.<id>.>, and stays separable if you want to deny it.

Canonical Subject Spaces

The flat subjects above name only the domain and action half. Every subject on the wire carries an operator-assigned identity token after $INS, in one of two spaces:

Space	Shape	Id source
System	$INS.sys.<system_id>.{domain}.{action}	system.id, or each systems[].id
Node	$INS.node.<node_id>.{domain}.{action}	node.id (default: the served system's id; required on a node serving several systems; the hostname on a node serving none)

Most subjects are system-scoped. They address a resource that belongs to the monitored NATS system (its database, its checks, its scraper): all db.*, all checks.*, all feed.*, scrape.>, ops.info, ops.sizing, ops.memory, ops.storage, ops.sink.info, ops.scraper.*, ops.diagnostics.*, ops.disconnect.history, and usage.record. For example, $INS.sys.prod.db.query queries the system named prod.

Node-scoped subjects describe the Go process itself, independent of any system or database: ops.pprof, ops.version, ops.version.check, ops.nats.info, ops.metrics, and the webhook.delivery.* log. For example, $INS.node.prod-1.ops.pprof profiles the specific process named prod-1, which matters as soon as more than one instance shares a NATS, where an unscoped form would queue-balance to a random one.

No unscoped form is served. $INS.ping is the sole flat subject, and it is a broadcast rather than a queue-balanced endpoint, so several nodes answering it is the mechanism rather than a collision.

Subject authorization scopes naturally: grant $INS.sys.<id>.> for one system's full API, $INS.sys.*.db.> for database access to every system. That prefix is the whole of a system's traffic. The scrape stream it fills ($INS.sys.<id>.scrape.>) is inside it, so a grant written against it costs nothing else.

Discovery: $INS.ping

$INS.ping is the one intentionally flat subject: a broadcast every instance answers (it is a plain subscription, not a queue-balanced micro endpoint). Publish a request and collect replies for a window; each reply reports the instance's node id, binary version, capability snapshot, labels, and the systems it participates in with its role for each:

{
  "node_id": "prod-1",
  "version": "1.2.0",
  "capabilities": {"geo": {...}, "realtime": {...}, "scraper": {...}, "db": {...}, "checks": {...}},
  "systems": [{"id": "prod", "roles": ["scrape", "catalog"], "metadata": {"env": "prod"}}],
  "metadata": {"region": "us-east-1"}
}

capabilities is the ops.info payload of the first system the instance holds a catalog for, and empty on an instance holding none; ask each system's $INS.sys.<id>.ops.info when a node serves several. The top-level metadata is the instance's node.metadata labels. A system's metadata is its system.metadata labels, reported only by the instance holding that system's scrape side. Both are omitted when empty.

roles is per-system, not per-node: scrape means the instance holds the system's scrape side (the connection to the monitored system, the scraper, the stream), catalog means it holds the database and serves the query surface. The standard micro discovery subjects ($SRV.INFO, ...) remain available for micro-level introspection (each instance's micro metadata carries its node_id), but $INS.ping is the supported contract.

A system entry may also carry catalog_error, which means the instance holds that system's catalog but cannot serve it because its database would not open:

{"id": "edge-west", "roles": ["catalog"], "catalog_error": "open database: file is not a valid DuckDB file"}

The catalog role stays, deliberately. The instance really does own that system's catalog and no other instance will answer for it, so requests should still route here. They return 503 with the reason rather than timing out on a subject nobody serves. insights system list prints these below the topology table. Every other system on the instance is unaffected: one system's database failing does not stop the others, and does not stop the node.

The instance retries the failed database once a minute. Repair the underlying fault and it reopens on its own: catalog_error clears, queries start answering, and indexing resumes from wherever the stream still reaches, with no restart and no interruption to the instance's other systems. The system's finding feed and its webhook and Alertmanager deliveries resume only after the node restarts.

HTTP Gateway

The db query subjects are NATS-native, but they are also reachable over plain HTTP through the insights http gateway (see the CLI reference), a stateless, long-running process that translates each HTTP request into the matching $INS.sys.<id>.db.* NATS request and returns the reply verbatim. It exists for HTTP-only clients (a BI/dashboard datasource, a script, another service) that cannot speak NATS. The gateway owns no database and runs no queries itself; the read-only guard, the row cap, and per-account NATS auth all stay enforced inside the $INS.sys.<id>.db.* handlers.

Routes

Method & path	NATS subject	Request body
POST /sys/<id>/db/query	$INS.sys.<id>.db.query	{"sql": "...", "params": []}
POST /sys/<id>/db/explain	$INS.sys.<id>.db.explain	{"sql": "...", "analyze": false}
POST /sys/<id>/db/schemas	$INS.sys.<id>.db.schemas	{} (optional)
POST /sys/<id>/db/tables	$INS.sys.<id>.db.tables	{"schema": "hx"} (optional)
POST /sys/<id>/db/columns	$INS.sys.<id>.db.columns	{"schema": "hx", "table": "servers"} (optional)
POST /sys/<id>/db/macros	$INS.sys.<id>.db.macros	{"schema": "checks"} (optional)
GET /systems	$INS.ping	(none); returns the discovery replies as a JSON array
GET /healthz	(none)	Liveness + NATS connectivity; unauthenticated

Only the read-only db request/reply surface is exposed; the ops, checks, and streaming subjects are not. POST /sys/<id>/db/query honours the Accept header (text/csv for CSV, otherwise JSON), matching $INS.db.query.

System selection

Every db route names its system in the path, positioned exactly where the subject carries it: /sys/<id>/db/query becomes $INS.sys.<id>.db.query. Reading a URL therefore tells you the subject it produces, without knowing anything about the deployment.

There is no default and no pinning. The gateway holds no cached topology and consults nothing to route (the path segment is the subject token), so a request cannot be answered by a system other than the one it named, and adding a second system to a deployment changes nothing for clients already addressing the first.

GET /systems enumerates what is addressable (one $INS.ping round, the same data as insights system list --format json); it is unscoped for the same reason $INS.ping is, being how a client finds out what the scoped paths are. A dashboard populates a system template variable from it and interpolates that into the path.

Because the gateway holds no list of valid ids, it cannot report one as wrong, only that nothing answered for it. A mistyped id and a stopped indexer are indistinguishable from the gateway's position, and the 503 says so rather than guessing which: no responder for system "prd" (GET /systems lists the reachable systems). Naming the valid ids inline would need a cached set, and a cached set is wrong in both directions: it rejects a system that came up a moment ago and accepts one that has just gone away.

A system id must be a subject token (letters, digits, _, -); anything else is a 400 before the request reaches NATS, since the segment is spliced into a subject verbatim.

Authentication

Every route except /healthz requires a bearer token (Authorization: Bearer <token>), set via --auth-token. The gateway speaks to NATS with the single identity it was started with, so multi-tenant separation is one gateway (and one datasource) per NATS account. It refuses to start without a token unless --allow-unauthenticated is given.

Error mapping

An upstream micro error keeps its HTTP-style status, so failures stay actionable rather than collapsing to a generic 500: a row-cap overflow surfaces as 413, invalid SQL as 400. Transport failures are mapped too: a missing responder to 503, an upstream timeout to 504, anything else to 502. (This is why the gateway forwards the bounded $INS.db.query rather than the streaming endpoint, which would surface errors mid-read after a 200 was already sent.)

Example

# what is addressable
curl -s localhost:8080/systems -H "Authorization: Bearer $TOKEN"

curl -s localhost:8080/sys/prod-us/db/query \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"sql": "SELECT count(*) AS n FROM hx.server_ident"}'

# CSV instead of JSON:
curl -s localhost:8080/sys/prod-us/db/query \
  -H "Authorization: Bearer $TOKEN" -H "Accept: text/csv" \
  -d '{"sql": "SELECT name, cpu FROM hx.servers ORDER BY cpu DESC LIMIT 10"}'

Endpoints

Query: $INS.db.query

Execute arbitrary read-only SQL queries against the DuckDB database.

Read-only enforcement. The database connection is read-write (it is shared with the indexer), so read-only is enforced at the API layer using DuckDB's own parser: the statement is prepared (parsed and bound, never executed) and rejected unless it is a single statement classified as SELECT (which also covers WITH, FROM-first, DESCRIBE, SHOW, and SUMMARIZE). Because DuckDB, not a string heuristic, does the classification, every write/DDL and state-changing statement (INSERT/UPDATE/DELETE/CREATE/DROP/COPY/ATTACH/SET/mutating PRAGMA/…) maps to a non-read-only type and is rejected, comments and quoted identifiers are handled correctly, and a multi-statement payload (SELECT 1; DROP …) fails to prepare. EXPLAIN is rejected too: EXPLAIN ANALYZE executes its wrapped statement (so it can write) yet is indistinguishable by statement type from a plain EXPLAIN, so use the $INS.db.explain endpoint for query plans. No data is scanned during validation.

Request:

{
  "sql": "SELECT * FROM hx.server_ident LIMIT 10",
  "params": []
}

Field	Type	Required	Description
sql	string	yes	SQL query to execute. Must be a read-only SELECT (incl. WITH/DESCRIBE/SHOW/SUMMARIZE)
params	array	no	Positional parameters for parameterized queries

Response (JSON, default):

[
  { "name": "server-1", "cluster": "us-east", "version": "2.10.0" },
  { "name": "server-2", "cluster": "us-west", "version": "2.10.0" }
]

An array of objects, one per row, keyed by column name.

Response (CSV):

Set the Accept: text/csv header to receive results as CSV.

This endpoint buffers the whole result in memory before responding, so it caps the row count (db.query-max-rows, default 100000). A result that would exceed the cap is rejected with 413. Add a LIMIT, or use the streaming endpoint below, which streams the result instead of buffering it.

Errors:

Code	Condition
400	Invalid JSON, missing SQL, non-read-only query, or SQL that fails to parse, bind or run
408	The query ran past db.query-timeout, or the server is shutting down
413	Result exceeds db.query-max-rows; add a LIMIT or use .stream
503	This node holds the system's catalog but its database is not open
500	Internal server error

Example using nats CLI, for the system with id prod:

nats req '$INS.sys.prod.db.query' '{"sql": "SELECT count(*) as n FROM hx.server_ident"}'

Streaming Query: $INS.db.query.stream

Streams an unbounded read-only result instead of buffering it. The result is pushed to the request's reply inbox as ordered Apache Arrow IPC chunks; the insights db query CLI decodes them and renders CSV/JSON locally. Neither the server nor the client materializes the whole result, so this is the path for large exports (the insights db query command uses it automatically, falling back to $INS.db.query when no streaming responder is present).

The same read-only guard applies. Flow control works by acknowledgement: the client acknowledges chunks and the server keeps at most db.query-stream.window chunks in flight, so a slow client throttles the server (and a client that disappears aborts the scan) rather than dropping data or exhausting memory. Because the stream stays memory-bounded, there is no buffering row cap, but a safety limit, db.query-stream.max-rows (default 1,000,000; 0 disables), aborts a runaway unscoped scan. Scope the query by epoch or add a LIMIT rather than raising it.

Canonical output (differs from $INS.db.query). Because values are rendered from Arrow rather than the SQL driver's Go types, formatting is canonical and not byte-compatible with $INS.db.query: timestamps are RFC3339 in CSV as well as JSON ($INS.db.query CSV prints Go's time format), NULL is an empty CSV field / JSON null, BLOB is base64 ($INS.db.query CSV prints a decimal byte array), and DECIMAL/HUGEINT are canonical decimal strings (minimal form, trailing zeros trimmed).

Chunks may optionally be compressed (db.query-stream.compression: none|lz4|zstd, default none); Arrow IPC readers decompress transparently, so clients need no change either way.

This endpoint is only present in builds compiled with the duckdb_arrow tag (the released binaries are). Builds without it expose only $INS.db.query.

Explain: $INS.db.explain

Plan a read-only SQL query, and return the query plan. By default the query is bound and planned via DuckDB EXPLAIN, catching unknown tables/columns, type errors, and syntax, but no data is scanned. Useful for inspecting a query's plan or checking it before issuing it against the query endpoint. Set analyze: true to run EXPLAIN ANALYZE instead, which executes the query to add per-operator runtime timing to the plan. This is safe because the server first validates the statement is read-only (a read-only query only reads, so it cannot smuggle a write through EXPLAIN ANALYZE); an analyze run is bound by the same server query timeout as the query endpoint.

Request:

{
  "sql": "SELECT name, cluster FROM hx.server_ident",
  "analyze": false
}

Field	Type	Required	Description
sql	string	yes	SQL query to plan. Must be read-only
analyze	boolean	no	Execute the query with EXPLAIN ANALYZE and include runtime timing (default false)

Response:

{
  "valid": true,
  "plan": [
    { "explain_key": "physical_plan", "explain_value": "┌───────────────────────────┐\n│         PROJECTION        │\n..." }
  ]
}

valid is always true on success. plan carries the DuckDB EXPLAIN output rows (typically a single physical_plan row whose explain_value is the plan tree). Validation or bind/parse failures are returned as 400 errors carrying the message.

Errors:

Code	Condition
400	Invalid JSON, missing SQL, non-read-only query, or SQL that fails to bind/parse
408	An analyze run exceeded db.query-timeout
503	This node holds the system's catalog but its database is not open

Example using nats CLI:

nats req '$INS.sys.prod.db.explain' '{"sql": "SELECT count(*) FROM hx.server_ident"}'

Schema discovery

The database holds more than the hx entity schema: the geo-IP enrichment table ips lives in main, each check is a table macro in checks (check server-001 is checks.server_001), and the macros behind a finding's supporting context panels are in contexts. Four endpoints let a client discover what is queryable without assuming any single schema, drilling from schemas to tables/macros to columns. Discovery is scoped to the primary catalog and the allowlisted schemas (hx, main, checks, contexts); internal schemas (cache, nk, staging) and any attached catalog (e.g. ops) are excluded. No data is scanned: the endpoints read catalog metadata only.

All four return 503 when this node holds the system's catalog but its database is not open, and 500 (Internal server error) on any other failure.

Schemas: $INS.db.schemas

List the discoverable schemas, each with a curated description and live counts of tables, views, and macros. The entry point for discovery.

Request: Empty payload {}.

Response:

[
  { "schema_name": "checks",   "tables": 0,  "views": 0,  "macros": 176, "description": "Checks framework: one table macro per check code (server-001 is checks.server_001) returning (code, entity, ...) finding tuples, plus the shared helper macros they call." },
  { "schema_name": "contexts", "tables": 0,  "views": 0,  "macros": 35,  "description": "Context macros behind the evidence panels shown with a finding: cluster peer comparisons, raft peers, top streams and similar supporting views." },
  { "schema_name": "hx",       "tables": 40, "views": 13, "macros": 0,   "description": "Historical entity tables (*_ident/*_opts/*_stats) and the pre-joined views ..." },
  { "schema_name": "main",     "tables": 1,  "views": 0,  "macros": 0,   "description": "Default (unqualified) schema. Holds the geo-IP enrichment table `ips`." }
]

nats req '$INS.sys.prod.db.schemas' '{}'

Tables: $INS.db.tables

List the tables and views (column-bearing relations) in a schema, each with its object-level comment. Macros are listed separately via $INS.db.macros.

Request:

Field	Type	Required	Description
schema	string	no	Restrict to one schema. Omit to list across all allowlisted schemas

Response:

[
  { "schema_name": "hx",   "name": "server_stats", "type": "table", "comment": "Server runtime metrics. Written every epoch ..." },
  { "schema_name": "hx",   "name": "servers",      "type": "view",  "comment": "Pre-joined server view ..." },
  { "schema_name": "main", "name": "ips",          "type": "table", "comment": "IP geolocation lookup table ..." }
]

type is table or view. Rows are ordered by schema, type, then name.

nats req '$INS.sys.prod.db.tables' '{"schema": "hx"}'

Columns: $INS.db.columns

Return the columns of a table or view: the entity-level table_comment, plus each column's name, type, and the one-line comment that documents its meaning (gauge/counter, units, FK targets).

Request:

Field	Type	Required	Description
schema	string	no	Restrict to one schema. Omit to list across all allowlisted schemas
table	string	no	Restrict to a single table or view. Omit for every object in scope

Response:

[
  {
    "schema_name": "hx",
    "table_name": "server_stats",
    "table_comment": "Server runtime metrics. Written every epoch. Contains point-in-time counters and gauges.",
    "column_name": "memory",
    "data_type": "BIGINT",
    "comment": "Server process memory usage (bytes)."
  }
]

An array of rows, one per column, ordered by schema, table name, and column index. Each row repeats its object's table_comment. For the hx views, column comments live on the _ident/_opts/_stats base tables; the views carry only table_comment, so read the base tables for per-column meaning.

nats req '$INS.sys.prod.db.columns' '{"schema": "main", "table": "ips"}'

Macros: $INS.db.macros

List the user-defined macros in a schema with their parameter signatures. Macros have no columns, so the signature is the detail needed to invoke them (e.g. the check macros).

Request:

Field	Type	Required	Description
schema	string	no	Restrict to one schema. Omit to list across all allowlisted schemas

Response:

[
  { "schema_name": "checks", "name": "stream_002", "type": "table_macro", "parameters": "epoch_start, epoch_end, p_lag_percent", "comment": null }
]

type distinguishes a scalar macro from a table-returning table_macro. Rows are ordered by schema, then name.

nats req '$INS.sys.prod.db.macros' '{"schema": "checks"}'

Checks List: $INS.checks.list

Returns all checks grouped by category, including configurable parameters.

Request: Empty payload.

Response:

[
  {
    "label": "Server",
    "checks": [
      {
        "code": "server-003",
        "name": "High CPU Usage",
        "description": "Flags servers where per-core CPU usage exceeds the CPU threshold.",
        "scope": "server",
        "optimization": false,
        "params": [
          {
            "name": "cpu_percent",
            "default": 90.0,
            "resolved": 85.0,
            "description": "Per-core CPU usage threshold percentage"
          }
        ]
      }
    ]
  }
]

The resolved value reflects any overrides from check-thresholds configuration.

Each entry also carries disabled, present and true only when the addressed system turned the check off via disabled-checks. A disabled check still runs and still produces findings; it is left out of the health grades and never delivered as a notification.

An entry carries min_server_version when the check reads a monitoring field that older NATS Server releases do not report. Entities on a server below that release are not evaluated, so their absence from the findings is not a pass.

Checks Findings: $INS.checks.findings

Execute a specific check by code over a time window and return matching findings.

Request:

{
  "code": "server-003",
  "time": {
    "duration": "1 hour"
  },
  "page": 1
}

Field	Type	Required	Description
code	string	yes	Check code (e.g., server-003)
time	object	no	Time parameters. duration is a DuckDB interval (e.g., 1 hour, 30 minutes)
page	int	no	Page number for pagination

Response (JSON, default):

{
  "code": "server-003",
  "name": "High CPU Usage",
  "detail_columns": [
    {
      "name": "cpu_percent",
      "type": "DOUBLE",
      "description": "Per-core CPU usage percentage."
    }
  ],
  "rows": [
    {
      "code": "server-003",
      "severity": "warning",
      "entity": "us-east / nats-1",
      "entity_pk": 42,
      "check_name": "High CPU Usage",
      "cpu_percent": 92.3
    }
  ],
  "page_info": {
    "page": 1,
    "page_size": 20,
    "total_rows": 1,
    "total_pages": 1
  }
}

Field	Description
code	The check code that was executed
name	Human-readable check name
dynamic_severity	Present and true when the check computes severity per-row
disabled	Present and true when this system turned the check off: the rows are its real findings, but they are not graded and never delivered as notifications
detail_columns	Per-check extra columns appearing in each row, with name, type, and description
rows	Matching findings, keyed by column name
page_info	Pagination metadata; pages hold 20 rows

name, dynamic_severity, detail_columns and disabled are echoed so clients can render findings without a second round-trip to the $INS.checks.info endpoint.

Response (CSV): Set Accept: text/csv header. The CSV form carries the rows only, so it has no disabled marker; ask $INS.checks.info when that matters.

Errors:

Code	Condition
400	Missing check code or invalid duration
404	Unknown check code
503	This node holds the system's catalog but its database is not open
500	Internal server error

Check Info: $INS.checks.info

Return the full metadata for a single check (description, remediation guidance, scope, severity, parameters, and any detail columns) without executing it.

Request:

{
  "code": "server-003"
}

Field	Type	Required	Description
code	string	yes	Check code (e.g., server-003)

Response:

{
  "code": "server-003",
  "name": "High CPU Usage",
  "description": "Flags servers where per-core CPU usage exceeds the CPU threshold.",
  "remediation": "Identify the workload driving CPU usage using the /pprof/profile debug endpoint. ...",
  "scope": "server",
  "severity": "warning",
  "optimization": false,
  "category": "performance",
  "category_label": "Performance & Latency",
  "params": [
    {
      "name": "cpu_percent",
      "default": 90.0,
      "resolved": 85.0,
      "description": "Per-core CPU usage threshold percentage"
    }
  ],
  "detail_columns": [
    {
      "name": "cpu_percent",
      "type": "DOUBLE",
      "description": "Per-core CPU usage percentage."
    }
  ]
}

Field	Description
code	The check code
name	Human-readable check name
description	What the check detects
remediation	Guidance for resolving findings
scope	Entity scope (e.g., server, cluster, account)
severity	Default severity
dynamic_severity	Present and true when the check computes severity per-row
optimization	Whether this is an optimization (vs. health) check
disabled	Present and true when the addressed system turned the check off in config
min_server_version	Oldest NATS Server release the check can be evaluated against; absent when the check reads nothing version-dependent
lookback_seconds	Fixed lookback window, present only for optimization checks
category	Framework category slug (health, performance, errors, saturation, consistency, change)
category_label	Human-readable framework category label
params	Configurable parameters with default and resolved values
detail_columns	Per-check extra columns appearing in findings rows

The resolved value reflects any overrides from check-thresholds configuration, and disabled reflects disabled-checks.

Errors:

Code	Condition
400	Invalid JSON or missing check code
404	Unknown check code

Backup: $INS.db.backup.stream

Stream one table's rows so a client can build a backup database locally. The server reads and streams; it writes nothing and buffers nothing. insights ops backup save resolves the epoch window once, then issues one request per table and inserts each stream into a DuckDB file it creates; see insights ops backup.

The chunk protocol is the one $INS.db.query.stream uses: Arrow IPC chunks on the reply inbox, windowed acknowledgements, an in-band trailer. Only the request differs.

Request:

{
  "schema": "hx",
  "table": "stream_opts",
  "start_epoch": "2024-01-01T00:00:00Z",
  "end_epoch": "2024-01-02T00:00:00Z"
}

Field	Type	Required	Description
schema	string	yes	hx, or main for the ips table
table	string	yes	Base table name. hx.migrations is not backed up
start_epoch	string (RFC 3339)	no	Inclusive lower bound. Ignored for a table with no epoch column, and for _ident and _opts tables, which are scoped by what the window references rather than by time
end_epoch	string (RFC 3339)	no	Inclusive upper bound. Ignored for a table with no epoch column

The request carries a table reference rather than SQL, and the server builds the query. That is what makes this endpoint exempt from db.query-stream.max-rows: the cap guards operator-supplied queries, and a full-table scan is this endpoint's whole purpose. It is also its own micro endpoint rather than a mode of $INS.db.query.stream, so a backup running for minutes never holds the delivery goroutine db query, MCP, the HTTP gateway and Grafana share.

Errors are reported in-band on the reply inbox as an error trailer, including an unknown system, a table outside the backup set, and a mid-scan failure.

Instance Info: $INS.ops.info

Reports the backend instance's capabilities and configuration so a client (notably a standalone insights web tier) can configure itself without a local copy of the server config. Empty request payload.

Response (JSON):

Field	Type	Description
geo.enabled	bool	Whether IP geolocation enrichment is configured
realtime.enabled	bool	Whether the realtime advisory feed is available
scraper.interval	duration	Scrape cadence (e.g. 1m); drives time-picker presets and gap-fill granularity
db.retention	duration	Data-retention duration (e.g. 768h; 0 = disabled); bounds time-range controls
checks.disabled	array	Check codes this system turned off; omitted when none. A remote web tier reads its own copy from here, so a check disabled on the indexer is marked and ungraded in that UI too
license	object	Parsed license claims; omitted when the instance is unlicensed

The duration fields use Go's duration notation (1m, 768h). On the wire, JSON encodes them as the equivalent integer nanoseconds (1m is 60000000000), so a client decoding the response should expect a number.

Example using nats CLI:

nats req '$INS.sys.prod.ops.info' ''

Sizing: $INS.ops.sizing

Compute a recommended DuckDB memory_limit / threads pair for the live database. The report is computed server-side against the open handle and the host capacity (cores, RAM) reflects the running host the limits will apply to. This backs insights ops sizing.

Request:

{
  "per_thread": 2,
  "threads": 0
}

Field	Type	Required	Description
per_thread	int	no	GiB per thread used in the query budget (DuckDB recommends 1–4). Default 2.
threads	int	no	Threads to size for. 0 (default) recommends min(cores, 4).
measured	object	no	Externally observed profiler peaks (same shape as the response's measured). When set, it takes precedence over the server's own rolling-window snapshot. It is used by insights ops sizing --observe to supply peaks aggregated over a longer window (e.g. across a retention prune).

All fields are optional; an empty payload {} uses the defaults.

Response: a sizing report object with the measured database shape, the host capacity it was computed against, and the derived recommendation:

{
  "db_path": "/data/insights.db",
  "db_size_bytes": 5368709120,
  "goos": "linux",
  "cores": 8,
  "ram_gib": 32,
  "epoch_count": 2880,
  "first_epoch": "2024-01-01T00:00:00Z",
  "last_epoch": "2024-01-03T00:00:00Z",
  "epoch_interval": 60000000000,
  "retention_duration": 2764800000000000,
  "projected_db_size_bytes": 85899345920,
  "projected_epochs": 46080,
  "per_thread": 2,
  "threads_rec": 4,
  "query_gib": 8,
  "write_gib": 4,
  "index_gib": 2,
  "total_gib": 14,
  "host_cap_gib": 19,
  "warnings": ["database size (80.0 GiB, projected at full retention) exceeds host RAM (32 GiB): history-spanning queries will spill to disk and slow"],
  "measured": {
    "window": 3600000000000,
    "samples": 1800,
    "transaction_peak_bytes": 4294967296,
    "index_peak_bytes": 1073741824,
    "metadata_peak_bytes": 1073741824,
    "base_table_peak_bytes": 6442450944,
    "total_peak_bytes": 12884901888
  }
}

total_gib is the recommended memory_limit and threads_rec the recommended threads; the query_gib / write_gib / index_gib breakdown shows how total_gib is built (and is capped at host_cap_gib, 60% of host RAM, when set). write_gib is the largest write transaction (a per-epoch commit or a retention prune); index_gib is the resident ART-index + catalog/block-metadata floor.

epoch_interval (observed median gap between epochs, the scrape cadence) and retention_duration (the configured retention policy, 0 = disabled) are reference context, both durations encoded as integer nanoseconds (e.g. 60000000000 = 1 minute, 2764800000000000 = 768 hours / 32 days). projected_db_size_bytes / projected_epochs forecast the steady-state on-disk size at full retention from observed bytes-per-epoch. It is a disk and cache signal, not the memory floor (the un-spillable floor does not scale with retention). warnings carries advisory caveats: unmeasured (fallback) floors, or a (projected) DB size exceeding host RAM.

measured is present when the memory profiler is enabled and has samples, or when the request supplied its own measured peaks. It carries the peak resident bytes observed under live load and sets the un-spillable floors directly: write_gib from transaction_peak_bytes and index_gib from index_peak_bytes + metadata_peak_bytes (ground truth replaces the conservative fallback). base_table_peak_bytes (buffered table data) is reported for context but is not added to the budget; it lives within memory_limit.

Errors:

Code	Condition
400	Invalid request JSON or per_thread/threads out of range
503	This node holds the system's catalog but its database is not open
500	Internal server error

Memory: $INS.ops.memory

Return the live DuckDB memory profile: current and peak resident bytes per component (duckdb_memory() tag) over the profiler's rolling window. An always-on sampler in the server process records readings every db.memory-profile.interval; this endpoint returns that retained snapshot; it does not sample on demand. The readings are also exported as the insights_duckdb_memory_bytes{tag=...} Prometheus gauge.

Request: empty payload.

Response: when the profiler has samples:

{
  "window": 900000000000,
  "interval": 2000000000,
  "samples": 450,
  "since": "2024-01-15T09:45:00Z",
  "tags": [
    { "tag": "ART_INDEX", "current_bytes": 3670016, "peak_bytes": 3670016, "peak_at": "2024-01-15T09:45:00Z", "spark": [3670016, 3670016, "…"] },
    { "tag": "BASE_TABLE", "current_bytes": 1089208320, "peak_bytes": 1153433600, "peak_at": "2024-01-15T09:58:00Z", "spark": [1010827264, 1023410176, "…32 buckets"] }
  ],
  "current_total_bytes": 1092878336,
  "peak_total_bytes": 2621440000,
  "peak_total_at": "2024-01-15T09:52:00Z",
  "spark": [1014497280, 1027080192, "…32 buckets"]
}

tags is ordered by a fixed canonical layout (un-spillable components first, then spillable), so the order is stable across snapshots rather than shifting with magnitude. window and interval are durations in nanoseconds. Each spark is the value (or total) downsampled to 32 buckets over the window (the max of each bucket, oldest first), which the CLI renders as an ASCII trend line. When the profiler is disabled, samples is 0 and tags is empty (the request still succeeds). The un-spillable memory floor is ART_INDEX + TRANSACTION; BASE_TABLE is the buffer pool (bounded by memory_limit); HASH_TABLE / ORDER_BY are spillable query work.

Pprof: $INS.node.<node>.ops.pprof

Capture a Go runtime profile of the running instance and return it in the reply. The server does not expose the net/http/pprof HTTP handlers, so this is the way to reach the standard runtime/pprof snapshots, most usefully a goroutine dump to see where a process is parked (e.g. a stalled scrape/index loop). This backs insights ops pprof.

Request:

Field	Type	Required	Description
name	string	yes	One of goroutine, heap, allocs, threadcreate, block, mutex, spelled as Go's pprof.Lookup takes it, and as the system-side ops.diagnostics.profilez carries it. The CPU profile and execution trace are not offered: both need a sampling window, which does not fit request/reply.
debug	int	no	runtime/pprof debug level: 0, 1, or 2. 0 (default) returns the binary protobuf for go tool pprof. 1 returns the legacy human-readable text format with aggregated counts. 2 only differs for goroutine, where it emits full per-goroutine stacks (the panic-style dump); for every other profile it is identical to 1.

Response: the raw profile bytes. The Content-Type header is application/octet-stream for debug=0 and text/plain; charset=utf-8 otherwise.

block and mutex need their sampling enabled at startup (runtime.SetBlockProfileRate / runtime.SetMutexProfileFraction). Insights does not enable either. Both add per-event runtime overhead on every blocking operation and mutex contention for the life of the process, a cost not worth paying continuously just to keep the snapshot available on demand, so requesting them today returns a valid but empty profile. Use goroutine at debug=2 to see goroutines parked on a lock; it needs no setup. The whole profile is returned in a single reply, so a profile larger than the NATS max payload fails with a 413 rather than being truncated.

The subject carries the node id, so a profile is always addressed to one named process. Use $INS.ping to list the node ids currently answering.

# goroutine stack dump as text
nats req '$INS.node.prod-1.ops.pprof' '{"name":"goroutine","debug":2}'

# heap profile for go tool pprof
nats req '$INS.node.prod-1.ops.pprof' '{"name":"heap"}' --raw > heap.pprof && go tool pprof heap.pprof

Events

Epoch Committed: $INS.feed.epoch

A fire-and-forget broadcast (not a request/reply endpoint) published by the indexer after an epoch's data is atomically committed and visible in the hx tables. Subscribe to drive live refreshes: when it fires, a live view can re-run its queries against the new epoch.

Payload:

{ "node_id": "indexer-002", "epoch": "2024-01-15T09:30:00Z" }

Field	Type	Description
node_id	string	Stable ID of the node that committed the catalog
epoch	string (RFC 3339)	Wall-clock time of the committed epoch

Subscribe with a plain NATS subscription (no reply needed):

nats sub '$INS.sys.prod.feed.epoch'

Memory Profile: $INS.feed.memory

A fire-and-forget broadcast (not a request/reply endpoint) published by the server once per sampler tick (db.memory-profile.interval, default 2s) whenever the profiler is enabled. The payload is the same Snapshot the $INS.ops.memory endpoint returns; each message is a complete frame, not a delta. It backs insights ops memory --live, which renders an initial frame from the request/reply and then re-renders on each broadcast.

Payload: a Snapshot, identical in shape to the $INS.ops.memory response above.

Subscribe with a plain NATS subscription (no reply needed):

nats sub '$INS.sys.prod.feed.memory'

Error Handling

All endpoints return errors using the NATS micro error format:

• Code: a string error code (HTTP-style: 400, 404, 500, etc.)
• Description: a human-readable error message
• Data: optional additional context (JSON bytes)

Internal errors are masked with a generic 500 / "internal server error" response to avoid leaking implementation details.

Content Negotiation

The db.query and checks.findings endpoints support content negotiation via the NATS message Accept header:

• application/json (default): results as a JSON array/object
• text/csv: results as CSV with a header row