API

Insights exposes a programmatic API over NATS request/reply using NATS micro. The API starts automatically whenever a sink connection is available and provides the same query capabilities that power the web UI.

Any NATS client connected to the same server (or cluster) can issue requests to these endpoints. This enables automation, integration with external tooling, and operation without the web UI.

Service Info

The API registers as a NATS micro service named insights with version 0.0.1. You can discover it using the standard micro service discovery subjects ($SRV.INFO, $SRV.PING, etc.). Each instance's micro metadata carries its node_id, but $INS.ping is the supported discovery contract.

Subject Hierarchy

All API subjects use the $INS prefix and follow a $INS.{domain}.{action} convention. The table names that domain and action half. On the wire, every subject also carries the id of the system or node it addresses, shown in the Space column and described in Subject Spaces.

An endpoint is registered only on a node that holds the matching side of the system: the db, checks and usage subjects and ops.info, ops.sizing, ops.memory and ops.storage on the node holding the catalog; the scraper, diagnostics, disconnect-history and sink subjects on the node holding the scrape side.

The first token after the id identifies the domain, the second the action. That makes subjects predictable and easy to authorize with NATS subject permissions (for example, grant $INS.sys.*.db.> for all database operations).

Subject Spaces

Every subject on the wire carries an operator-assigned identity token after $INS, in one of two spaces:

System-scoped subjects address a resource that belongs to the monitored NATS system: its database, its checks, its scraper. For example, $INS.sys.prod.db.query queries the system named prod.

Node-scoped subjects describe the Insights process itself, independent of any system or database: ops.pprof, ops.version, ops.version.check, ops.nats.info, ops.metrics, and the webhook.delivery.* log. For example, $INS.node.prod-1.ops.pprof profiles the process named prod-1. This matters as soon as more than one instance shares a NATS account, where an unscoped subject would queue-balance to a random one.

No unscoped form is served. $INS.ping is the only flat subject, and it is a broadcast rather than a queue-balanced endpoint.

Subject authorization scopes naturally: grant $INS.sys.<id>.> for one system's full API, or $INS.sys.*.db.> for database access to every system. That prefix is the whole of a system's traffic. The scrape stream it fills ($INS.sys.<id>.scrape.>) is inside it, so a grant written against the prefix covers it, and it stays separable if you want to deny it.

Broadcast Feeds

feed is the exception to request/reply. Every subject under it is a fire-and-forget broadcast: a complete frame, no reply, no queue group, no delivery guarantee. It is a domain of its own so that a live-updating client can be granted $INS.sys.<id>.feed.> and nothing else.

feed.finding appends three tokens naming what changed:

$INS.sys.<id>.feed.finding.<status>.<severity>.<code>
$INS.sys.prod.feed.finding.firing.critical.server-001

The tokens are in the subject, not only the payload, so a subscriber filters server-side: feed.finding.firing.> for everything firing, feed.finding.*.critical.> for every critical transition, feed.finding.*.*.server-001 for one check in both directions.

status is firing, resolved, or changed. A changed advisory is a check whose severity moved while it stayed up, which happens for checks with a dynamic severity. severity is the word (info, warning, critical); on a resolved advisory it is the severity the finding held when it went away.

The payload is one finding's transition:

{
  "code": "server-003",
  "entity_type": "server",
  "entity_pk": 42,
  "entity_name": "nats-1",
  "status": "resolved",
  "severity": "warning",
  "starts_at": "2024-01-15T09:12:00Z",
  "ends_at": "2024-01-15T09:30:00Z"
}

prev_severity appears only on a changed advisory, naming the severity left behind; ends_at only on a resolved one. entity_key carries the entity's key for a finding that has one, such as a service name.

starts_at is when the finding first appeared, carried across every epoch it stays up, so a resolve reports how long the condition actually lasted. Entity names are resolved at transition time and cached, so a resolve still names an entity that has since been deleted.

On start, a node seeds its baseline from the latest epoch already in its catalog, so the first epoch it indexes publishes only what changed since. A catalog with no epoch yet has nothing to seed from: its first epoch establishes the baseline and publishes nothing. A check turned off with disabled-checks publishes nothing on the feed.

feed.epoch is published after an epoch's data is committed and visible in the hx tables, so a live view can re-run its queries against it:

{ "node_id": "prod-1", "epoch": "2024-01-15T09:30:00Z" }

nats sub '$INS.sys.prod.feed.epoch'

feed.memory carries the same snapshot $INS.ops.memory returns, once per sampler tick (db.memory-profile.interval, default 2s) while the profiler is enabled.

Discovery: $INS.ping

$INS.ping is a broadcast every instance answers; it is a plain subscription, not a queue-balanced micro endpoint. Publish a request and collect replies for a window. Each reply reports the instance's node id, binary version, capability snapshot, labels, and the systems it participates in with its role for each:

{
  "node_id": "prod-1",
  "version": "1.2.0",
  "capabilities": {"geo": {...}, "realtime": {...}, "scraper": {...}, "db": {...}, "checks": {...}},
  "systems": [{"id": "prod", "roles": ["scrape", "catalog"], "metadata": {"env": "prod"}}],
  "metadata": {"region": "us-east-1"}
}

roles is per system, not per node. scrape means the instance holds the system's scrape side (the connection to the monitored system, the scraper, the stream). catalog means it holds the database and serves the query surface.

capabilities is the ops.info payload of the first system the instance holds a catalog for, and empty on an instance holding none; ask each system's $INS.sys.<id>.ops.info when a node serves several. The top-level metadata is the instance's node.metadata labels. A system's metadata is its system.metadata labels, reported only by the instance holding that system's scrape side. Both are omitted when empty.

A system entry may also carry catalog_error, which means the instance holds that system's catalog but cannot serve it because its database would not open:

{ "id": "edge-west", "roles": ["catalog"], "catalog_error": "open database: file is not a valid DuckDB file" }

The catalog role stays. No other instance will answer for that system, so requests still route here and return 503 with the reason rather than timing out. insights system list prints these below its table. Every other system on the instance is unaffected.

The instance retries the failed database once a minute. Once the underlying fault is repaired, it reopens on its own: catalog_error clears, queries start answering, and indexing resumes from wherever the stream still reaches, with no restart. The system's finding feed and its webhook and Alertmanager deliveries resume only after the node restarts.

HTTP Gateway

For clients that can't speak NATS (for example, Grafana's Infinity data source), the insights http subcommand runs a stateless HTTP-to-NATS bridge. It translates each HTTP request into the matching read-only $INS.sys.<id>.db.* NATS request and returns the reply. The gateway owns no database and runs no queries itself; the read-only guard and row caps stay enforced by the db handlers.

Only the read-only db request/reply surface is exposed; the ops, checks, and streaming subjects are not. POST /sys/<id>/db/query honors the Accept header (text/csv for CSV, otherwise JSON).

System selection. Every db route names its system in the path, where the subject carries it: /sys/<id>/db/query becomes $INS.sys.<id>.db.query. There is no default system and no cached topology, so a request is always answered by the system it named, and adding a second system changes nothing for clients already addressing the first. GET /systems lists what is addressable (one $INS.ping round, the same data as insights system list --format json), so a dashboard can populate a system variable from it and interpolate it into the path.

Because the gateway holds no list of valid ids, a mistyped id and a stopped indexer look the same from its position. Both return 503: no responder for system "prd" (GET /systems lists the reachable systems). A system id must be a subject token (letters, digits, _, -); anything else is a 400 before the request reaches NATS.

Authentication. Every route except /healthz requires a bearer token (Authorization: Bearer <token>), set with --auth-token. The gateway speaks to NATS with the single identity it was started with, so multi-tenant separation is one gateway per NATS account. It refuses to start without a token unless --allow-unauthenticated is given. See the CLI reference for the full flag set.

Error mapping. An upstream error keeps its HTTP-style status: a row-cap overflow surfaces as 413, invalid SQL as 400. A missing responder maps to 503, an upstream timeout to 504, and any other transport failure to 502.

# what is addressable
curl -s localhost:8080/systems -H "Authorization: Bearer $TOKEN"

curl -s localhost:8080/sys/prod/db/query \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"sql": "SELECT count(*) AS n FROM hx.server_ident"}'

# CSV instead of JSON:
curl -s localhost:8080/sys/prod/db/query \
  -H "Authorization: Bearer $TOKEN" -H "Accept: text/csv" \
  -d '{"sql": "SELECT name, cpu FROM hx.servers ORDER BY cpu DESC LIMIT 10"}'

Endpoints

The examples below address a system with id prod.

Query: $INS.db.query

Execute read-only SQL queries against the DuckDB database.

Read-only is enforced with DuckDB's own parser: the statement is prepared (parsed and bound, never executed) and rejected unless it is a single statement DuckDB classifies as SELECT, which also covers WITH, FROM-first queries, DESCRIBE, SHOW, and SUMMARIZE. Writes, DDL, and state-changing statements (INSERT, CREATE, COPY, ATTACH, SET, and so on) are rejected, as is a multi-statement payload. EXPLAIN is rejected too, because EXPLAIN ANALYZE executes its wrapped statement; use $INS.db.explain for query plans.

Request:

{
  "sql": "SELECT * FROM hx.server_ident LIMIT 10",
  "params": []
}

Response (JSON, default):

[
  { "name": "server-1", "cluster": "us-east", "version": "2.10.0" },
  { "name": "server-2", "cluster": "us-west", "version": "2.10.0" }
]

An array of objects, one per row, keyed by column name.

Response (CSV):

Set the Accept: text/csv header to receive results as CSV.

This endpoint buffers the whole result before responding, so it caps the row count (db.query-max-rows, default 100000). A result that would exceed the cap is rejected with 413. Add a LIMIT, or use the streaming endpoint ($INS.db.query.stream), which streams the result instead of buffering it.

Errors:

Example using nats CLI:

nats req '$INS.sys.prod.db.query' '{"sql": "SELECT count(*) as n FROM hx.server_ident"}'

Streaming Query: $INS.db.query.stream

Streams a large read-only result instead of buffering it. The result is delivered to the request's reply inbox as ordered Apache Arrow IPC chunks, so neither the server nor the client holds the whole result in memory. This is the path for large exports. The same read-only rules as $INS.db.query apply.

The client acknowledges chunks and the server keeps at most db.query-stream.window chunks in flight, so a slow client throttles the server and a client that disappears aborts the scan. There is no buffering row cap, but a safety limit, db.query-stream.max-rows (default 1000000; 0 disables), aborts a runaway unscoped scan. Scope the query by epoch or add a LIMIT rather than raising it.

Values are rendered from Arrow, so formatting differs from $INS.db.query: timestamps are RFC 3339 in CSV as well as JSON, NULL is an empty CSV field or JSON null, BLOB is base64, and DECIMAL/HUGEINT are canonical decimal strings. Chunks can be compressed with db.query-stream.compression (none, lz4, zstd; default none); Arrow IPC readers decompress transparently.

The insights db query command uses this endpoint automatically, falling back to $INS.db.query when no streaming responder is available.

Explain and Schema Discovery: $INS.db.explain / .schemas / .tables / .columns / .macros

These read-only endpoints back the insights db subcommands and the matching HTTP gateway routes. Discovery reads catalog metadata only (no data is scanned) and is scoped to the allowlisted hx, main, checks, and contexts schemas. Each check is a table macro in checks (check server-001 is checks.server_001), and the macros behind a finding's supporting context panels are in contexts.

• $INS.db.explain. Plan a read-only query without executing it. Request { "sql": "…", "analyze": false }; set "analyze": true to run EXPLAIN ANALYZE, which executes the query (still read-only-guarded, and bound by db.query-timeout) and adds per-operator runtime timing. Returns {"valid": true, "plan": [...]} with the DuckDB plan rows.
• $INS.db.schemas. List schemas with descriptions and table, view, and macro counts. Empty request ({}).
• $INS.db.tables. List tables and views. Request { "schema": "hx" }; omit schema to list across all allowlisted schemas.
• $INS.db.columns. List columns of a table or view, with the table comment and each column's comment. Request { "schema": "hx", "table": "server_stats" }.
• $INS.db.macros. List macros with signatures. Request { "schema": "checks" }; omit schema for all.

Each returns 503 when this node holds the system's catalog but its database is not open.

nats req '$INS.sys.prod.db.explain' '{"sql": "SELECT count(*) FROM hx.server_ident"}'
nats req '$INS.sys.prod.db.tables' '{"schema": "hx"}'

Checks List: $INS.checks.list

Returns all checks grouped by category, including configurable parameters.

Request: Empty payload.

Response:

[
  {
    "label": "Server",
    "checks": [
      {
        "code": "server-003",
        "name": "High CPU Usage",
        "description": "Flags servers where per-core CPU usage exceeds 90%.",
        "scope": "server",
        "optimization": false,
        "params": [
          {
            "name": "cpu_percent",
            "default": 90.0,
            "resolved": 85.0,
            "description": "Per-core CPU usage threshold percentage"
          }
        ]
      }
    ]
  }
]

The resolved value reflects any overrides from check-thresholds.

An entry carries disabled: true only when the addressed system turned the check off with disabled-checks. A disabled check still runs and still produces findings; it is left out of health grades and never delivered as a notification.

An entry carries min_server_version when the check reads a monitoring field that older NATS Server releases don't report. Entities on a server below that release aren't evaluated, so their absence from the findings is not a pass.

Checks Info: $INS.checks.info

Return full metadata for a single check without running it: description, remediation, scope, severity, and configurable parameters with default and resolved (in-effect) values.

Request:

{ "code": "server-003" }

Response:

{
  "code": "server-003",
  "name": "High CPU Usage",
  "description": "Flags servers where per-core CPU usage exceeds 90%.",
  "remediation": "Identify the workload driving CPU usage using the /pprof/profile debug endpoint. ...",
  "scope": "server",
  "severity": "warning",
  "optimization": false,
  "category": "performance",
  "category_label": "Performance & Latency",
  "params": [
    {
      "name": "cpu_percent",
      "default": 90.0,
      "resolved": 85.0,
      "description": "Per-core CPU usage threshold percentage"
    }
  ],
  "detail_columns": [
    {
      "name": "cpu_percent",
      "type": "DOUBLE",
      "description": "Per-core CPU usage percentage."
    }
  ]
}

The response also carries dynamic_severity when the check computes severity per row, disabled when the system turned the check off, min_server_version when the check needs a minimum NATS Server release, and lookback_seconds for optimization checks.

Errors:

Checks Findings: $INS.checks.findings

Execute a specific check by code over a time window and return matching findings.

Request:

{
  "code": "server-003",
  "time": {
    "duration": "1 hour"
  },
  "page": 1
}

Response (JSON, default):

{
  "code": "server-003",
  "name": "High CPU Usage",
  "detail_columns": [
    { "name": "cpu_percent", "type": "DOUBLE", "description": "Per-core CPU usage percentage." }
  ],
  "rows": [
    {
      "code": "server-003",
      "severity": "warning",
      "entity": "us-east / nats-1",
      "entity_pk": 42,
      "check_name": "High CPU Usage",
      "cpu_percent": 92.3
    }
  ],
  "page_info": {
    "page": 1,
    "page_size": 20,
    "total_rows": 1,
    "total_pages": 1
  }
}

name, detail_columns, dynamic_severity (when the check computes severity per row), and disabled (when the system turned the check off) are echoed so clients can render findings without a second request to $INS.checks.info.

Response (CSV): Set the Accept: text/csv header. The CSV form carries the rows only, so it has no disabled marker.

Errors:

Backup: $INS.db.backup.stream

Stream one table's rows so a client can build a backup database locally. The server reads and streams; it writes nothing and buffers nothing. insights ops backup save resolves the epoch window once, then issues one request per table and inserts each stream into a DuckDB file it creates. See insights ops backup.

The chunk protocol is the one $INS.db.query.stream uses: Arrow IPC chunks on the reply inbox, windowed acknowledgements, and an in-band trailer. Only the request differs.

Request:

{
  "schema": "hx",
  "table": "stream_opts",
  "start_epoch": "2024-01-01T00:00:00Z",
  "end_epoch": "2024-01-02T00:00:00Z"
}

The request names a table rather than SQL, and the server builds the query, so this endpoint is exempt from db.query-stream.max-rows. It is its own endpoint rather than a mode of $INS.db.query.stream, so a backup running for minutes doesn't hold up streaming queries.

Errors are reported in-band on the reply inbox as an error trailer, including an unknown system, a table outside the backup set, and a mid-scan failure.

Instance Info: $INS.ops.info

Report the system's capabilities and configuration. A standalone web tier uses this to discover the settings of the backend it connects to, rather than reading its own local config. The request payload is empty.

Response:

{
  "geo": { "enabled": true },
  "realtime": { "enabled": false },
  "scraper": { "interval": 60000000000 },
  "db": { "retention": 2764800000000000 },
  "checks": { "disabled": ["server-061"] }
}

scraper.interval and db.retention are durations serialized as integer nanoseconds: 60000000000 is 1m, 2764800000000000 is 768h.

nats req '$INS.sys.prod.ops.info' ''

The related $INS.ops.storage subject reports the database file and WAL sizes and connection-pool statistics.

Error Handling

All endpoints return errors using the NATS micro error format:

• Code. A string error code (HTTP-style: 400, 404, 500, and so on).
• Description. A human-readable error message.
• Data. Optional additional context (JSON bytes).

Internal errors get masked with a generic 500 / "internal server error" response, so implementation details don't leak.

Content Negotiation

The db.query and checks.findings endpoints support content negotiation via the NATS message Accept header:

• application/json (default). Results as a JSON array/object.
• text/csv. Results as CSV with a header row.