Metrics

Insights exposes its own runtime metrics as a Prometheus-compatible /metrics endpoint. These metrics describe Insights itself (scrape and ingest latency, retention sweeps, and the current check findings), not the NATS system it observes.

Every metric is per-system: one node can own several monitored systems, each with its own catalog, indexer, scraper, and epoch clock. So every series carries a system label naming the monitored system it describes. Aggregate on it with sum by (system) (...), or a two-system node's work reads as one blended series.

Endpoint

The Prometheus endpoint is enabled by default on 127.0.0.1:9091/metrics. Configuration lives under the prometheus.* section; see the prometheus section of the Configuration reference for every flag.

To scrape it from another host, set --prometheus.hostname 0.0.0.0 (or bind to a specific interface).

One endpoint for a whole deployment

A node's /metrics describes that node. In a deployment of several nodes, such as a web tier split from an indexer or several monitored systems on separate hosts, Prometheus needs a route to each one, and a scrape-target list that changes as nodes come and go.

Setting prometheus.aggregate: true on one node makes its endpoint answer for all of them. Each scrape discovers the deployment's nodes over $INS.ping, asks every one of them for its snapshot at once, and serves the merged result. Every series gains a source label naming the node it came from, alongside the system label naming the monitored system it describes.

insights prometheus serves exactly the same thing as a standalone process, holding no database and running no indexer. Use it when the scrape target should not be a full node: Prometheus can reach it, but the nodes themselves sit behind a leafnode, NAT, or a tailnet it cannot route to.

Two things to know:

• Enable it on exactly one endpoint. The source label is identical across every aggregator, so scraping two of them reports every series twice, separated only by Prometheus's own instance label, and a naive sum() double-counts.
• --prometheus.timeout must be shorter than your scrape_timeout. It bounds the whole collection (node discovery, then the per-node requests), and a node that does not answer inside it is reported down for that scrape rather than delaying it.

Collection happens per scrape, so a node that stops answering is reported down immediately rather than after a staleness window, and nothing is transferred when nobody is scraping.

A node that answers with an empty registry reports 1 and contributes no series. A collector, which measures nothing of its own, looks like this. That is deliberately distinct from a node that answered discovery but not the metrics request, which reports 0.

A node that has stopped entirely answers neither, so it leaves the node set rather than reporting 0. This is how Prometheus service discovery behaves generally (a target that disappears stops producing up), so alert on a vanished node with absent(), and on insights_metrics_source_up == 0 for a node that is running but cannot answer.

A web-only node (insights web) owns no catalog and scrapes nothing, so it runs no node API and takes no part in discovery or aggregation. It can still serve an aggregated endpoint by setting prometheus.aggregate; it just does not appear as a source in one.

Metric Catalog

insights_check_epoch_timestamp is NaN for a system until that system's first epoch commits, so a time() - insights_check_epoch_timestamp staleness query yields a gap rather than a decades-old timestamp on a fresh instance.

The three insights_epoch_* completeness gauges are likewise NaN when the scraper did not report completeness for that epoch, because the scraper is older or the epoch was replayed from before it did. NaN means unknown, not complete, and propagates through expected - observed so an unreported epoch never alerts.

insights_check_failures and insights_check_failures_total count every finding recorded at the epoch: operational and optimization checks alike, and checks listed under disabled-checks too. A disabled check is never delivered as a notification, but an alert built on these series still sees it. To leave one out, alert on sum by (system, severity) (insights_check_failures{code!="account-012"}) rather than on the total, which carries no code label.

All histogram metrics also expose the standard Prometheus companion time series (_bucket, _sum, _count) and are compatible with histogram_quantile and rate.

Label Values

Code

The check code as it appears in the Checks Reference: lowercase and hyphenated, such as server-001.

Severity

Scope

Category

Reason

Termination reason for insights_stream_queries_total. Every streamed query is counted exactly once under one of these values.

Example Queries

Rates and quantiles are computed with the standard Prometheus functions; the examples below assume the scrape job is named insights.

Every example keeps system in its grouping. Drop it only when you want a deployment-wide total. Otherwise a multi-system node's systems blend into one series and the number stops being actionable.

Total critical failures right now

insights_check_failures_total{severity="critical"}

Current failure count by category

insights_check_failures is a gauge that is reset and repopulated each epoch, so query the current value directly rather than using rate() (which misinterprets the periodic resets).

sum by (system, category) (insights_check_failures)

p95 ingest duration per endpoint

histogram_quantile(0.95,
  sum by (system, endpoint, le) (rate(insights_ingest_duration_seconds_bucket[5m]))
)

Time since the last epoch was processed

time() - insights_check_epoch_timestamp

Servers missing from the last epoch

A server that misses the scraper's discovery window is dropped from the scrape with no error, and the epoch's totals silently lose its connections, routes and throughput. Anything above zero means the last epoch's numbers are partial.

insights_epoch_servers_expected - insights_epoch_servers_observed > 0

This catches a server the scrape never reached. It does not catch one that answered the servers endpoint and nothing else. That server is counted as observed while contributing only the counters STATSZ carries. The endpoints it failed to answer are counted as partial, so this fires for it, and the server-065 finding names the server:

insights_epoch_partial_endpoints > 0

These gauges depend on the scraper reporting completeness, and read NaN until it does. Independently of that, the indexer compares each epoch's server set against the previous one and logs the servers that disappeared by name:

WARN epoch/servers missing epoch=1754826141 servers=[aws-useast2-natscj1-1]

Retention-sweep latency percentile

histogram_quantile(0.99,
  sum by (system, le) (rate(insights_retention_sweep_duration_seconds_bucket[30m]))
)

Streamed query rate by termination reason

sum by (system, reason) (rate(insights_stream_queries_total[5m]))

p95 streamed-query duration

histogram_quantile(0.95,
  sum by (system, le) (rate(insights_stream_query_duration_seconds_bucket[5m]))
)

Legacy-endpoint row-cap rejection rate

A sustained non-zero rate means clients are still hitting the buffered $INS.db.query endpoint with oversized results and should migrate to $INS.db.query.stream.

sum by (system) (rate(insights_query_rejected_total[5m]))

Alerting Rules

The following rules are a reasonable starting point. Tune thresholds and for durations to your deployment's SLOs.

Each rule alerts per system: the expressions preserve the system label, so a node owning three systems produces three independent alerts rather than one blended one, and every annotation names the system it fired for.

groups:
  - name: insights
    rules:
      - alert: InsightsEpochStale
        expr: time() - insights_check_epoch_timestamp{job="insights"} > 180
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: Insights epoch data is stale for {{ $labels.system }}
          description: >
            No new epoch has been processed for system {{ $labels.system }} in over 3 minutes.
            Check that Insights is running and the scraper is healthy.

      - alert: InsightsCriticalFailures
        expr: insights_check_failures_total{job="insights",severity="critical"} > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: Insights reports critical check failures for {{ $labels.system }}
          description: >
            {{ $value }} critical check failure(s) detected on system {{ $labels.system }}.
            Review the Insights UI or Grafana dashboard for details.

      - alert: InsightsHighWarnings
        expr: insights_check_failures_total{job="insights",severity="warning"} > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: Elevated warning-level check failures for {{ $labels.system }}
          description: >
            {{ $value }} warning check failures detected on system {{ $labels.system }}
            for over 5 minutes.

      - alert: InsightsSlowIngest
        expr: |
          histogram_quantile(0.95,
            sum by (system, endpoint, le) (rate(insights_ingest_duration_seconds_bucket{job="insights"}[5m]))
          ) > 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: Slow ingest for {{ $labels.endpoint }} on {{ $labels.system }}
          description: >
            p95 ingest duration for endpoint {{ $labels.endpoint }} on system {{ $labels.system }}
            exceeded 1s for 10 minutes — the indexer may be falling behind the scraper.