Monitoring with Prometheus & Grafana

This guide covers setting up Prometheus to scrape Insights metrics and importing the provided Grafana dashboard. For the full metrics catalogue, label values, and example PromQL, see the Metrics reference.

Prerequisites

• Insights running with the Prometheus endpoint enabled (default)
• Prometheus v2+
• Grafana v10+

Prometheus Configuration

Add the following scrape job to your prometheus.yml:

scrape_configs:
  - job_name: insights
    scrape_interval: 30s
    static_configs:
      - targets: ["127.0.0.1:9091"]

Adjust the target address if you've changed the Insights bind hostname or port. Keep job_name: insights, or change the dashboard's Job variable to match.

Every series carries a system label naming the monitored system it describes, so one node monitoring several systems produces one series per system. Keep system in any aggregation (sum by (system) (...)) unless you want a deployment-wide total.

A node's endpoint describes that node only. In a deployment of several nodes, either list each node as a target, or set prometheus.aggregate: true on one node (or run insights prometheus as a standalone process) and scrape that single endpoint: each scrape discovers the nodes over $INS.ping and serves all of their metrics, with a source label naming the node each series came from, plus insights_metrics_source_up per node. Enable aggregation on exactly one endpoint, or every series is reported twice. See One endpoint for a whole deployment.

Insights Configuration

The Prometheus metrics endpoint is enabled by default on 127.0.0.1:9091/metrics.

Flag	Environment Variable	Default	Description
--prometheus	INSIGHTS_PROMETHEUS	true	Enable the metrics endpoint
--prometheus.hostname	INSIGHTS_PROMETHEUS_HOSTNAME	127.0.0.1	Bind hostname
--prometheus.port	INSIGHTS_PROMETHEUS_PORT	9091	Bind port
--prometheus.aggregate	INSIGHTS_PROMETHEUS_AGGREGATE	false	Serve every discovered node's metrics from this endpoint
--prometheus.timeout	INSIGHTS_PROMETHEUS_TIMEOUT	2s	Budget for one aggregated collection; keep it below Prometheus' scrape timeout. Ignored unless aggregate is set

Or in config.yaml:

prometheus:
  enabled: true
  hostname: 0.0.0.0
  port: 9091

Importing the Dashboard

The dashboard has a Job variable (default insights), which must match the Prometheus job_name, and a System variable, populated from the system label the metrics carry. It defaults to All, so a single-system deployment needs no attention; on a multi-system node, select one system to scope every panel to it, or leave it on All to compare them side by side (each series is suffixed with its system).

Via Grafana UI

1. Open Grafana and navigate to Dashboards > New > Import
2. Click Upload dashboard JSON file and select operations/grafana/insights-dashboard.json
3. Select your Prometheus data source
4. Click Import

Via Provisioning

Place the dashboard JSON in your Grafana provisioning directory and add a provider:

# /etc/grafana/provisioning/dashboards/insights.yaml
apiVersion: 1
providers:
  - name: Insights
    type: file
    options:
      path: /var/lib/grafana/dashboards/insights
      foldersFromFilesStructure: false

Then copy insights-dashboard.json to /var/lib/grafana/dashboards/insights/.

Querying indexed data (HTTP gateway)

The Prometheus integration above exposes Insights' own operational metrics. To chart the indexed NATS data itself (the hx.* tables — servers, streams, consumers, connections) in Grafana, run the insights http gateway and query it from Grafana's Infinity data source, which speaks HTTP rather than NATS.

insights http --server nats://data-cluster:4222 --auth-token "$TOKEN" --listen 0.0.0.0:8080

Point an Infinity data source at the gateway and POST SQL to /sys/<id>/db/query (send Authorization: Bearer <token> and a JSON body {"sql": "…"}); the gateway returns JSON (or CSV with Accept: text/csv). The read-only guard and row caps stay enforced server-side. See the HTTP gateway reference for routes and error mapping.

One gateway serves every system, and each query names the one it wants in the path. Populate a dashboard system variable from GET /systems, which lists the reachable systems as JSON, and interpolate it:

https://insights-gw.example.com/sys/$system/db/query

One dashboard then charts several systems side by side, or re-scopes entirely through the variable. A panel cannot be answered by a system other than the one its URL names, so a dashboard built against one system keeps working unchanged when others are added.

Next Steps

• Metrics reference — full metric catalogue, label values, example PromQL, and alerting rules
• HTTP gateway — query the indexed data over HTTP