# Prometheus & Grafana

This guide covers setting up Prometheus to scrape Insights metrics and importing the provided Grafana dashboard. For the full metrics catalogue, label values, and example PromQL, see the Metrics reference.

## Prerequisites

* Insights running with the Prometheus endpoint enabled (default)
* Prometheus v2+
* Grafana v10+

## Prometheus configuration

Add the following scrape job to your `prometheus.yml`:

```
scrape_configs:

  - job_name: insights

    scrape_interval: 30s

    static_configs:

      - targets: ["127.0.0.1:9091"]
```

Adjust the target address if you've changed the Insights bind hostname or port. Keep `job_name: insights`, or change the dashboard's **Job** variable to match.

Every series carries a `system` label naming the monitored system it describes, so one node monitoring several systems produces one series per system. Keep `system` in any aggregation (`sum by (system) (...)`) unless you want a deployment-wide total.

A node's endpoint describes that node only. In a deployment of several nodes, either list each node as a target, or set `prometheus.aggregate: true` on one node (or run `insights prometheus` as a standalone process) and scrape that single endpoint: each scrape discovers the nodes over `$INS.ping` and serves all of their metrics, with a `source` label naming the node each series came from, plus `insights_metrics_source_up` per node. Enable aggregation on exactly one endpoint, or every series is reported twice. See One endpoint for a whole deployment.

## Insights configuration

The Prometheus metrics endpoint is enabled by default on `127.0.0.1:9091/metrics`.

| Flag                     | Environment Variable            | Default     | Description                                                                                                       |
| ------------------------ | ------------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------- |
| `--prometheus`           | `INSIGHTS_PROMETHEUS`           | `true`      | Enable the metrics endpoint                                                                                       |
| `--prometheus.hostname`  | `INSIGHTS_PROMETHEUS_HOSTNAME`  | `127.0.0.1` | Bind hostname                                                                                                     |
| `--prometheus.port`      | `INSIGHTS_PROMETHEUS_PORT`      | `9091`      | Bind port                                                                                                         |
| `--prometheus.aggregate` | `INSIGHTS_PROMETHEUS_AGGREGATE` | `false`     | Serve every discovered node's metrics from this endpoint                                                          |
| `--prometheus.timeout`   | `INSIGHTS_PROMETHEUS_TIMEOUT`   | `2s`        | Budget for one aggregated collection; keep it below Prometheus' scrape timeout. Ignored unless `aggregate` is set |

Or in `config.yaml`:

```
prometheus:

  enabled: true

  hostname: 0.0.0.0

  port: 9091
```

## Importing the dashboard

The dashboard has a **Job** variable (default `insights`), which must match the Prometheus `job_name`, and a **System** variable, populated from the `system` label the metrics carry. It defaults to *All*, so a single-system deployment needs no attention; on a multi-system node, select one system to scope every panel to it, or leave it on *All* to compare them side by side (each series is suffixed with its system).

### Via Grafana UI

1. Open Grafana and navigate to **Dashboards > New > Import**
2. Click **Upload dashboard JSON file** and select `operations/grafana/insights-dashboard.json`
3. Select your Prometheus data source
4. Click **Import**

### Via provisioning

Place the dashboard JSON in your Grafana provisioning directory and add a provider:

```
# /etc/grafana/provisioning/dashboards/insights.yaml

apiVersion: 1

providers:

  - name: Insights

    type: file

    options:

      path: /var/lib/grafana/dashboards/insights

      foldersFromFilesStructure: false
```

Then copy `insights-dashboard.json` to `/var/lib/grafana/dashboards/insights/`.

## Querying indexed data (HTTP Gateway)

The Prometheus integration above exposes Insights' own operational metrics. To chart the *indexed NATS data* itself (the `hx.*` tables — servers, streams, consumers, connections) in Grafana, run the `insights http` gateway and query it from Grafana's [Infinity](https://grafana.com/grafana/plugins/yesoreyeram-infinity-datasource/) data source, which speaks HTTP rather than NATS.

```
insights http --server nats://data-cluster:4222 --auth-token "$TOKEN" --listen 0.0.0.0:8080
```

Point an Infinity data source at the gateway and `POST` SQL to `/sys/<id>/db/query` (send `Authorization: Bearer <token>` and a JSON body `{"sql": "…"}`); the gateway returns JSON (or CSV with `Accept: text/csv`). The read-only guard and row caps stay enforced server-side. See the HTTP gateway reference for routes and error mapping.

One gateway serves every system, and each query names the one it wants in the path. Populate a dashboard *system* variable from `GET /systems`, which lists the reachable systems as JSON, and interpolate it:

```
https://insights-gw.example.com/sys/$system/db/query
```

One dashboard then charts several systems side by side, or re-scopes entirely through the variable. A panel cannot be answered by a system other than the one its URL names, so a dashboard built against one system keeps working unchanged when others are added.

## Next steps

* Metrics reference — full metric catalogue, label values, example PromQL, and alerting rules
* HTTP gateway — query the indexed data over HTTP
