# Upgrading to v1.2.0

v1.2.0 introduces multi-system support: one Insights node can monitor several NATS systems at once, each with its own database, API subtree and web pages. That shows up in several places in this guide, because anything that used to be implicit now names the system it belongs to.

Most of the upgrade takes care of itself. The database migrates on first start and your check history comes with it, and a single-system config file from v0.1.10 starts as it is: keys that no longer exist are ignored rather than rejected.

Two settings are worth a look before you restart, and only if you use them. Check codes have clearer names, so a `check-thresholds` override, a `disabled-checks` entry or a notification `subscriptions` or `ignore` entry that names a code needs the new name. A handful of threshold parameters were also renamed, and some take typed values like `100ms` and `1 MiB` rather than a bare number with the unit in the parameter name.

`insights config check` finds those names against your existing config without starting anything, so run it first. It reports one group of problems at a time, so run it again after each round of edits until it prints `configuration is valid`. It checks names, not values: a new threshold value is still yours to read against the tables below.

## Does this affect you?

Each row is something you can check in a few seconds.

| Area                     | Affects you if                                                                                                                                                                                                                                                                                 | Go to                                     |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- |
| Check codes              | `grep -nE '\b[A-Z]+_[0-9]{3}\b' config.yaml` returns anything                                                                                                                                                                                                                                  | [Checks](#checks)                         |
| Check parameters         | a `check-thresholds` override names a parameter ending `_ms`, `_secs`, `_minutes`, `_mib`, `_gib`, `_kib`, `_pct` or `_frac`                                                                                                                                                                   | [Checks](#checks)                         |
| Notification config      | `grep -n 'max-retries' config.yaml` returns a line, or an endpoint sets `timeout:`                                                                                                                                                                                                             | [Configuration](#configuration)           |
| Backup automation        | a script runs `insights ops backup` with `--output -`, or publishes to `$INS.db.backup`                                                                                                                                                                                                        | [Operations](#operations)                 |
| Saved SQL and dashboards | a query names `hx.check_results` or an `audit.` macro, calls `json_extract` on `metadata`, `backoff`, `mirror_subject_transforms` or `subject_transforms`, or compares `storage_type`, `retention_policy`, `discard_policy`, `store_compression` or `persist_mode` against a capitalized value | [Data](#data)                             |
| NATS clients             | a tool you wrote publishes to `$INS.db.*`, `$INS.checks.*` or `$INS.ops.*`                                                                                                                                                                                                                     | [Interfaces](#interfaces)                 |
| Sizing automation        | a script reads `total_gb`, `query_gb`, `write_gb` or `index_gb` from `ops.sizing`                                                                                                                                                                                                              | [Interfaces](#interfaces)                 |
| HTTP gateway callers     | a caller posts to a `/db/...` path                                                                                                                                                                                                                                                             | [Interfaces](#interfaces)                 |
| MCP automations          | a saved prompt or script calls an Insights MCP tool without a `system` argument                                                                                                                                                                                                                | [Interfaces](#interfaces)                 |
| Alert receivers          | an Alertmanager route, inhibition or silence, or a webhook consumer, matches the `entity_name` label                                                                                                                                                                                           | [Interfaces](#interfaces)                 |
| Prometheus and Grafana   | a dashboard aggregates without grouping by `system`                                                                                                                                                                                                                                            | [Interfaces](#interfaces)                 |
| Container tooling        | a script, probe or init step runs a shell or package manager inside the Insights image                                                                                                                                                                                                         | [Operations](#operations)                 |
| Sink stream name         | `sink.stream`, `--sink.stream` or `INSIGHTS_SINK_STREAM` is set to anything other than `scrape`                                                                                                                                                                                                | [Interfaces](#interfaces)                 |
| Monitored NATS servers   | any of them runs NATS Server older than 2.10.27                                                                                                                                                                                                                                                | [Before you upgrade](#before-you-upgrade) |
| Telemetry                | you run a trial build (a `-licensed` archive or the `insights-licensed` image) and would rather opt out                                                                                                                                                                                        | [Configuration](#configuration)           |

## Before you upgrade

**1. Check the minimum NATS Server version.** Insights v1.2.0 requires NATS Server 2.10.27 or later. Earlier 2.10 patches do not serve the `raftz` and `ipqueuesz` endpoints, so the checks that read them stay quiet rather than reporting an error, which is easy to miss.

```
nats server report jetstream   # or: curl -s http://<server>:8222/varz | jq .version
```

**2. Keep a backup from your current version.** The database moves forward on first start and a v0.1.10 binary cannot read a v1.2.0 catalog, so a copy taken now is what makes going back an option.

```
insights ops backup --output insights-pre-1.2.0.db
```

**3. Validate your config with the new binary.** This reads the config file and `identity.json`, and creates the data directory if it is missing, as `serve` would. It opens no database and starts no listener, so it is safe to run against a live deployment.

```
insights config check --config /etc/insights/config.yaml
```

Many configs pass unchanged. Where one does not, the report names the offending code or parameter. A code that changed only in case and separator names its replacement (`SERVER_003` was renamed to `server-003`), and a parameter lists the valid names. A code whose number or entity changed prints `unknown check code` with no suggestion, so look it up in the [mapping table](#full-code-mapping). The report shows one group of problems per run, notification patterns before check codes, so repeat it until it prints `configuration is valid`. Add `--strict` to make unclaimed state a non-zero exit for a deploy gate.

**4. Reconcile ids if the report mentions unclaimed state.** Because a node can now serve several systems, a data directory records the node and system ids it was built under, which keeps each catalog matched to the system it holds. Renaming an id in config used to leave the old catalog behind and start an empty history; it is now something you declare. Against a **stopped** instance:

```
insights config migrate                              # adopt the new naming scheme

insights config migrate --rename system:west=north   # an id that actually changed

insights config migrate --prune system:retired-eu    # forget one, deleting nothing
```

With no flags this adopts the v1.2.0 naming scheme, which is the ordinary upgrade case: the ids held still and the names beneath them moved. Neither `--rename` nor `--prune` deletes data.

## Configuration

### Removed

| Key                           | Environment variable               | Note                                                                                                                                                                                                |
| ----------------------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `backup.object-store-ttl`     | `INSIGHTS_BACKUP_OBJECT_STORE_TTL` | Backups are built locally now and are not uploaded anywhere.                                                                                                                                        |
| `web.metrics-timeout`         | `INSIGHTS_WEB_METRICS_TIMEOUT`     | The component metrics poller they configured is off; nothing in Insights displayed it.                                                                                                              |
| `web.metrics-interval`        | `INSIGHTS_WEB_METRICS_INTERVAL`    |                                                                                                                                                                                                     |
| `notifications.*.timeout`     | none                               | Each request times out after 10s.                                                                                                                                                                   |
| `notifications.*.max-retries` | none                               | Each delivery makes up to 3 attempts (one when the receiver answers with a 4xx other than 429), and a failed delivery is retried for that endpoint at up to one-minute intervals until it succeeds. |

None of these stop a node from starting when they come from a config file or the environment: the keys are ignored, so remove them whenever it suits you. The `--web.metrics-interval` and `--web.metrics-timeout` flags are the exception; on a command line they fail with `unknown flag`.

### Renamed

| Old                                   | New                                        | Note                                                                               |
| ------------------------------------- | ------------------------------------------ | ---------------------------------------------------------------------------------- |
| `insights query`                      | `insights db query`                        | The old spelling remains as an alias on the same command, so scripts keep working. |
| `insights ops backup --output <file>` | `insights ops backup save --output <file>` | `save` is the default subcommand, so the old invocation still works.               |

### New

**Node and system identity.** Both subtrees are new, and they name the API space each part of the deployment answers on. `system.id` defaults to `default` and `node.id` defaults to the id of the single system the node serves, so a single-system deployment works without setting either. Set them when you want to address something by a name you chose.

```
node:

  id: "us-east"              # INSIGHTS_NODE_ID

  metadata:                  # labels for `insights node list` and the Nodes tab

    region: us-east-1

system:

  id: "prod"                 # INSIGHTS_SYSTEM_ID

  metadata:

    env: prod
```

See [Architecture](/insights/1.2/concepts/architecture.md) and Configuration.

**Multiple systems.** One node can own several monitored systems, each with its own DuckDB catalog, JetStream stream and `$INS.sys.<id>.*` API. Set `systems:` only on a node monitoring more than one system; it replaces `system.id` and the flat scraper config rather than sitting alongside them. A node serving more than one system has no single system id to derive its own from, so `node.id` is required there.

```
node:

  id: "central-1"

systems:

  - id: "core-prod"

    scrape:

      nats:

        server: "nats://core.internal:4222"

        creds: /etc/insights/core-prod.creds

      interval: 20s

    retention: 24h

  - id: "edge-west"          # collector-fed: no scrape block

    retention: 6h
```

See [Deployment](/insights/1.2/guides/deployment.md).

**Collector deployments.** `insights-collector` runs a scrape-and-stream collector with no local database, reading the same config file but using only the `data-dir`, `node`, `system`, `sys`, `scraper`, `sink`, `license`, `updater` and `telemetry` settings. A central node picks the system up by declaring it with no `scrape` block, as above. The collector's `node.id` defaults to its `system.id`, as does a central node's that serves only that system, and a second node claiming an id already in use refuses to start, so give them distinct `node.id` values. See [Federated Deployment](/insights/1.2/guides/federated-deployment.md).

**Turning a check off.** `disabled-checks` is new. A disabled check keeps running and stays visible marked disabled, with its real findings; it is left out of health grades and is never delivered as a notification. It suits a check that cannot be acted on in your topology, rather than one whose signal you would still want to see.

```
disabled-checks:

  - account-012
```

**Notification subscriptions take patterns.** `subscriptions` and the new `ignore` list accept an exact code, a trailing-wildcard prefix, or `*`. `ignore` applies after `subscriptions`, so it carves exceptions out of a wildcard. An empty `subscriptions` list delivers nothing, so subscribing stays explicit.

```
notifications:

  webhook:

    - name: "PagerDuty Relay"

      url: "https://example.com/webhook"

      subscriptions:

        - server-*

        - stream-002

      ignore:

        - server-046

      systems:               # empty = every system this node owns

        - core-prod
```

`systems:` is new alongside multi-system support: it scopes an endpoint to particular systems, and an empty list delivers for all of them. See [Notifications](/insights/1.2/guides/notifications.md).

**Sink stream shape.** `sink.replicas` and `sink.storage` are new. Replicas above 1 need a clustered sink, since the embedded server is a single node. The stream is a buffer rather than the history, so `storage: memory` is a reasonable choice where losing the buffer on restart is acceptable.

**Aggregated Prometheus endpoint.** `prometheus.aggregate` serves every node's metrics from one endpoint, discovering nodes over `$INS.ping`, which saves adding a scrape target per node. Enable it on exactly one node: scraping two aggregators would report every series twice.

```
prometheus:

  aggregate: true

  timeout: 2s               # keep this shorter than your scrape timeout
```

See Metrics.

**Inbox prefix.** `nats.inbox-prefix` sets the request/reply inbox prefix, for accounts that are not granted `_INBOX.>`.

**Telemetry, on by default in trial builds.** A pulse every five minutes carries a locally generated random instance id, the release version and platform, a few configuration durations and counts, the size of the watched environment, how many checks have a finding, and usage counters for the web UI and for requests made by the CLI, coding agents, MCP and HTTP clients. It carries nothing about the monitored system: no server, cluster, account, subject, stream or consumer names, no addresses, no message data, no configuration values, no query text. Only trial builds carry the reporter; the Insights build contains none. `insights web` sends no pulse of its own but passes the page views it serves to the node holding each system's database, so set the same switch there to stop that.

```
telemetry:

  enabled: false            # or --telemetry=false, INSIGHTS_TELEMETRY=false

  print: true               # log every pulse and read exactly what would leave
```

See Telemetry.

### Changed defaults

`db.memory-limit` defaults to `4GiB` rather than `4GB`. DuckDB reads `GB` as 10^9 bytes, so the old default capped each database at 3.7 GiB and the new one at 4.0 GiB. A value you set yourself keeps its meaning; write it in `MiB` or `GiB` to get the size it names.

Otherwise, anything you already set keeps its meaning. The subsystem switches (`scraper`, `indexer`, `web`, `simulator`, `prometheus`, `updater`) each gained an environment variable, so `INSIGHTS_WEB=false` works alongside `INSIGHTS_WEB_ENABLED=false`, and `updater` gained the `--updater` flag the others already had. The `--<subsystem>.enabled` flags still work but are hidden from `--help`. In YAML the section is still a map, so keep using its `enabled:` key.

## Checks

Check codes now name the entity a finding is about. A finding on a server starts with `server-`, one on a stream with `stream-`, and so on, so the code tells you where to look before you read anything else. The previous codes grouped by subsystem instead, which is why `CLUSTER_007` and `META_001` both described servers. Codes are also lowercase and hyphenated throughout, matching how they appear in the UI and the API.

Your history comes with you. A migration rewrites stored findings onto the new codes in one transaction, so timelines stay unbroken and open findings keep their age.

The config file is the one place that may need an edit, because codes appear there by name: in `check-thresholds`, in `disabled-checks`, and in the `subscriptions` and `ignore` lists on a webhook or Alertmanager endpoint. If you set none of them, there is nothing to change.

### Codes and parameters are validated at startup

A code matching no check is reported at startup with an error naming it, and the node does not start. A `check-thresholds` entry under an unrecognized code used to be passed over without comment, which meant a threshold you thought you had set was quietly doing nothing. Where only the spelling changed the error names the replacement, so `SERVER_001` tells you to write `server-001`. Where the number or the entity changed, the new name cannot be derived from the old one and the error says only `unknown check code`, so look it up in the [mapping table](#full-code-mapping). A parameter name that no longer exists fails the same way and lists the valid names.

Every place a code appears in the config is checked this way, the top-level and per-system `check-thresholds` and `disabled-checks` included:

```
check-thresholds:

  server-003:              # was SERVER_003

    cpu_percent: 80



notifications:

  webhook:

    - name: "Relay"

      url: "https://example.com/webhook"

      subscriptions:

        - server-001       # was SERVER_001
```

Codes also appear outside the config file, where nothing validates them for you: the Alertmanager `alertname`, the finding feed's subject token, and `hx.check_findings.code` in saved SQL and agent prompts.

### Eight checks were retired

Seven of them have no replacement and no new spelling, so a config entry naming one fails startup as an unknown code; remove it. Their stored findings are dropped rather than remapped.

`CHANGE_003`, `JETSTREAM_007`, `OPT_IDLE_006`, `OPT_PLACE_002`, `SERVER_006`, `SERVER_009`, `SERVER_017`.

Several were folded into another check rather than dropped outright. Their rows still go, because the absorbing check measures a different threshold and inheriting the old findings would attribute them to the wrong measurement.

The eighth, `server-049` (Leafnode Auto Compression with High Count, was `OPT_SYS_017`), is retired after the rename: `leafz` reports the negotiated compression, not the configured `s2_auto` mode, so the check could not establish what it claimed. Its stored findings are kept and still display. An entry naming exactly `server-049` in `check-thresholds`, `disabled-checks`, `subscriptions` or `ignore` is accepted with a warning at startup and has no effect; `OPT_SYS_017` fails as an unknown code.

`OPT_PLACE_004` is a related case: it continues as `account-014`, but its stored findings do not carry over. The old check recorded a server's key where the new one records an account's, so the next evaluation writes them correctly instead.

### Some thresholds take typed values

A threshold that encoded its unit in the parameter name now takes a value that states it. Durations take Go duration strings, sizes take IEC binary strings or a plain byte count, and percentages end in `_percent` and take 0 to 100. This makes an override readable on its own. `insights config check` names any parameter whose name needs updating, but not a value written for the old unit, so convert values yourself: `min_mono_frac: 0.8` becomes `min_mono_percent: 80`.

| Check (v1.2.0)             | Was                                                          | Now                                                                                                         |
| -------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------- |
| `connection-001`           | `rtt_ms: 100`                                                | `rtt: 100ms`                                                                                                |
| `connection-002`           | `pending_mib: 1`                                             | `warn_percent: 75`                                                                                          |
| `connection-004`           | `idle_minutes: 5`                                            | `idle: 5m`                                                                                                  |
| `account-008`              | `max_size_kib: 64`                                           | `max_size: 64 KiB`                                                                                          |
| `account-011`              | `min_conn_pct: 10`                                           | `min_conn_percent: 10`                                                                                      |
| `account-012`              | `warn_percent`, `min_total_bytes`                            | `gateway_percent: 30`, `min_traffic: 1 MiB`                                                                 |
| `leafnode-002`             | `rtt_ms: 100`                                                | `rtt: 100ms`                                                                                                |
| `stream-010`               | `stale_minutes: 5`                                           | `stale: 5m`                                                                                                 |
| `stream-020`               | `consumer_leader_pct`                                        | `consumer_leader_percent`                                                                                   |
| `stream-022`               | `max_memory_mib: 100`                                        | `max_memory: 100 MiB`                                                                                       |
| `stream-023`               | `max_uncompressed_gib: 1`                                    | `max_uncompressed: 1 GiB`                                                                                   |
| `stream-028`               | `warn_window_minutes`, `crit_window_minutes`                 | `elevated_window: 1h`, `severe_window: 6h`                                                                  |
| `stream-031`               | `min_ack_wait_secs: 30`                                      | `min_ack_wait: 30s`                                                                                         |
| `stream-032`               | `min_mono_frac: 0.8`                                         | `min_mono_percent: 80`                                                                                      |
| `system-004`               | `window_minutes: 10`                                         | `window: 10m`                                                                                               |
| `server-020`               | `max_delta: 500`                                             | `churn_percent: 50`, `min_conns: 100`                                                                       |
| `server-037`               | `saturation_pct`                                             | `saturation_percent`                                                                                        |
| `server-041`, `server-042` | `pending_mib: 1`                                             | `pending: 1 MiB`                                                                                            |
| `server-050`               | `min_growth_pct`, `min_mono_frac`                            | `min_growth_percent`, `min_mono_percent`                                                                    |
| `server-051`               | `warn_bytes_gib`, `crit_bytes_gib`, `warn_pct`, `crit_pct`   | `warn_size: 10 GiB`, `crit_size: 50 GiB`, `warn_percent: 50`, `crit_percent: 80`                            |
| `server-063`               | `warn_seconds`, `crit_seconds`                               | `warn_duration: 5s`, `crit_duration: 30s`                                                                   |
| `consumer-019`             | `warn_abs`, `warn_multiplier`, `crit_abs`, `crit_multiplier` | `elevated_abs`, `elevated_multiplier`, `severe_abs`, `severe_multiplier`                                    |
| `server-010`               | `rtt_ms: 50`                                                 | `multiplier: 2`, `severe_multiplier: 3`, `min_rtt: 10ms`, measured against the rest of the cluster's routes |
| `server-018`               | `rtt_ms: 50`                                                 | `multiplier: 2`, `min_rtt: 25ms`, measured against the other gateways between the same two clusters         |

Two defaults moved, both toward fewer findings: `account-007` (was `OPT_ACCT_001`) warns at 90% rather than 85%, and `consumer-017` (was `OPT_SYS_003`) at 90% rather than 80%. If you had raised either one yourself, the override is no longer needed.

Several checks gained optional parameters whose defaults preserve current behavior, so there is nothing to set: `crit_percent` on `server-005` and `server-011`, `max_restarts` on `server-008`, `sustain` on `server-014` and `server-048`, and `sustain` with `baseline` on `system-009`.

### Some severities moved

Nothing to edit, though alert routing keyed on severity will catch a slightly different set.

* Warning to critical: `consumer-006`, `consumer-007`, `consumer-008`, `consumer-009`, `consumer-011`.
* Critical to warning: `server-014` (critical again for a disabled JetStream, or once a recovering meta layer outlasts `sustain`), `system-007` (critical again when offline peers leave the meta group below quorum).
* Critical to info: `server-007`, `consumer-004`, `consumer-005`, `stream-009`.
* Warning to info: `server-002`, `server-010`, `server-048`, `consumer-018`.

`server-010` and `server-048` can escalate from info the same way.

### Nine checks are new

`server-053`, `server-054`, `server-055`, `server-056`, `server-064`, `server-065`, `stream-033`, `stream-034`, `consumer-020`. They are on by default. Run `insights checks info <code>` for what each one measures, or see the checks reference.

`server-065` is the one most likely to have something to say on an existing deployment: it reports a server with no `varz` reply in an epoch where another server replied and a peer still routes to it. `stream-034` and `consumer-020` report `io.nats.monitor.*` metadata values that are invalid for their key.

### Full code mapping

`insights checks list` prints the current set. The mapping below is the one the migration applies, so it is also how your stored history was rewritten.

| v0.1.10           | v1.2.0           | Note                                                    |
| ----------------- | ---------------- | ------------------------------------------------------- |
| `ACCOUNTS_001`    | `account-001`    |                                                         |
| `ACCOUNTS_002`    | `account-002`    |                                                         |
| `ACCOUNTS_003`    | `account-003`    |                                                         |
| `ACCOUNTS_004`    | `account-004`    |                                                         |
| `ACCOUNTS_005`    | `account-005`    |                                                         |
| `ACCOUNTS_006`    | `account-006`    |                                                         |
| `CHANGE_001`      | `server-023`     |                                                         |
| `CHANGE_002`      | `server-024`     |                                                         |
| `CHANGE_003`      | retired          | the check no longer exists; stored findings are dropped |
| `CHANGE_004`      | `stream-001`     |                                                         |
| `CLUSTER_001`     | `server-057`     |                                                         |
| `CLUSTER_003`     | `server-059`     |                                                         |
| `CLUSTER_004`     | `server-060`     |                                                         |
| `CLUSTER_005`     | `server-061`     |                                                         |
| `CLUSTER_006`     | `server-020`     |                                                         |
| `CLUSTER_007`     | `server-021`     |                                                         |
| `CLUSTER_008`     | `server-022`     |                                                         |
| `CONN_001`        | `connection-001` |                                                         |
| `CONN_002`        | `connection-002` |                                                         |
| `CONN_003`        | `connection-003` |                                                         |
| `CONSUMER_001`    | `consumer-001`   |                                                         |
| `CONSUMER_002`    | `consumer-002`   |                                                         |
| `CONSUMER_003`    | `consumer-003`   |                                                         |
| `CONSUMER_004`    | `consumer-004`   |                                                         |
| `CONSUMER_005`    | `consumer-005`   |                                                         |
| `CONSUMER_006`    | `consumer-006`   |                                                         |
| `CONSUMER_007`    | `consumer-007`   |                                                         |
| `CONSUMER_008`    | `consumer-008`   |                                                         |
| `CONSUMER_009`    | `consumer-009`   |                                                         |
| `CONSUMER_010`    | `consumer-010`   |                                                         |
| `CONSUMER_011`    | `consumer-011`   |                                                         |
| `CONSUMER_012`    | `consumer-012`   |                                                         |
| `JETSTREAM_001`   | `stream-002`     |                                                         |
| `JETSTREAM_002`   | `stream-003`     |                                                         |
| `JETSTREAM_003`   | `stream-004`     |                                                         |
| `JETSTREAM_004`   | `server-062`     |                                                         |
| `JETSTREAM_005`   | `server-025`     |                                                         |
| `JETSTREAM_006`   | `system-003`     |                                                         |
| `JETSTREAM_007`   | retired          | the check no longer exists; stored findings are dropped |
| `JETSTREAM_008`   | `stream-007`     |                                                         |
| `JETSTREAM_009`   | `server-028`     |                                                         |
| `JETSTREAM_010`   | `stream-005`     |                                                         |
| `JETSTREAM_011`   | `stream-006`     |                                                         |
| `JETSTREAM_013`   | `stream-008`     |                                                         |
| `JETSTREAM_014`   | `stream-009`     |                                                         |
| `JETSTREAM_015`   | `stream-010`     |                                                         |
| `JETSTREAM_017`   | `stream-011`     |                                                         |
| `JETSTREAM_018`   | `stream-012`     |                                                         |
| `JETSTREAM_019`   | `stream-013`     |                                                         |
| `JETSTREAM_020`   | `stream-014`     |                                                         |
| `JETSTREAM_021`   | `stream-015`     |                                                         |
| `JETSTREAM_022`   | `stream-016`     |                                                         |
| `JETSTREAM_023`   | `stream-017`     |                                                         |
| `JETSTREAM_024`   | `stream-018`     |                                                         |
| `JETSTREAM_025`   | `stream-019`     |                                                         |
| `LEAF_001`        | `leafnode-001`   |                                                         |
| `LEAF_002`        | `leafnode-002`   |                                                         |
| `LEAF_003`        | `leafnode-003`   |                                                         |
| `META_001`        | `server-058`     |                                                         |
| `META_002`        | `system-011`     |                                                         |
| `META_003`        | `system-004`     |                                                         |
| `META_004`        | `server-063`     |                                                         |
| `META_005`        | `system-006`     |                                                         |
| `META_006`        | `system-007`     |                                                         |
| `META_007`        | `system-008`     |                                                         |
| `META_008`        | `server-030`     |                                                         |
| `META_009`        | `system-009`     |                                                         |
| `OPT_ACCT_001`    | `account-007`    |                                                         |
| `OPT_ACCT_002`    | `account-008`    |                                                         |
| `OPT_BALANCE_001` | `server-031`     |                                                         |
| `OPT_BALANCE_002` | `server-032`     |                                                         |
| `OPT_BALANCE_003` | `server-033`     |                                                         |
| `OPT_BALANCE_004` | `server-034`     |                                                         |
| `OPT_BALANCE_005` | `server-035`     |                                                         |
| `OPT_BALANCE_006` | `server-036`     |                                                         |
| `OPT_BALANCE_007` | `stream-020`     |                                                         |
| `OPT_BALANCE_008` | `server-037`     |                                                         |
| `OPT_COST_001`    | `stream-021`     |                                                         |
| `OPT_COST_002`    | `stream-022`     |                                                         |
| `OPT_COST_003`    | `server-038`     |                                                         |
| `OPT_COST_004`    | `stream-023`     |                                                         |
| `OPT_COST_005`    | `server-039`     |                                                         |
| `OPT_IDLE_001`    | `server-040`     |                                                         |
| `OPT_IDLE_002`    | `stream-024`     |                                                         |
| `OPT_IDLE_003`    | `consumer-013`   |                                                         |
| `OPT_IDLE_004`    | `consumer-014`   |                                                         |
| `OPT_IDLE_005`    | `account-009`    |                                                         |
| `OPT_IDLE_006`    | retired          | the check no longer exists; stored findings are dropped |
| `OPT_IDLE_007`    | `connection-004` |                                                         |
| `OPT_PLACE_001`   | `account-011`    |                                                         |
| `OPT_PLACE_002`   | retired          | the check no longer exists; stored findings are dropped |
| `OPT_PLACE_003`   | `account-012`    |                                                         |
| `OPT_PLACE_004`   | `account-014`    | findings not carried over                               |
| `OPT_SYS_001`     | `stream-025`     |                                                         |
| `OPT_SYS_002`     | `consumer-016`   |                                                         |
| `OPT_SYS_003`     | `consumer-017`   |                                                         |
| `OPT_SYS_004`     | `consumer-018`   |                                                         |
| `OPT_SYS_005`     | `server-041`     |                                                         |
| `OPT_SYS_006`     | `leafnode-004`   |                                                         |
| `OPT_SYS_007`     | `server-043`     |                                                         |
| `OPT_SYS_008`     | `account-013`    |                                                         |
| `OPT_SYS_009`     | `server-044`     |                                                         |
| `OPT_SYS_010`     | `server-045`     |                                                         |
| `OPT_SYS_011`     | `server-046`     |                                                         |
| `OPT_SYS_012`     | `server-047`     |                                                         |
| `OPT_SYS_013`     | `server-048`     |                                                         |
| `OPT_SYS_014`     | `server-042`     |                                                         |
| `OPT_SYS_015`     | `consumer-019`   |                                                         |
| `OPT_SYS_016`     | `stream-026`     |                                                         |
| `OPT_SYS_017`     | `server-049`     | retired after the rename; stored findings kept          |
| `OPT_SYS_018`     | `stream-027`     |                                                         |
| `OPT_SYS_019`     | `stream-028`     |                                                         |
| `OPT_SYS_020`     | `stream-029`     |                                                         |
| `OPT_SYS_021`     | `stream-030`     |                                                         |
| `OPT_SYS_022`     | `server-050`     |                                                         |
| `OPT_SYS_023`     | `server-051`     |                                                         |
| `OPT_SYS_024`     | `stream-031`     |                                                         |
| `OPT_SYS_025`     | `stream-032`     |                                                         |
| `OPT_SYS_026`     | `server-052`     |                                                         |
| `SERVER_001`      | `server-001`     |                                                         |
| `SERVER_002`      | `server-002`     |                                                         |
| `SERVER_003`      | `server-003`     |                                                         |
| `SERVER_004`      | `server-004`     |                                                         |
| `SERVER_005`      | `server-005`     |                                                         |
| `SERVER_006`      | retired          | the check no longer exists; stored findings are dropped |
| `SERVER_007`      | `server-007`     |                                                         |
| `SERVER_008`      | `server-008`     |                                                         |
| `SERVER_009`      | retired          | the check no longer exists; stored findings are dropped |
| `SERVER_010`      | `server-010`     |                                                         |
| `SERVER_011`      | `server-011`     |                                                         |
| `SERVER_012`      | `server-012`     |                                                         |
| `SERVER_013`      | `server-013`     |                                                         |
| `SERVER_014`      | `server-014`     |                                                         |
| `SERVER_015`      | `server-015`     |                                                         |
| `SERVER_016`      | `server-016`     |                                                         |
| `SERVER_017`      | retired          | the check no longer exists; stored findings are dropped |
| `SERVER_018`      | `server-018`     |                                                         |
| `SERVER_019`      | `server-019`     |                                                         |
| `SERVICE_001`     | `service-001`    |                                                         |
| `SERVICE_002`     | `service-002`    |                                                         |
| `USER_001`        | `user-001`       |                                                         |
| `USER_002`        | `user-002`       |                                                         |

## Data

Eighteen migrations run on first start: seven SQL migrations that add columns or a table, and eleven Go migrations that rewrite stored check codes, unwrap JSON columns, rename the findings table, backfill a table, add a column to the findings table, retire a check's macro, rewrite enum spellings, drop repeated account identity rows and drop the `audit` schema. Five of them change what an existing query returns, and all five are worth a look if you have saved SQL or Grafana panels.

### `hx.check_results` is now `hx.check_findings`

Every row in the table is an entity that tripped a check. The macros emit only what fails, so there is no pass row and no severity meaning "ok", and the rest of the domain already said finding. The table now matches.

The rename is a catalog operation and copies no rows, but no view is left behind under the old name, so a query naming `hx.check_results` needs updating. Find them with:

```
grep -rn 'check_results' /path/to/your/queries /path/to/grafana/dashboards
```

The `code` column is rewritten to the new spelling at the same time, so a query filtering `WHERE code = 'SERVER_001'` returns no rows rather than an error. Worth updating in the same pass.

### Check macros moved from `audit` to `checks` and `contexts`

The macros behind the checks lived in a schema named `audit`. They now live in two schemas named for what they hold, and a migration drops `audit`:

| v0.1.10                                             | v1.2.0                            |
| --------------------------------------------------- | --------------------------------- |
| `audit.check_accounts_001(...)`                     | `checks.account_001(...)`         |
| `audit.subjects_overlap(...)` and the other helpers | `checks.subjects_overlap(...)`    |
| `audit.context_stream_raft_peers(...)`              | `contexts.stream_raft_peers(...)` |

A check macro is the check's code with hyphens turned to underscores, so the [Full code mapping](#full-code-mapping) gives the new name. A threshold parameter the check renamed takes its new name, per [Some thresholds take typed values](#some-thresholds-take-typed-values). A query calling an `audit.` macro fails with a catalog error rather than returning nothing. Find them with:

```
grep -rn 'audit\.' /path/to/your/queries /path/to/grafana/dashboards
```

The discovery endpoints list the new schemas. A client that passes `{"schema": "audit"}` to `$INS.db.macros` or runs `insights db macros --schema audit` gets an empty list; ask for `checks` or `contexts` instead.

### JSON columns hold documents rather than encoded strings

A JSON column handed to the appender as an already-encoded string held that string encoded a second time: the JSON string `"{\"k\":\"v\"}"` rather than the object. `json_type()` reported `VARCHAR` and every `json_extract` path on it returned NULL. Two consumer enums that NATS Server renders as JSON literals kept their quotes the same way, so `priority_policy` never equalled `pinned_client`.

The migration unwraps each affected column in place:

| Table                   | Columns                                                   |
| ----------------------- | --------------------------------------------------------- |
| `hx.server_ident`       | `metadata`                                                |
| `hx.stream_opts`        | `metadata`, `mirror_subject_transforms`                   |
| `hx.stream_source_opts` | `subject_transforms`                                      |
| `hx.consumer_opts`      | `metadata`, `backoff`, `priority_policy`, `replay_policy` |

These columns now behave the way their documentation always described, so a plain `json_extract(metadata, '$.key')` works. If you had written a workaround, usually a nested extract or a comparison against a quoted literal like `priority_policy = '"pinned_client"'`, replace it with the direct form.

### Enum columns hold the NATS API spelling

The indexer stored stream and consumer enums in Go's display spelling, which is not what the NATS API sends, and several checks compared against the API spelling and never matched. It now stores the API spelling, and a migration rewrites the recognized values already stored. Empty, NULL and unrecognized values are left as they are.

| Columns                                                              | Was                               | Now                               |
| -------------------------------------------------------------------- | --------------------------------- | --------------------------------- |
| `hx.stream_ident.storage_type`, `hx.consumer_opts.storage_type`      | `File`, `Memory`                  | `file`, `memory`                  |
| `hx.stream_opts.retention_policy`                                    | `Limits`, `Interest`, `WorkQueue` | `limits`, `interest`, `workqueue` |
| `hx.stream_opts.discard_policy`                                      | `DiscardOld`, `DiscardNew`        | `old`, `new`                      |
| `hx.stream_opts.store_compression`                                   | `None`, `S2`                      | `none`, `s2`                      |
| `hx.stream_opts.persist_mode`                                        | `Default`, `Async`                | `default`, `async`                |
| `hx.consumer_opts.replay_policy`, `hx.consumer_opts.priority_policy` | quoted, such as `"instant"`       | unquoted                          |

A saved query such as `WHERE storage_type = 'File'` returns no rows after the upgrade rather than an error; switch it to the lowercase value. Newly indexed rows also record the API's defaults where the server omits the field, so `persist_mode` reads `default` and `priority_policy` reads `none`. `hx.consumer_opts.storage_type` is still the parent stream's storage; the consumer page shows the consumer's effective storage separately.

### Raft state is uppercase

`hx.raft_groups.state` holds the value NATS Server reports: `LEADER`, `FOLLOWER`, `CANDIDATE`, `CLOSED`. The schema documentation and several checks had described it as `Leader` and `Follower`, which matched nothing. They now agree with what is stored, so the checks reading raft state report correctly.

The stored data is unchanged. A query of your own comparing this column against a capitalized value was returning no rows before and still does; switch it to uppercase.

### Repeated account identity rows are removed

`hx.account_ident` holds one row per account identity. A database indexed before that rule was enforced carries copies, and retention never removed them while the account stayed active. Every join on `pk` multiplied an account's rows by the number of copies it had, so any account total built that way was inflated.

The migration keeps the first observation of each `(pk, name)` and deletes the rest. A query of your own that joins `hx.account_ident` will return smaller numbers afterwards; those are the correct ones. A saved account panel whose values drop on upgrade is showing this, not a loss of data.

### New columns

All are additive with defaults, so nothing existing changes behavior. Each one answers a question that previously had no column.

| Table                     | Column                                                      | What it records                                                                                                                                                                          |
| ------------------------- | ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `hx.epochs`               | `servers_expected`, `servers_observed`, `partial_endpoints` | How much of the fleet a scrape reached. NULL means unknown, which should not be read as complete.                                                                                        |
| `hx.server_opts`          | `config_digest`                                             | Hash of the running configuration, which separates a restart with the same config from a reconfiguration. `''` means unreported, as with NATS Server before 2.11.                        |
| `hx.server_opts`          | `route_pool_size`                                           | Route connection pool size. `0` means unreported.                                                                                                                                        |
| `hx.server_stats`         | `statsz_fallback`                                           | True when the row came from STATSZ because the server's `varz` was stalled. Such a row carries only server-level stats, so `leafs`, `remotes` and `gateways` are NULL rather than zero.  |
| `hx.server_health_stats`  | `error_accounts`, `error_streams`, `error_consumers`        | Which assets a `healthz` error named.                                                                                                                                                    |
| `hx.raft_group_stats`     | `wal_error`                                                 | The write-ahead log error `raftz` reports for a group. `''` means none.                                                                                                                  |
| `hx.check_findings`       | `observed_epoch`                                            | The last epoch that confirmed the finding. NULL means the row's own `epoch`; an earlier value marks an `account-005` or `service-002` finding held through incomplete subscription data. |
| `hx.account_js_tier_opts` | new table                                                   | One row per JetStream limit set an account JWT declares, keyed by replication tier. Tier `''` is the untiered block. Backfilled from stored claims.                                      |

If you aggregate over `hx.server_stats`, either exclude `statsz_fallback` rows or treat their NULL columns as missing, so a fallback row does not read as a server with zero leafnodes.

### Retention

`db.retention.interval` is a minimum rather than a fixed period. A sweep runs on the indexer right after the first epoch committed past the interval, and never at the same time as a commit. Nothing to change; sweep timestamps are no longer evenly spaced.

## Interfaces

### Subjects name the system or node they address

With several systems on one node, every `$INS` subject carries the id of the system or node it addresses. `$INS.ping` stays flat, since discovery is the one case where every instance should answer on the same subject.

| Was                                                                                                                          | Now                                        |
| ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ |
| `$INS.db.*` (`query`, `query.stream`, `explain`, `schemas`, `tables`, `columns`, `macros`)                                   | `$INS.sys.<system>.db.*`                   |
| `$INS.db.backup`                                                                                                             | `$INS.sys.<system>.db.backup.stream`       |
| `$INS.checks.*`                                                                                                              | `$INS.sys.<system>.checks.*`               |
| `$INS.ops.info`, `ops.sizing`, `ops.memory`, `ops.storage`, `ops.scraper.toggle`, `ops.scraper.trigger`, `ops.diagnostics.*` | `$INS.sys.<system>.ops.*`                  |
| `$INS.ops.scraper`                                                                                                           | `$INS.sys.<system>.ops.scraper.status`     |
| `$INS.ops.events.history`                                                                                                    | `$INS.sys.<system>.ops.disconnect.history` |
| `$INS.ops.pprof`, `ops.nats.info`, `ops.version`, `ops.version.check`                                                        | `$INS.node.<node>.ops.*`                   |
| `$INS.epoch.committed`                                                                                                       | `$INS.sys.<system>.feed.epoch`             |
| `$INS.events.disconnect`                                                                                                     | `$INS.sys.<system>.feed.disconnect`        |
| `$INS.ops.memory.profile`                                                                                                    | `$INS.sys.<system>.feed.memory`            |

A useful side effect is that several Insights deployments can share one NATS account: each has its own subject space, so their streams and endpoints cannot be mistaken for one another.

The CLI and web tier discover an id before they call, so they need nothing from you. A client you wrote against the flat subjects needs the id added. `insights system list` and `insights node list` show what is reachable. See the API reference.

### A named sink stream needs a new name

Scrapes are published under `$INS.sys.<system>.scrape.>`. v0.1.10 created its sink stream capturing `<stream>.>`, and a stream that already exists is used as found, never edited. With the default name nothing collides: 1.2.0 creates a fresh `scrape_<system>` stream and the v0.1.10 `scrape` stream is left unused. With a name set through `sink.stream`, 1.2.0 finds the v0.1.10 stream under that name, the scraper refuses its subject, and nothing new is indexed. The log shows it once at startup:

```
level=ERROR msg="scraper runner error" ... error="stream \"<stream>\" should have subject \"$INS.sys.<system>.scrape.>\""
```

Check before you restart:

```
grep -n 'stream:' config.yaml     # under sink:

env | grep INSIGHTS_SINK_STREAM
```

If either names a stream, remove the setting (1.2.0 then uses `scrape_<system>`) or change it to a name no stream has yet. The indexer tracks its position per stream name, so a new name starts at the beginning of the new stream. Once epochs are arriving, delete the old stream with `nats stream rm <stream>`, and the unused `scrape` stream too on a node that ran with the default.

### Sizing reports memory in GiB

The `ops.sizing` reply renames `total_gb`, `query_gb`, `write_gb` and `index_gb` to `total_gib`, `query_gib`, `write_gib` and `index_gib`. The values were already whole GiB. `insights ops sizing` prints its recommendation as `--db.memory-limit=<n>GiB`, where it used to print `GB` and so applied about 7% less than it recommended.

### The HTTP Gateway names the system in the path

```
POST /sys/<system>/db/query      # was POST /db/query

GET  /systems                    # the systems this node serves
```

Every `/db/*` route moves the same way: `columns`, `explain`, `macros`, `query`, `schemas` and `tables`. The path token is the same id as the subject token. There is no default system and no cached topology, so a request is always answered by the system it named.

### MCP tools take a `system` argument

Every Insights MCP tool except `systems` requires a `system` argument, and a call without one is refused before it runs. An agent working interactively resolves it by calling `systems` first; a saved prompt or script that calls tools directly needs the id added.

### Alert payloads

`entity_name` is an annotation rather than a label, so a name that resolves a few epochs after a finding first fires no longer changes the Alertmanager fingerprint and fires a stable alert again. Routes, inhibitions and silences that match `entity_name` stop matching; match `entity_key` or `entity_pk`, which are still labels. Every alert and the payload's group labels carry `system`.

Alertmanager endpoints now receive every standing finding as firing once a minute, with `endsAt` four minutes out, so Alertmanager resolves an alert on its own when Insights stops sending it. Delivery to an endpoint is at-least-once: a failed delivery is retried until it succeeds, so a receiver can occasionally see a transition twice.

Every check's `description` and `remediation` annotation is reworded in this release to state what the check's SQL applies rather than a default an operator may have changed. Both are annotations rather than labels, so no fingerprint moves and no alert re-fires, but a receiver that templates or greps that text sees different words.

### Prometheus series carry a `system` label

Every metric family is per-system, since one node can own several systems with their own catalog, indexer, scraper and epoch clock, and without the label their samples would collide into one series.

On a single-system node the label has one value, so a `sum()` gives the same number as before. `topk` and label joins behave differently, so add `by (system)` where a panel is meant to be per-system.

New series: `insights_epoch_servers_expected`, `insights_epoch_servers_observed`, `insights_epoch_partial_endpoints`, and `insights_metrics_source_up` on an aggregating endpoint, where every series also gains a `source` label. See Metrics and [Grafana](/insights/1.2/guides/grafana.md).

### Saved search and filter expressions

List pages and the search box share one parser, which changes two things a saved expression or bookmarked filter can depend on. `and`, `or` and `not` are operators in any case, so a bare lowercase value spelled that way needs quoting. `check:` takes the new codes, so `check:SERVER_003` matches nothing; write `check:server-003`.

### Web URLs include the system

Every page lives under `/systems/<id>/…`. On a node serving one system, old paths redirect into it and preserve the query string, so existing bookmarks and deep links keep working. On a node serving several, an old path opens the system list. Tooling that constructs Insights URLs is worth updating to the new form.

## Operations

### The Insights image has no shell

The Linux images on `registry.synadia.io` (`insights`, `insights-licensed` and the `-collector` tag) are built on distroless static, which carries CA certificates, time zone data and a handful of OS packages, and no shell or package manager, so scanners have far less to report against them. The Insights image still runs as UID 1000 with `/var/lib/insights` owned by that UID, as in v0.1.10, so a data volume that worked before needs nothing. The collector image is new in this release, built the same way, and has nothing to migrate.

What changes is anything that runs a command inside the container: `docker exec <container> sh`, a Kubernetes exec probe calling `sh -c`, or an init container that reuses the Insights image to `chown` a volume. Point probes at `GET /healthz` instead, and use a small general-purpose image for init steps.

### `ops backup` builds the file on your machine

Saving is `insights ops backup save`, still the default subcommand, so an existing invocation keeps working. The copy streams table by table and is assembled locally, so the disk it needs is yours rather than the server's, and a backup no longer competes for space on the instance. It runs at the same Insights version as the server, and a version difference is reported up front rather than producing a file you cannot restore.

Three things changed with it:

* **`--output -` is no longer available.** The copy writes into a seekable DuckDB file, so it cannot be piped until complete. A script doing `insights ops backup --output - | gzip > b.gz` should write the file and compress it afterwards.
* **The endpoint is `$INS.sys.<system>.db.backup.stream`**, in place of `$INS.db.backup`.
* **The `backup:` config section is gone**, since backups are no longer uploaded to an object store.

The command confirms the size at a terminal, so add `--force` for anything running from cron or CI.

```
insights ops backup save --force --output /backups/insights-$(date +%F).db
```

`insights ops backup view <file>` serves the web UI against a backup, opened read-only, which is a convenient way to check one.

### Run under a restart policy

A DuckDB recovery attempt that cannot return within 30 seconds now exits so a supervisor can restart the process, rather than leaving listeners serving a database that cannot answer. Under a Kubernetes Deployment, a systemd unit with `Restart=`, or `--restart` on a container, this recovers on its own. Without one, the process stays down where it would previously have stayed up in a state it could not serve from.

### Naming the target makes client commands faster

With nothing naming a target, a client subcommand waits out the two-second `$INS.ping` discovery window so every responder has a chance to reply. Set `--system` (or `INSIGHTS_SYSTEM`, or `system.id` in a shared config file) for repeated invocations. A named target skips discovery, which also means silence from it is reported as the error it is.

Discovery needs permission to publish `$INS.ping` and receive replies; a named call needs only its endpoint permissions.

### Notification endpoint state left JetStream

Endpoint config is no longer mirrored into a JetStream KV bucket, and the delivery log moved to a stream named for the node (`webhook-deliveries_<nodeID>`). Nothing removes what v0.1.10 created: the `webhook-endpoints` KV bucket and the `webhook-deliveries` stream stay behind in the sink's JetStream, unused. Remove them once the node is on 1.2.0:

```
nats kv del webhook-endpoints

nats stream rm webhook-deliveries
```

If your NATS permissions granted either one explicitly, that grant is unused too.

### Agent skill downloads moved

The info dialog no longer builds and serves the per-agent skill bundles. Install them from [synadia-io/insights](https://github.com/synadia-io/insights), which is where the plugin marketplace and Gemini extension already lived.

## After you upgrade

**Confirm your checks resolve.** Startup would have reported an unknown code, so this is mostly about a check you meant to disable and spelled correctly for the wrong entity.

```
insights config check

insights checks list
```

**Give the new critical severities a day.** Five consumer checks moved from warning to critical, so a pager routed on critical may see traffic it did not before. `server-065` is also new and reports a server with no `varz` reply in an epoch where a peer still routes to it.

**Check that epochs are complete.** The new completeness columns show whether a scrape reached the whole fleet, which is useful to establish as a baseline.

```
SELECT epoch, servers_expected, servers_observed, partial_endpoints

FROM hx.epochs

WHERE servers_observed < servers_expected

ORDER BY epoch DESC LIMIT 20;
```

### Going back

The database moves forward on first start: the findings table is renamed and rewritten, JSON columns are unwrapped, enum values are rewritten, repeated account identity rows are deleted, the `audit` schema is dropped, and columns and a table are added. A v0.1.10 binary will not find `hx.check_results` in a v1.2.0 catalog.

Going back means restoring the backup from [Before you upgrade](#before-you-upgrade) into a fresh data directory, pointing the old binary at it, and letting it scrape forward again. Epochs indexed since the upgrade are not carried back. Keep the pre-upgrade config file alongside the backup, since the new check codes are not valid on v0.1.10 either.
