Upgrading to v1.2.0

v1.2.0 introduces multi-system support: one Insights node can monitor several NATS systems at once, each with its own database, API subtree and web pages. That shows up in several places in this guide, because anything that used to be implicit now names the system it belongs to.

Most of the upgrade takes care of itself. The database migrates on first start and your check history comes with it, and a single-system config file from v0.1.10 starts as it is: keys that no longer exist are ignored rather than rejected.

Two settings are worth a look before you restart, and only if you use them. Check codes have clearer names, so a check-thresholds override, a disabled-checks entry or a notification subscriptions or ignore entry that names a code needs the new name. A handful of threshold parameters were also renamed, and some take typed values like 100ms and 1 MiB rather than a bare number with the unit in the parameter name.

insights config check finds those names against your existing config without starting anything, so run it first. It reports one group of problems at a time, so run it again after each round of edits until it prints configuration is valid. It checks names, not values: a new threshold value is still yours to read against the tables below.

Does this affect you?

Each row is something you can check in a few seconds.

Area	Affects you if	Go to
Check codes	grep -nE '\b[A-Z]+_[0-9]{3}\b' config.yaml returns anything	Checks
Check parameters	a check-thresholds override names a parameter ending _ms, _secs, _minutes, _mib, _gib, _kib, _pct or _frac	Checks
Notification config	grep -n 'max-retries' config.yaml returns a line, or an endpoint sets timeout:	Configuration
Backup automation	a script runs insights ops backup with --output -, or publishes to $INS.db.backup	Operations
Saved SQL and dashboards	a query names hx.check_results or an audit. macro, calls json_extract on metadata, backoff, mirror_subject_transforms or subject_transforms, or compares storage_type, retention_policy, discard_policy, store_compression or persist_mode against a capitalized value	Data
NATS clients	a tool you wrote publishes to $INS.db.*, $INS.checks.* or $INS.ops.*	Interfaces
Sizing automation	a script reads total_gb, query_gb, write_gb or index_gb from ops.sizing	Interfaces
HTTP gateway callers	a caller posts to a /db/... path	Interfaces
MCP automations	a saved prompt or script calls an Insights MCP tool without a system argument	Interfaces
Alert receivers	an Alertmanager route, inhibition or silence, or a webhook consumer, matches the entity_name label	Interfaces
Prometheus and Grafana	a dashboard aggregates without grouping by system	Interfaces
Container tooling	a script, probe or init step runs a shell or package manager inside the Insights image	Operations
Sink stream name	sink.stream, --sink.stream or INSIGHTS_SINK_STREAM is set to anything other than scrape	Interfaces
Monitored NATS servers	any of them runs NATS Server older than 2.10.27	Before you upgrade
Telemetry	you run a trial build (a -licensed archive or the insights-licensed image) and would rather opt out	Configuration

Before you upgrade

1. Check the minimum NATS Server version. Insights v1.2.0 requires NATS Server 2.10.27 or later. Earlier 2.10 patches do not serve the raftz and ipqueuesz endpoints, so the checks that read them stay quiet rather than reporting an error, which is easy to miss.

nats server report jetstream   # or: curl -s http://<server>:8222/varz | jq .version

2. Keep a backup from your current version. The database moves forward on first start and a v0.1.10 binary cannot read a v1.2.0 catalog, so a copy taken now is what makes going back an option.

insights ops backup --output insights-pre-1.2.0.db

3. Validate your config with the new binary. This reads the config file and identity.json, and creates the data directory if it is missing, as serve would. It opens no database and starts no listener, so it is safe to run against a live deployment.

insights config check --config /etc/insights/config.yaml

Many configs pass unchanged. Where one does not, the report names the offending code or parameter. A code that changed only in case and separator names its replacement (SERVER_003 was renamed to server-003), and a parameter lists the valid names. A code whose number or entity changed prints unknown check code with no suggestion, so look it up in the mapping table. The report shows one group of problems per run, notification patterns before check codes, so repeat it until it prints configuration is valid. Add --strict to make unclaimed state a non-zero exit for a deploy gate.

4. Reconcile ids if the report mentions unclaimed state. Because a node can now serve several systems, a data directory records the node and system ids it was built under, which keeps each catalog matched to the system it holds. Renaming an id in config used to leave the old catalog behind and start an empty history; it is now something you declare. Against a stopped instance:

insights config migrate                              # adopt the new naming scheme
insights config migrate --rename system:west=north   # an id that actually changed
insights config migrate --prune system:retired-eu    # forget one, deleting nothing

With no flags this adopts the v1.2.0 naming scheme, which is the ordinary upgrade case: the ids held still and the names beneath them moved. Neither --rename nor --prune deletes data.

Configuration

Removed

Key	Environment variable	Note
backup.object-store-ttl	INSIGHTS_BACKUP_OBJECT_STORE_TTL	Backups are built locally now and are not uploaded anywhere.
web.metrics-timeout	INSIGHTS_WEB_METRICS_TIMEOUT	The component metrics poller they configured is off; nothing in Insights displayed it.
web.metrics-interval	INSIGHTS_WEB_METRICS_INTERVAL
notifications.*.timeout	none	Each request times out after 10s.
notifications.*.max-retries	none	Each delivery makes up to 3 attempts (one when the receiver answers with a 4xx other than 429), and a failed delivery is retried for that endpoint at up to one-minute intervals until it succeeds.

None of these stop a node from starting when they come from a config file or the environment: the keys are ignored, so remove them whenever it suits you. The --web.metrics-interval and --web.metrics-timeout flags are the exception; on a command line they fail with unknown flag.

Renamed

Old	New	Note
insights query	insights db query	The old spelling remains as an alias on the same command, so scripts keep working.
insights ops backup --output <file>	insights ops backup save --output <file>	save is the default subcommand, so the old invocation still works.

New

Node and system identity. Both subtrees are new, and they name the API space each part of the deployment answers on. system.id defaults to default and node.id defaults to the id of the single system the node serves, so a single-system deployment works without setting either. Set them when you want to address something by a name you chose.

node:
  id: "us-east"              # INSIGHTS_NODE_ID
  metadata:                  # labels for `insights node list` and the Nodes tab
    region: us-east-1
system:
  id: "prod"                 # INSIGHTS_SYSTEM_ID
  metadata:
    env: prod

See Architecture and Configuration.

Multiple systems. One node can own several monitored systems, each with its own DuckDB catalog, JetStream stream and $INS.sys.<id>.* API. Set systems: only on a node monitoring more than one system; it replaces system.id and the flat scraper config rather than sitting alongside them. A node serving more than one system has no single system id to derive its own from, so node.id is required there.

node:
  id: "central-1"
systems:
  - id: "core-prod"
    scrape:
      nats:
        server: "nats://core.internal:4222"
        creds: /etc/insights/core-prod.creds
      interval: 20s
    retention: 24h
  - id: "edge-west"          # collector-fed: no scrape block
    retention: 6h

See Deployment.

Collector deployments. insights-collector runs a scrape-and-stream collector with no local database, reading the same config file but using only the data-dir, node, system, sys, scraper, sink, license, updater and telemetry settings. A central node picks the system up by declaring it with no scrape block, as above. The collector's node.id defaults to its system.id, as does a central node's that serves only that system, and a second node claiming an id already in use refuses to start, so give them distinct node.id values. See Federated Deployment.

Turning a check off. disabled-checks is new. A disabled check keeps running and stays visible marked disabled, with its real findings; it is left out of health grades and is never delivered as a notification. It suits a check that cannot be acted on in your topology, rather than one whose signal you would still want to see.

disabled-checks:
  - account-012

Notification subscriptions take patterns. subscriptions and the new ignore list accept an exact code, a trailing-wildcard prefix, or *. ignore applies after subscriptions, so it carves exceptions out of a wildcard. An empty subscriptions list delivers nothing, so subscribing stays explicit.

notifications:
  webhook:
    - name: "PagerDuty Relay"
      url: "https://example.com/webhook"
      subscriptions:
        - server-*
        - stream-002
      ignore:
        - server-046
      systems:               # empty = every system this node owns
        - core-prod

systems: is new alongside multi-system support: it scopes an endpoint to particular systems, and an empty list delivers for all of them. See Notifications.

Sink stream shape. sink.replicas and sink.storage are new. Replicas above 1 need a clustered sink, since the embedded server is a single node. The stream is a buffer rather than the history, so storage: memory is a reasonable choice where losing the buffer on restart is acceptable.

Aggregated Prometheus endpoint. prometheus.aggregate serves every node's metrics from one endpoint, discovering nodes over $INS.ping, which saves adding a scrape target per node. Enable it on exactly one node: scraping two aggregators would report every series twice.

prometheus:
  aggregate: true
  timeout: 2s               # keep this shorter than your scrape timeout

See Metrics.

Inbox prefix. nats.inbox-prefix sets the request/reply inbox prefix, for accounts that are not granted _INBOX.>.

Telemetry, on by default in trial builds. A pulse every five minutes carries a locally generated random instance id, the release version and platform, a few configuration durations and counts, the size of the watched environment, how many checks have a finding, and usage counters for the web UI and for requests made by the CLI, coding agents, MCP and HTTP clients. It carries nothing about the monitored system: no server, cluster, account, subject, stream or consumer names, no addresses, no message data, no configuration values, no query text. Only trial builds carry the reporter; the Insights build contains none. insights web sends no pulse of its own but passes the page views it serves to the node holding each system's database, so set the same switch there to stop that.

telemetry:
  enabled: false            # or --telemetry=false, INSIGHTS_TELEMETRY=false
  print: true               # log every pulse and read exactly what would leave

See Telemetry.

Changed defaults

db.memory-limit defaults to 4GiB rather than 4GB. DuckDB reads GB as 10^9 bytes, so the old default capped each database at 3.7 GiB and the new one at 4.0 GiB. A value you set yourself keeps its meaning; write it in MiB or GiB to get the size it names.

Otherwise, anything you already set keeps its meaning. The subsystem switches (scraper, indexer, web, simulator, prometheus, updater) each gained an environment variable, so INSIGHTS_WEB=false works alongside INSIGHTS_WEB_ENABLED=false, and updater gained the --updater flag the others already had. The --<subsystem>.enabled flags still work but are hidden from --help. In YAML the section is still a map, so keep using its enabled: key.

Checks

Check codes now name the entity a finding is about. A finding on a server starts with server-, one on a stream with stream-, and so on, so the code tells you where to look before you read anything else. The previous codes grouped by subsystem instead, which is why CLUSTER_007 and META_001 both described servers. Codes are also lowercase and hyphenated throughout, matching how they appear in the UI and the API.

Your history comes with you. A migration rewrites stored findings onto the new codes in one transaction, so timelines stay unbroken and open findings keep their age.

The config file is the one place that may need an edit, because codes appear there by name: in check-thresholds, in disabled-checks, and in the subscriptions and ignore lists on a webhook or Alertmanager endpoint. If you set none of them, there is nothing to change.

Codes and parameters are validated at startup

A code matching no check is reported at startup with an error naming it, and the node does not start. A check-thresholds entry under an unrecognized code used to be passed over without comment, which meant a threshold you thought you had set was quietly doing nothing. Where only the spelling changed the error names the replacement, so SERVER_001 tells you to write server-001. Where the number or the entity changed, the new name cannot be derived from the old one and the error says only unknown check code, so look it up in the mapping table. A parameter name that no longer exists fails the same way and lists the valid names.

Every place a code appears in the config is checked this way, the top-level and per-system check-thresholds and disabled-checks included:

check-thresholds:
  server-003:              # was SERVER_003
    cpu_percent: 80

notifications:
  webhook:
    - name: "Relay"
      url: "https://example.com/webhook"
      subscriptions:
        - server-001       # was SERVER_001

Codes also appear outside the config file, where nothing validates them for you: the Alertmanager alertname, the finding feed's subject token, and hx.check_findings.code in saved SQL and agent prompts.

Eight checks were retired

Seven of them have no replacement and no new spelling, so a config entry naming one fails startup as an unknown code; remove it. Their stored findings are dropped rather than remapped.

CHANGE_003, JETSTREAM_007, OPT_IDLE_006, OPT_PLACE_002, SERVER_006, SERVER_009, SERVER_017.

Several were folded into another check rather than dropped outright. Their rows still go, because the absorbing check measures a different threshold and inheriting the old findings would attribute them to the wrong measurement.

The eighth, server-049 (Leafnode Auto Compression with High Count, was OPT_SYS_017), is retired after the rename: leafz reports the negotiated compression, not the configured s2_auto mode, so the check could not establish what it claimed. Its stored findings are kept and still display. An entry naming exactly server-049 in check-thresholds, disabled-checks, subscriptions or ignore is accepted with a warning at startup and has no effect; OPT_SYS_017 fails as an unknown code.

OPT_PLACE_004 is a related case: it continues as account-014, but its stored findings do not carry over. The old check recorded a server's key where the new one records an account's, so the next evaluation writes them correctly instead.

Some thresholds take typed values

A threshold that encoded its unit in the parameter name now takes a value that states it. Durations take Go duration strings, sizes take IEC binary strings or a plain byte count, and percentages end in _percent and take 0 to 100. This makes an override readable on its own. insights config check names any parameter whose name needs updating, but not a value written for the old unit, so convert values yourself: min_mono_frac: 0.8 becomes min_mono_percent: 80.

Check (v1.2.0)	Was	Now
connection-001	rtt_ms: 100	rtt: 100ms
connection-002	pending_mib: 1	warn_percent: 75
connection-004	idle_minutes: 5	idle: 5m
account-008	max_size_kib: 64	max_size: 64 KiB
account-011	min_conn_pct: 10	min_conn_percent: 10
account-012	warn_percent, min_total_bytes	gateway_percent: 30, min_traffic: 1 MiB
leafnode-002	rtt_ms: 100	rtt: 100ms
stream-010	stale_minutes: 5	stale: 5m
stream-020	consumer_leader_pct	consumer_leader_percent
stream-022	max_memory_mib: 100	max_memory: 100 MiB
stream-023	max_uncompressed_gib: 1	max_uncompressed: 1 GiB
stream-028	warn_window_minutes, crit_window_minutes	elevated_window: 1h, severe_window: 6h
stream-031	min_ack_wait_secs: 30	min_ack_wait: 30s
stream-032	min_mono_frac: 0.8	min_mono_percent: 80
system-004	window_minutes: 10	window: 10m
server-020	max_delta: 500	churn_percent: 50, min_conns: 100
server-037	saturation_pct	saturation_percent
server-041, server-042	pending_mib: 1	pending: 1 MiB
server-050	min_growth_pct, min_mono_frac	min_growth_percent, min_mono_percent
server-051	warn_bytes_gib, crit_bytes_gib, warn_pct, crit_pct	warn_size: 10 GiB, crit_size: 50 GiB, warn_percent: 50, crit_percent: 80
server-063	warn_seconds, crit_seconds	warn_duration: 5s, crit_duration: 30s
consumer-019	warn_abs, warn_multiplier, crit_abs, crit_multiplier	elevated_abs, elevated_multiplier, severe_abs, severe_multiplier
server-010	rtt_ms: 50	multiplier: 2, severe_multiplier: 3, min_rtt: 10ms, measured against the rest of the cluster's routes
server-018	rtt_ms: 50	multiplier: 2, min_rtt: 25ms, measured against the other gateways between the same two clusters

Two defaults moved, both toward fewer findings: account-007 (was OPT_ACCT_001) warns at 90% rather than 85%, and consumer-017 (was OPT_SYS_003) at 90% rather than 80%. If you had raised either one yourself, the override is no longer needed.

Several checks gained optional parameters whose defaults preserve current behavior, so there is nothing to set: crit_percent on server-005 and server-011, max_restarts on server-008, sustain on server-014 and server-048, and sustain with baseline on system-009.

Some severities moved

Nothing to edit, though alert routing keyed on severity will catch a slightly different set.

• Warning to critical: consumer-006, consumer-007, consumer-008, consumer-009, consumer-011.
• Critical to warning: server-014 (critical again for a disabled JetStream, or once a recovering meta layer outlasts sustain), system-007 (critical again when offline peers leave the meta group below quorum).
• Critical to info: server-007, consumer-004, consumer-005, stream-009.
• Warning to info: server-002, server-010, server-048, consumer-018.

server-010 and server-048 can escalate from info the same way.

Nine checks are new

server-053, server-054, server-055, server-056, server-064, server-065, stream-033, stream-034, consumer-020. They are on by default. Run insights checks info <code> for what each one measures, or see the checks reference.

server-065 is the one most likely to have something to say on an existing deployment: it reports a server with no varz reply in an epoch where another server replied and a peer still routes to it. stream-034 and consumer-020 report io.nats.monitor.* metadata values that are invalid for their key.

Full code mapping

insights checks list prints the current set. The mapping below is the one the migration applies, so it is also how your stored history was rewritten.

v0.1.10	v1.2.0	Note
ACCOUNTS_001	account-001
ACCOUNTS_002	account-002
ACCOUNTS_003	account-003
ACCOUNTS_004	account-004
ACCOUNTS_005	account-005
ACCOUNTS_006	account-006
CHANGE_001	server-023
CHANGE_002	server-024
CHANGE_003	retired	the check no longer exists; stored findings are dropped
CHANGE_004	stream-001
CLUSTER_001	server-057
CLUSTER_003	server-059
CLUSTER_004	server-060
CLUSTER_005	server-061
CLUSTER_006	server-020
CLUSTER_007	server-021
CLUSTER_008	server-022
CONN_001	connection-001
CONN_002	connection-002
CONN_003	connection-003
CONSUMER_001	consumer-001
CONSUMER_002	consumer-002
CONSUMER_003	consumer-003
CONSUMER_004	consumer-004
CONSUMER_005	consumer-005
CONSUMER_006	consumer-006
CONSUMER_007	consumer-007
CONSUMER_008	consumer-008
CONSUMER_009	consumer-009
CONSUMER_010	consumer-010
CONSUMER_011	consumer-011
CONSUMER_012	consumer-012
JETSTREAM_001	stream-002
JETSTREAM_002	stream-003
JETSTREAM_003	stream-004
JETSTREAM_004	server-062
JETSTREAM_005	server-025
JETSTREAM_006	system-003
JETSTREAM_007	retired	the check no longer exists; stored findings are dropped
JETSTREAM_008	stream-007
JETSTREAM_009	server-028
JETSTREAM_010	stream-005
JETSTREAM_011	stream-006
JETSTREAM_013	stream-008
JETSTREAM_014	stream-009
JETSTREAM_015	stream-010
JETSTREAM_017	stream-011
JETSTREAM_018	stream-012
JETSTREAM_019	stream-013
JETSTREAM_020	stream-014
JETSTREAM_021	stream-015
JETSTREAM_022	stream-016
JETSTREAM_023	stream-017
JETSTREAM_024	stream-018
JETSTREAM_025	stream-019
LEAF_001	leafnode-001
LEAF_002	leafnode-002
LEAF_003	leafnode-003
META_001	server-058
META_002	system-011
META_003	system-004
META_004	server-063
META_005	system-006
META_006	system-007
META_007	system-008
META_008	server-030
META_009	system-009
OPT_ACCT_001	account-007
OPT_ACCT_002	account-008
OPT_BALANCE_001	server-031
OPT_BALANCE_002	server-032
OPT_BALANCE_003	server-033
OPT_BALANCE_004	server-034
OPT_BALANCE_005	server-035
OPT_BALANCE_006	server-036
OPT_BALANCE_007	stream-020
OPT_BALANCE_008	server-037
OPT_COST_001	stream-021
OPT_COST_002	stream-022
OPT_COST_003	server-038
OPT_COST_004	stream-023
OPT_COST_005	server-039
OPT_IDLE_001	server-040
OPT_IDLE_002	stream-024
OPT_IDLE_003	consumer-013
OPT_IDLE_004	consumer-014
OPT_IDLE_005	account-009
OPT_IDLE_006	retired	the check no longer exists; stored findings are dropped
OPT_IDLE_007	connection-004
OPT_PLACE_001	account-011
OPT_PLACE_002	retired	the check no longer exists; stored findings are dropped
OPT_PLACE_003	account-012
OPT_PLACE_004	account-014	findings not carried over
OPT_SYS_001	stream-025
OPT_SYS_002	consumer-016
OPT_SYS_003	consumer-017
OPT_SYS_004	consumer-018
OPT_SYS_005	server-041
OPT_SYS_006	leafnode-004
OPT_SYS_007	server-043
OPT_SYS_008	account-013
OPT_SYS_009	server-044
OPT_SYS_010	server-045
OPT_SYS_011	server-046
OPT_SYS_012	server-047
OPT_SYS_013	server-048
OPT_SYS_014	server-042
OPT_SYS_015	consumer-019
OPT_SYS_016	stream-026
OPT_SYS_017	server-049	retired after the rename; stored findings kept
OPT_SYS_018	stream-027
OPT_SYS_019	stream-028
OPT_SYS_020	stream-029
OPT_SYS_021	stream-030
OPT_SYS_022	server-050
OPT_SYS_023	server-051
OPT_SYS_024	stream-031
OPT_SYS_025	stream-032
OPT_SYS_026	server-052
SERVER_001	server-001
SERVER_002	server-002
SERVER_003	server-003
SERVER_004	server-004
SERVER_005	server-005
SERVER_006	retired	the check no longer exists; stored findings are dropped
SERVER_007	server-007
SERVER_008	server-008
SERVER_009	retired	the check no longer exists; stored findings are dropped
SERVER_010	server-010
SERVER_011	server-011
SERVER_012	server-012
SERVER_013	server-013
SERVER_014	server-014
SERVER_015	server-015
SERVER_016	server-016
SERVER_017	retired	the check no longer exists; stored findings are dropped
SERVER_018	server-018
SERVER_019	server-019
SERVICE_001	service-001
SERVICE_002	service-002
USER_001	user-001
USER_002	user-002

Data

Eighteen migrations run on first start: seven SQL migrations that add columns or a table, and eleven Go migrations that rewrite stored check codes, unwrap JSON columns, rename the findings table, backfill a table, add a column to the findings table, retire a check's macro, rewrite enum spellings, drop repeated account identity rows and drop the audit schema. Five of them change what an existing query returns, and all five are worth a look if you have saved SQL or Grafana panels.

hx.check_results is now hx.check_findings

Every row in the table is an entity that tripped a check. The macros emit only what fails, so there is no pass row and no severity meaning "ok", and the rest of the domain already said finding. The table now matches.

The rename is a catalog operation and copies no rows, but no view is left behind under the old name, so a query naming hx.check_results needs updating. Find them with:

grep -rn 'check_results' /path/to/your/queries /path/to/grafana/dashboards

The code column is rewritten to the new spelling at the same time, so a query filtering WHERE code = 'SERVER_001' returns no rows rather than an error. Worth updating in the same pass.

Check macros moved from audit to checks and contexts

The macros behind the checks lived in a schema named audit. They now live in two schemas named for what they hold, and a migration drops audit:

v0.1.10	v1.2.0
audit.check_accounts_001(...)	checks.account_001(...)
audit.subjects_overlap(...) and the other helpers	checks.subjects_overlap(...)
audit.context_stream_raft_peers(...)	contexts.stream_raft_peers(...)

A check macro is the check's code with hyphens turned to underscores, so the Full code mapping gives the new name. A threshold parameter the check renamed takes its new name, per Some thresholds take typed values. A query calling an audit. macro fails with a catalog error rather than returning nothing. Find them with:

grep -rn 'audit\.' /path/to/your/queries /path/to/grafana/dashboards

The discovery endpoints list the new schemas. A client that passes {"schema": "audit"} to $INS.db.macros or runs insights db macros --schema audit gets an empty list; ask for checks or contexts instead.

JSON columns hold documents rather than encoded strings

A JSON column handed to the appender as an already-encoded string held that string encoded a second time: the JSON string "{\"k\":\"v\"}" rather than the object. json_type() reported VARCHAR and every json_extract path on it returned NULL. Two consumer enums that NATS Server renders as JSON literals kept their quotes the same way, so priority_policy never equalled pinned_client.

The migration unwraps each affected column in place:

Table	Columns
hx.server_ident	metadata
hx.stream_opts	metadata, mirror_subject_transforms
hx.stream_source_opts	subject_transforms
hx.consumer_opts	metadata, backoff, priority_policy, replay_policy

These columns now behave the way their documentation always described, so a plain json_extract(metadata, '$.key') works. If you had written a workaround, usually a nested extract or a comparison against a quoted literal like priority_policy = '"pinned_client"', replace it with the direct form.

Enum columns hold the NATS API spelling

The indexer stored stream and consumer enums in Go's display spelling, which is not what the NATS API sends, and several checks compared against the API spelling and never matched. It now stores the API spelling, and a migration rewrites the recognized values already stored. Empty, NULL and unrecognized values are left as they are.

Columns	Was	Now
hx.stream_ident.storage_type, hx.consumer_opts.storage_type	File, Memory	file, memory
hx.stream_opts.retention_policy	Limits, Interest, WorkQueue	limits, interest, workqueue
hx.stream_opts.discard_policy	DiscardOld, DiscardNew	old, new
hx.stream_opts.store_compression	None, S2	none, s2
hx.stream_opts.persist_mode	Default, Async	default, async
hx.consumer_opts.replay_policy, hx.consumer_opts.priority_policy	quoted, such as "instant"	unquoted

A saved query such as WHERE storage_type = 'File' returns no rows after the upgrade rather than an error; switch it to the lowercase value. Newly indexed rows also record the API's defaults where the server omits the field, so persist_mode reads default and priority_policy reads none. hx.consumer_opts.storage_type is still the parent stream's storage; the consumer page shows the consumer's effective storage separately.

Raft state is uppercase

hx.raft_groups.state holds the value NATS Server reports: LEADER, FOLLOWER, CANDIDATE, CLOSED. The schema documentation and several checks had described it as Leader and Follower, which matched nothing. They now agree with what is stored, so the checks reading raft state report correctly.

The stored data is unchanged. A query of your own comparing this column against a capitalized value was returning no rows before and still does; switch it to uppercase.

Repeated account identity rows are removed

hx.account_ident holds one row per account identity. A database indexed before that rule was enforced carries copies, and retention never removed them while the account stayed active. Every join on pk multiplied an account's rows by the number of copies it had, so any account total built that way was inflated.

The migration keeps the first observation of each (pk, name) and deletes the rest. A query of your own that joins hx.account_ident will return smaller numbers afterwards; those are the correct ones. A saved account panel whose values drop on upgrade is showing this, not a loss of data.

New columns

All are additive with defaults, so nothing existing changes behavior. Each one answers a question that previously had no column.

Table	Column	What it records
hx.epochs	servers_expected, servers_observed, partial_endpoints	How much of the fleet a scrape reached. NULL means unknown, which should not be read as complete.
hx.server_opts	config_digest	Hash of the running configuration, which separates a restart with the same config from a reconfiguration. '' means unreported, as with NATS Server before 2.11.
hx.server_opts	route_pool_size	Route connection pool size. 0 means unreported.
hx.server_stats	statsz_fallback	True when the row came from STATSZ because the server's varz was stalled. Such a row carries only server-level stats, so leafs, remotes and gateways are NULL rather than zero.
hx.server_health_stats	error_accounts, error_streams, error_consumers	Which assets a healthz error named.
hx.raft_group_stats	wal_error	The write-ahead log error raftz reports for a group. '' means none.
hx.check_findings	observed_epoch	The last epoch that confirmed the finding. NULL means the row's own epoch; an earlier value marks an account-005 or service-002 finding held through incomplete subscription data.
hx.account_js_tier_opts	new table	One row per JetStream limit set an account JWT declares, keyed by replication tier. Tier '' is the untiered block. Backfilled from stored claims.

If you aggregate over hx.server_stats, either exclude statsz_fallback rows or treat their NULL columns as missing, so a fallback row does not read as a server with zero leafnodes.

Retention

db.retention.interval is a minimum rather than a fixed period. A sweep runs on the indexer right after the first epoch committed past the interval, and never at the same time as a commit. Nothing to change; sweep timestamps are no longer evenly spaced.

Interfaces

Subjects name the system or node they address

With several systems on one node, every $INS subject carries the id of the system or node it addresses. $INS.ping stays flat, since discovery is the one case where every instance should answer on the same subject.

Was	Now
$INS.db.* (query, query.stream, explain, schemas, tables, columns, macros)	$INS.sys.<system>.db.*
$INS.db.backup	$INS.sys.<system>.db.backup.stream
$INS.checks.*	$INS.sys.<system>.checks.*
$INS.ops.info, ops.sizing, ops.memory, ops.storage, ops.scraper.toggle, ops.scraper.trigger, ops.diagnostics.*	$INS.sys.<system>.ops.*
$INS.ops.scraper	$INS.sys.<system>.ops.scraper.status
$INS.ops.events.history	$INS.sys.<system>.ops.disconnect.history
$INS.ops.pprof, ops.nats.info, ops.version, ops.version.check	$INS.node.<node>.ops.*
$INS.epoch.committed	$INS.sys.<system>.feed.epoch
$INS.events.disconnect	$INS.sys.<system>.feed.disconnect
$INS.ops.memory.profile	$INS.sys.<system>.feed.memory

A useful side effect is that several Insights deployments can share one NATS account: each has its own subject space, so their streams and endpoints cannot be mistaken for one another.

The CLI and web tier discover an id before they call, so they need nothing from you. A client you wrote against the flat subjects needs the id added. insights system list and insights node list show what is reachable. See the API reference.

A named sink stream needs a new name

Scrapes are published under $INS.sys.<system>.scrape.>. v0.1.10 created its sink stream capturing <stream>.>, and a stream that already exists is used as found, never edited. With the default name nothing collides: 1.2.0 creates a fresh scrape_<system> stream and the v0.1.10 scrape stream is left unused. With a name set through sink.stream, 1.2.0 finds the v0.1.10 stream under that name, the scraper refuses its subject, and nothing new is indexed. The log shows it once at startup:

level=ERROR msg="scraper runner error" ... error="stream \"<stream>\" should have subject \"$INS.sys.<system>.scrape.>\""

Check before you restart:

grep -n 'stream:' config.yaml     # under sink:
env | grep INSIGHTS_SINK_STREAM

If either names a stream, remove the setting (1.2.0 then uses scrape_<system>) or change it to a name no stream has yet. The indexer tracks its position per stream name, so a new name starts at the beginning of the new stream. Once epochs are arriving, delete the old stream with nats stream rm <stream>, and the unused scrape stream too on a node that ran with the default.

Sizing reports memory in GiB

The ops.sizing reply renames total_gb, query_gb, write_gb and index_gb to total_gib, query_gib, write_gib and index_gib. The values were already whole GiB. insights ops sizing prints its recommendation as --db.memory-limit=<n>GiB, where it used to print GB and so applied about 7% less than it recommended.

The HTTP gateway names the system in the path

POST /sys/<system>/db/query      # was POST /db/query
GET  /systems                    # the systems this node serves

Every /db/* route moves the same way: columns, explain, macros, query, schemas and tables. The path token is the same id as the subject token. There is no default system and no cached topology, so a request is always answered by the system it named.

MCP tools take a system argument

Every Insights MCP tool except systems requires a system argument, and a call without one is refused before it runs. An agent working interactively resolves it by calling systems first; a saved prompt or script that calls tools directly needs the id added.

Alert payloads

entity_name is an annotation rather than a label, so a name that resolves a few epochs after a finding first fires no longer changes the Alertmanager fingerprint and fires a stable alert again. Routes, inhibitions and silences that match entity_name stop matching; match entity_key or entity_pk, which are still labels. Every alert and the payload's group labels carry system.

Alertmanager endpoints now receive every standing finding as firing once a minute, with endsAt four minutes out, so Alertmanager resolves an alert on its own when Insights stops sending it. Delivery to an endpoint is at-least-once: a failed delivery is retried until it succeeds, so a receiver can occasionally see a transition twice.

Every check's description and remediation annotation is reworded in this release to state what the check's SQL applies rather than a default an operator may have changed. Both are annotations rather than labels, so no fingerprint moves and no alert re-fires, but a receiver that templates or greps that text sees different words.

Prometheus series carry a system label

Every metric family is per-system, since one node can own several systems with their own catalog, indexer, scraper and epoch clock, and without the label their samples would collide into one series.

On a single-system node the label has one value, so a sum() gives the same number as before. topk and label joins behave differently, so add by (system) where a panel is meant to be per-system.

New series: insights_epoch_servers_expected, insights_epoch_servers_observed, insights_epoch_partial_endpoints, and insights_metrics_source_up on an aggregating endpoint, where every series also gains a source label. See Metrics and Grafana.

Saved search and filter expressions

List pages and the search box share one parser, which changes two things a saved expression or bookmarked filter can depend on. and, or and not are operators in any case, so a bare lowercase value spelled that way needs quoting. check: takes the new codes, so check:SERVER_003 matches nothing; write check:server-003.

Web URLs include the system

Every page lives under /systems/<id>/…. On a node serving one system, old paths redirect into it and preserve the query string, so existing bookmarks and deep links keep working. On a node serving several, an old path opens the system list. Tooling that constructs Insights URLs is worth updating to the new form.

Operations

The Insights image has no shell

The Linux images on registry.synadia.io (insights, insights-licensed and the -collector tag) are built on distroless static, which carries CA certificates, time zone data and a handful of OS packages, and no shell or package manager, so scanners have far less to report against them. The Insights image still runs as UID 1000 with /var/lib/insights owned by that UID, as in v0.1.10, so a data volume that worked before needs nothing. The collector image is new in this release, built the same way, and has nothing to migrate.

What changes is anything that runs a command inside the container: docker exec <container> sh, a Kubernetes exec probe calling sh -c, or an init container that reuses the Insights image to chown a volume. Point probes at GET /healthz instead, and use a small general-purpose image for init steps.

ops backup builds the file on your machine

Saving is insights ops backup save, still the default subcommand, so an existing invocation keeps working. The copy streams table by table and is assembled locally, so the disk it needs is yours rather than the server's, and a backup no longer competes for space on the instance. It runs at the same Insights version as the server, and a version difference is reported up front rather than producing a file you cannot restore.

Three things changed with it:

• --output - is no longer available. The copy writes into a seekable DuckDB file, so it cannot be piped until complete. A script doing insights ops backup --output - | gzip > b.gz should write the file and compress it afterwards.
• The endpoint is $INS.sys.<system>.db.backup.stream, in place of $INS.db.backup.
• The backup: config section is gone, since backups are no longer uploaded to an object store.

The command confirms the size at a terminal, so add --force for anything running from cron or CI.

insights ops backup save --force --output /backups/insights-$(date +%F).db

insights ops backup view <file> serves the web UI against a backup, opened read-only, which is a convenient way to check one.

Run under a restart policy

A DuckDB recovery attempt that cannot return within 30 seconds now exits so a supervisor can restart the process, rather than leaving listeners serving a database that cannot answer. Under a Kubernetes Deployment, a systemd unit with Restart=, or --restart on a container, this recovers on its own. Without one, the process stays down where it would previously have stayed up in a state it could not serve from.

Naming the target makes client commands faster

With nothing naming a target, a client subcommand waits out the two-second $INS.ping discovery window so every responder has a chance to reply. Set --system (or INSIGHTS_SYSTEM, or system.id in a shared config file) for repeated invocations. A named target skips discovery, which also means silence from it is reported as the error it is.

Discovery needs permission to publish $INS.ping and receive replies; a named call needs only its endpoint permissions.

Notification endpoint state left JetStream

Endpoint config is no longer mirrored into a JetStream KV bucket, and the delivery log moved to a stream named for the node (webhook-deliveries_<nodeID>). Nothing removes what v0.1.10 created: the webhook-endpoints KV bucket and the webhook-deliveries stream stay behind in the sink's JetStream, unused. Remove them once the node is on 1.2.0:

nats kv del webhook-endpoints
nats stream rm webhook-deliveries

If your NATS permissions granted either one explicitly, that grant is unused too.

Agent skill downloads moved

The info dialog no longer builds and serves the per-agent skill bundles. Install them from synadia-io/insights, which is where the plugin marketplace and Gemini extension already lived.

After you upgrade

Confirm your checks resolve. Startup would have reported an unknown code, so this is mostly about a check you meant to disable and spelled correctly for the wrong entity.

insights config check
insights checks list

Give the new critical severities a day. Five consumer checks moved from warning to critical, so a pager routed on critical may see traffic it did not before. server-065 is also new and reports a server with no varz reply in an epoch where a peer still routes to it.

Check that epochs are complete. The new completeness columns show whether a scrape reached the whole fleet, which is useful to establish as a baseline.

SELECT epoch, servers_expected, servers_observed, partial_endpoints
FROM hx.epochs
WHERE servers_observed < servers_expected
ORDER BY epoch DESC LIMIT 20;

Going back

The database moves forward on first start: the findings table is renamed and rewritten, JSON columns are unwrapped, enum values are rewritten, repeated account identity rows are deleted, the audit schema is dropped, and columns and a table are added. A v0.1.10 binary will not find hx.check_results in a v1.2.0 catalog.

Going back means restoring the backup from Before you upgrade into a fresh data directory, pointing the old binary at it, and letting it scrape forward again. Epochs indexed since the upgrade are not carried back. Keep the pre-upgrade config file alongside the backup, since the new check codes are not valid on v0.1.10 either.