Upgrading to 1.2.0

Insights 1.2.0 introduces multi-system support: one Insights node can monitor several NATS systems at once, each with its own database, API subtree and web pages. That shows up in several places in this guide, because anything that used to be implicit now names the system it belongs to.

Most of the upgrade takes care of itself. The database migrates on first start and your check history comes with it, and a single-system config file from 0.1.10 starts as it is: keys that no longer exist are ignored rather than rejected.

Two settings are worth a look before you restart, and only if you use them. Check codes have clearer names, so a check-thresholds override, a disabled-checks entry or a notification subscriptions or ignore entry that names a code needs the new name. A handful of threshold parameters were also renamed, and some take typed values like 100ms and 1 MiB rather than a bare number with the unit in the parameter name.

insights config check finds those names against your existing config without starting anything, so run it first. It reports one group of problems at a time, so run it again after each round of edits until it prints configuration is valid. It checks names, not values: a new threshold value is still yours to read against the tables below.

Does this affect you?

Each row is something you can check in a few seconds.

Before you upgrade

1. Check the minimum NATS Server version. Insights 1.2.0 requires NATS Server 2.10.27 or later. Earlier 2.10 patches do not serve the raftz and ipqueuesz endpoints, so the checks that read them stay quiet rather than reporting an error, which is easy to miss.

nats server report jetstream   # or: curl -s http://<server>:8222/varz | jq .version

2. Keep a backup from your current version. The database moves forward on first start and a 0.1.10 binary cannot read a 1.2.0 catalog, so a copy taken now is what makes going back an option.

insights ops backup --output insights-pre-1.2.0.db

3. Validate your config with the new binary. Get the 1.2.0 binary as described in Installation. insights config check reads the config file and identity.json, and creates the data directory if it is missing, as serve would. It opens no database and starts no listener, so it is safe to run against a live deployment.

insights config check --config /etc/insights/config.yaml

Many configs pass unchanged. Where one does not, the report names the offending code or parameter. A code that changed only in case and separator names its replacement (SERVER_003 was renamed to server-003), and a parameter lists the valid names. A code whose number or entity changed prints unknown check code with no suggestion, so look it up in the mapping table. The report shows one group of problems per run, notification patterns before check codes, so repeat it until it prints configuration is valid. Add --strict to make unclaimed state a non-zero exit for a deploy gate. See insights config check.

4. Reconcile ids if the report mentions unclaimed state. Because a node can now serve several systems, a data directory records the node and system ids it was built under, which keeps each catalog matched to the system it holds. Renaming an id in config used to leave the old catalog behind and start an empty history; it is now something you declare. Against a stopped instance:

insights config migrate                              # adopt the new naming scheme
insights config migrate --rename system:west=north   # an id that actually changed
insights config migrate --prune system:retired-eu    # forget one, deleting nothing

With no flags this adopts the 1.2.0 naming scheme, which is the ordinary upgrade case: the ids held still and the names beneath them moved. Neither --rename nor --prune deletes data. See insights config migrate.

Configuration

Removed

None of these stop a node from starting when they come from a config file or the environment: the keys are ignored, so remove them whenever it suits you. The --web.metrics-interval and --web.metrics-timeout flags are the exception; on a command line they fail with unknown flag.

Renamed

New

Node and system identity. Both subtrees are new, and they name the API space each part of the deployment answers on. system.id defaults to default and node.id defaults to the id of the single system the node serves, so a single-system deployment works without setting either. Set them when you want to address something by a name you chose.

node:
  id: 'hub-us-east' # INSIGHTS_NODE_ID
  metadata: # labels for `insights node list` and the Nodes tab
    region: us-east-1
system:
  id: 'prod' # INSIGHTS_SYSTEM_ID
  metadata:
    env: prod

See Architecture and Configuration.

Multiple systems. One node can own several monitored systems, each with its own DuckDB catalog, JetStream stream and $INS.sys.<id>.* API. Set systems: only on a node monitoring more than one system; it replaces system.id and the flat scraper config rather than sitting alongside them. A node serving more than one system has no single system id to derive its own from, so node.id is required there.

node:
  id: 'hub-1'
systems:
  - id: 'core-prod'
    scrape:
      nats:
        server: 'nats://core.internal:4222'
        creds: /etc/insights/core-prod.creds
      interval: 20s
    retention: 24h
  - id: 'edge-west' # collector-fed: no scrape block
    retention: 6h

See Deployment.

Collectors. insights-collector is a new, smaller binary that only scrapes: it has no database and reads only the data-dir, node, system, sys, scraper, sink, license, updater and telemetry settings. Its default command, serve, publishes each scrape to a stream, and a central node picks the system up by declaring it with no scrape block, as above. The collector's node.id defaults to its system.id, and a second node claiming an id already in use refuses to start, so give the collector and the central node distinct node.id values. See Federated Deployment.

Turning a check off. disabled-checks is new. A disabled check keeps running and stays visible marked disabled, with its real findings; it is left out of health grades and is never delivered as a notification. It suits a check that cannot be acted on in your topology, rather than one whose signal you would still want to see.

disabled-checks:
  - account-012

See Configuration.

Notification subscriptions take patterns. subscriptions and the new ignore list accept an exact code, a trailing-wildcard prefix, or *. ignore applies after subscriptions, so it carves exceptions out of a wildcard. An empty subscriptions list delivers nothing, so subscribing stays explicit.

notifications:
  webhook:
    - name: 'PagerDuty Relay'
      url: 'https://example.com/webhook'
      subscriptions:
        - server-*
        - stream-002
      ignore:
        - server-046
      systems: # empty = every system this node owns
        - core-prod

systems: is new alongside multi-system support: it scopes an endpoint to particular systems, and an empty list delivers for all of them. See Notifications.

Sink stream shape. sink.replicas and sink.storage are new. Replicas above 1 need a clustered sink, since the embedded server is a single node. The stream is a buffer rather than the history, so storage: memory is a reasonable choice where losing the buffer on restart is acceptable.

Aggregated Prometheus endpoint. prometheus.aggregate serves every node's metrics from one endpoint, discovering nodes over $INS.ping, which saves adding a scrape target per node. Enable it on exactly one node: scraping two aggregators would report every series twice.

prometheus:
  aggregate: true
  timeout: 2s # keep this shorter than your scrape timeout

See Metrics.

Inbox prefix. nats.inbox-prefix sets the request/reply inbox prefix, for accounts that are not granted _INBOX.>.

Telemetry, on by default in trial builds. A pulse every five minutes carries a locally generated random instance id, the release version and platform, a few configuration durations and counts, the size of the watched environment, how many checks have a finding, and usage counters for the web UI and for requests made by the CLI, coding agents, MCP and HTTP clients. It carries nothing about the monitored system: no server, cluster, account, subject, stream or consumer names, no addresses, no message data, no configuration values, no query text. Only trial builds carry the reporter; the Insights build contains none. insights web sends no pulse of its own but passes the page views it serves to the node holding each system's database, so set the same switch there to stop that.

telemetry:
  enabled: false # or --telemetry=false, INSIGHTS_TELEMETRY=false
  print: true # log every pulse and read exactly what would leave

See Telemetry.

Changed defaults

db.memory-limit defaults to 4GiB rather than 4GB. DuckDB reads GB as 10^9 bytes, so the old default capped each database at 3.7 GiB and the new one at 4.0 GiB. A value you set yourself keeps its meaning; write it in MiB or GiB to get the size it names.

Otherwise, anything you already set keeps its meaning. The subsystem switches (scraper, indexer, web, simulator, prometheus, updater) each gained an environment variable, so INSIGHTS_WEB=false works alongside INSIGHTS_WEB_ENABLED=false, and updater gained the --updater flag the others already had. The --<subsystem>.enabled flags still work but are hidden from --help. In YAML the section is still a map, so keep using its enabled: key.

Checks

Check codes now name the entity a finding is about. A finding on a server starts with server-, one on a stream with stream-, and so on, so the code tells you where to look before you read anything else. The previous codes grouped by subsystem instead, which is why CLUSTER_007 and META_001 both described servers. Codes are also lowercase and hyphenated throughout, matching how they appear in the UI and the API.

Your history comes with you. A migration rewrites stored findings onto the new codes in one transaction, so timelines stay unbroken and open findings keep their age.

The config file is the one place that may need an edit, because codes appear there by name: in check-thresholds, in disabled-checks, and in the subscriptions and ignore lists on a webhook or Alertmanager endpoint. If you set none of them, there is nothing to change.

Codes and parameters are validated at startup

A code matching no check is reported at startup with an error naming it, and the node does not start. A check-thresholds entry under an unrecognized code used to be passed over without comment, which meant a threshold you thought you had set was quietly doing nothing. Where only the spelling changed the error names the replacement, so SERVER_001 tells you to write server-001. Where the number or the entity changed, the new name cannot be derived from the old one and the error says only unknown check code, so look it up in the mapping table. A parameter name that no longer exists fails the same way and lists the valid names.

Every place a code appears in the config is checked this way, the top-level and per-system check-thresholds and disabled-checks included:

check-thresholds:
  server-003: # was SERVER_003
    cpu_percent: 80

notifications:
  webhook:
    - name: 'Relay'
      url: 'https://example.com/webhook'
      subscriptions:
        - server-001 # was SERVER_001

Codes also appear outside the config file, where nothing validates them for you: the Alertmanager alertname, the finding feed's subject token, and hx.check_findings.code in saved SQL and agent prompts.

Eight checks were retired

Seven of them have no replacement and no new spelling, so a config entry naming one fails startup as an unknown code; remove it. Their stored findings are dropped rather than remapped.

CHANGE_003, JETSTREAM_007, OPT_IDLE_006, OPT_PLACE_002, SERVER_006, SERVER_009, SERVER_017.

Several were folded into another check rather than dropped outright. Their rows still go, because the absorbing check measures a different threshold and inheriting the old findings would attribute them to the wrong measurement.

The eighth, server-049 (Leafnode Auto Compression with High Count, was OPT_SYS_017), is retired after the rename: leafz reports the negotiated compression, not the configured s2_auto mode, so the check could not establish what it claimed. Its stored findings are kept and still display. An entry naming exactly server-049 in check-thresholds, disabled-checks, subscriptions or ignore is accepted with a warning at startup and has no effect; OPT_SYS_017 fails as an unknown code.

OPT_PLACE_004 is a related case: it continues as account-014, but its stored findings do not carry over. The old check recorded a server's key where the new one records an account's, so the next evaluation writes them correctly instead.

Some thresholds take typed values

A threshold that encoded its unit in the parameter name now takes a value that states it. Durations take Go duration strings, sizes take IEC binary strings or a plain byte count, and percentages end in _percent and take 0 to 100. This makes an override readable on its own. insights config check names any parameter whose name needs updating, but not a value written for the old unit, so convert values yourself: min_mono_frac: 0.8 becomes min_mono_percent: 80.

Two defaults moved, both toward fewer findings: account-007 (was OPT_ACCT_001) warns at 90% rather than 85%, and consumer-017 (was OPT_SYS_003) at 90% rather than 80%. If you had raised either one yourself, the override is no longer needed.

Several checks gained optional parameters whose defaults preserve current behavior, so there is nothing to set: crit_percent on server-005 and server-011, max_restarts on server-008, sustain on server-014 and server-048, and sustain with baseline on system-009.

Some severities moved

Nothing to edit, though alert routing keyed on severity will catch a slightly different set.

• Warning to critical: consumer-006, consumer-007, consumer-008, consumer-009, consumer-011.
• Critical to warning: server-014 (critical again for a disabled JetStream, or once a recovering meta layer outlasts sustain), system-007 (critical again when offline peers leave the meta group below quorum).
• Critical to info: server-007, consumer-004, consumer-005, stream-009.
• Warning to info: server-002, server-010, server-048, consumer-018.

server-010 and server-048 can escalate from info the same way.

Nine checks are new

server-053, server-054, server-055, server-056, server-064, server-065, stream-033, stream-034, consumer-020. They are on by default. Run insights checks info <code> for what each one measures, or see the checks reference.

server-065 is the one most likely to have something to say on an existing deployment: it reports a server with no varz reply in an epoch where another server replied and a peer still routes to it. stream-034 and consumer-020 report io.nats.monitor.* metadata values that are invalid for their key.

Full code mapping

insights checks list prints the current set. The mapping below is the one the migration applies, so it is also how your stored history was rewritten.

Data

Eighteen migrations run on first start. They add columns and a table, rewrite stored check codes, unwrap JSON columns, rename the findings table, backfill the new table, add a column to the findings table, retire a check, rewrite enum spellings, drop repeated account identity rows and drop the audit schema. Five of them change what an existing query returns, and all five are worth a look if you have saved SQL or Grafana panels.

hx.check_results is now hx.check_findings

Every row in the table is an entity that tripped a check. Checks record only what fails, so there is no pass row and no severity meaning "ok", and the rest of the product already said finding. The table now matches.

The rename copies no rows, but no view is left behind under the old name, so a query naming hx.check_results needs updating. Find them with:

grep -rn 'check_results' /path/to/your/queries /path/to/grafana/dashboards

The code column is rewritten to the new spelling at the same time, so a query filtering WHERE code = 'SERVER_001' returns no rows rather than an error. Worth updating in the same pass.

Check macros moved from audit to checks and contexts

The macros behind the checks lived in a schema named audit. They now live in two schemas named for what they hold, and a migration drops audit:

A check's macro is its code with hyphens turned into underscores, so the Full code mapping gives the new name. A threshold parameter the check renamed takes its new name, per Some thresholds take typed values. A query that calls an audit. macro fails with a catalog error rather than returning nothing. Find them with:

grep -rn 'audit\.' /path/to/your/queries /path/to/grafana/dashboards

The discovery endpoints list the new schemas. A client that passes {"schema": "audit"} to $INS.sys.<id>.db.macros or runs insights db macros --schema audit gets an empty list. Ask for checks or contexts instead.

JSON columns hold documents rather than encoded strings

Several JSON columns held their document encoded a second time: the JSON string "{\"k\":\"v\"}" rather than the object. json_type() reported VARCHAR and every json_extract path on it returned NULL. Two consumer enums that NATS Server renders as JSON literals kept their quotes the same way, so priority_policy never equalled pinned_client.

The migration unwraps each affected column in place:

These columns now behave the way their documentation always described, so a plain json_extract(metadata, '$.key') works. If you had written a workaround, usually a nested extract or a comparison against a quoted literal like priority_policy = '"pinned_client"', replace it with the direct form.

Enum columns hold the NATS API spelling

Insights stored stream and consumer enums in a display spelling that is not what the NATS API sends, and several checks compared against the API spelling and never matched. It now stores the API spelling, and a migration rewrites the recognized values already stored. Empty, NULL and unrecognized values are left as they are.

A saved query such as WHERE storage_type = 'File' returns no rows after the upgrade rather than an error; switch it to the lowercase value. Newly indexed rows also record the API's defaults where the server omits the field, so persist_mode reads default and priority_policy reads none. hx.consumer_opts.storage_type is still the parent stream's storage; the consumer page shows the consumer's effective storage separately.

Raft state is uppercase

hx.raft_groups.state holds the value NATS Server reports: LEADER, FOLLOWER, CANDIDATE, CLOSED. Several checks compared it against Leader and Follower, which matched nothing. They now compare against what is stored, so the checks reading raft state report correctly.

The stored data is unchanged. A query of your own comparing this column against a capitalized value was returning no rows before and still does; switch it to uppercase.

Repeated account identity rows are removed

hx.account_ident holds one row per account identity. A database indexed before that rule was enforced carries copies, and retention never removed them while the account stayed active. Every join on pk multiplied an account's rows by the number of copies it had, so any account total built that way was inflated.

The migration keeps the first observation of each (pk, name) and deletes the rest. A query of your own that joins hx.account_ident will return smaller numbers afterwards; those are the correct ones. A saved account panel whose values drop on upgrade is showing this, not a loss of data.

New columns

All are additive with defaults, so nothing existing changes behavior. Each one answers a question that previously had no column.

If you aggregate over hx.server_stats, either exclude statsz_fallback rows or treat their NULL columns as missing, so a fallback row does not read as a server with zero leafnodes.

Retention

db.retention.interval is a minimum rather than a fixed period. A sweep runs on the indexer right after the first epoch committed past the interval, and never at the same time as a commit. Nothing to change; sweep timestamps are no longer evenly spaced.

Interfaces

Subjects name the system or node they address

With several systems on one node, every $INS subject carries the id of the system or node it addresses. $INS.ping stays flat, since discovery is the one case where every instance should answer on the same subject.

A useful side effect is that several Insights deployments can share one NATS account: each has its own subject space, so their streams and endpoints cannot be mistaken for one another.

The CLI and web tier discover an id before they call, so they need nothing from you. A client you wrote against the flat subjects needs the id added. insights system list and insights node list show what is reachable. See the API reference.

A named sink stream needs a new name

Scrapes are published under $INS.sys.<system>.scrape.>. Insights 0.1.10 created its sink stream capturing <stream>.>, and a stream that already exists is used as found, never edited. With the default name nothing collides: 1.2.0 creates a fresh scrape_<system> stream and the 0.1.10 scrape stream is left unused. With a name set through sink.stream, 1.2.0 finds the 0.1.10 stream under that name, the scraper refuses its subject, and nothing new is indexed. The log shows it once at startup:

level=ERROR msg="scraper runner error" ... error="stream \"<stream>\" should have subject \"$INS.sys.<system>.scrape.>\""

Check before you restart:

grep -n 'stream:' config.yaml     # under sink:
env | grep INSIGHTS_SINK_STREAM

If either names a stream, remove the setting (1.2.0 then uses scrape_<system>) or change it to a name no stream has yet. The indexer tracks its position per stream name, so a new name starts at the beginning of the new stream. Once epochs are arriving, delete the old stream with nats stream rm <stream>, and the unused scrape stream too on a node that ran with the default.

Sizing reports memory in GiB

The ops.sizing reply renames total_gb, query_gb, write_gb and index_gb to total_gib, query_gib, write_gib and index_gib. The values were already whole GiB. insights ops sizing prints its recommendation as --db.memory-limit=<n>GiB, where it used to print GB and so applied about 7% less than it recommended.

The HTTP gateway names the system in the path

POST /sys/<system>/db/query      # was POST /db/query
GET  /systems                    # the systems this node serves

Every /db/* route moves the same way: columns, explain, macros, query, schemas and tables. The path token is the same id as the subject token. There is no default system and no cached topology, so a request is always answered by the system it named. See HTTP gateway.

MCP tools take a system argument

Every Insights MCP tool except systems requires a system argument, and a call without one is refused before it runs. An agent working interactively resolves it by calling systems first; a saved prompt or script that calls tools directly needs the id added. See MCP server.

Alert payloads

entity_name is an annotation rather than a label, so a name that resolves a few epochs after a finding first fires no longer changes the Alertmanager fingerprint and fires a stable alert again. Routes, inhibitions and silences that match entity_name stop matching; match entity_key or entity_pk, which are still labels. Every alert and the payload's group labels carry system.

Alertmanager endpoints now receive every standing finding as firing once a minute, with endsAt four minutes out, so Alertmanager resolves an alert on its own when Insights stops sending it. Delivery to an endpoint is at-least-once: a failed delivery is retried until it succeeds, so a receiver can occasionally see a transition twice. See Payload labels and annotations.

Prometheus series carry a system label

Every metric family is per-system, since one node can own several systems with their own catalog, indexer, scraper and epoch clock, and without the label their samples would collide into one series.

On a single-system node the label has one value, so a sum() gives the same number as before. topk and label joins behave differently, so add by (system) where a panel is meant to be per-system.

New series: insights_epoch_servers_expected, insights_epoch_servers_observed, insights_epoch_partial_endpoints, and insights_metrics_source_up on an aggregating endpoint, where every series also gains a source label. See Metrics.

Saved search and filter expressions

List pages and the search box share one parser, which changes two things a saved expression or bookmarked filter can depend on. and, or and not are operators in any case, so a bare lowercase value spelled that way needs quoting. check: takes the new codes, so check:SERVER_003 matches nothing; write check:server-003. See Search.

Web URLs include the system

Every page lives under /systems/<id>/…. On a node serving one system, old paths redirect into it and preserve the query string, so existing bookmarks and deep links keep working. On a node serving several, an old path opens the system list. Tooling that constructs Insights URLs is worth updating to the new form.

Operations

The Insights image has no shell

The Linux images on registry.synadia.io (insights, insights-licensed and the -collector tag) are built on distroless static, which carries CA certificates, time zone data and a handful of OS packages, and no shell or package manager, so scanners have far less to report against them. The Insights image still runs as UID 1000 with /var/lib/insights owned by that UID, as in 0.1.10, so a data volume that worked before needs nothing. The collector image is new in this release, built the same way, and has nothing to migrate.

What changes is anything that runs a command inside the container: docker exec <container> sh, a Kubernetes exec probe calling sh -c, or an init container that reuses the Insights image to chown a volume. Point probes at GET /healthz instead, and use a small general-purpose image for init steps. See Health probe.

ops backup builds the file on your machine

Saving is insights ops backup save, still the default subcommand, so an existing invocation keeps working. The copy streams table by table and is assembled locally, so the disk it needs is yours rather than the server's, and a backup no longer competes for space on the instance. It runs at the same Insights version as the server, and a version difference is reported up front rather than producing a file you cannot restore.

Three things changed with it:

• --output - is no longer available. The copy writes into a seekable DuckDB file, so it cannot be piped until complete. A script doing insights ops backup --output - | gzip > b.gz should write the file and compress it afterwards.
• The endpoint is $INS.sys.<system>.db.backup.stream, in place of $INS.db.backup.
• The backup: config section is gone, since backups are no longer uploaded to an object store.

The command confirms the size at a terminal, so add --force for anything running from cron or CI.

insights ops backup save --force --output /backups/insights-$(date +%F).db

insights ops backup view <file> serves the web UI against a backup, opened read-only, which is a convenient way to check one. See insights ops backup.

Run under a restart policy

A DuckDB recovery attempt that cannot return within 30 seconds now exits so a supervisor can restart the process, rather than leaving listeners serving a database that cannot answer. Under a Kubernetes Deployment, a systemd unit with Restart=, or --restart on a container, this recovers on its own. Without one, the process stays down where it would previously have stayed up in a state it could not serve from. See Restart policy.

Naming the target makes client commands faster

With nothing naming a target, a client subcommand waits out the two-second $INS.ping discovery window so every responder has a chance to reply. Set --system (or INSIGHTS_SYSTEM, or system.id in a shared config file) for repeated invocations. A named target skips discovery, which also means silence from it is reported as the error it is.

Discovery needs permission to publish $INS.ping and receive replies; a named call needs only its endpoint permissions. See Selecting a system or node.

Notification endpoint state left JetStream

Endpoint config is no longer mirrored into a JetStream KV bucket, and the delivery log moved to a stream named for the node (webhook-deliveries_<nodeID>). Nothing removes what 0.1.10 created: the webhook-endpoints KV bucket and the webhook-deliveries stream stay behind in the sink's JetStream, unused. Remove them once the node is on 1.2.0:

nats kv del webhook-endpoints
nats stream rm webhook-deliveries

If your NATS permissions granted either one explicitly, that grant is unused too.

Agent skill downloads moved

The info dialog no longer builds and serves the per-agent skill bundles. Install them from synadia-io/insights, which is where the plugin marketplace and Gemini extension already lived. See AI Agents.

After you upgrade

Confirm your checks resolve. Startup would have reported an unknown code, so this is mostly about a check you meant to disable and spelled correctly for the wrong entity.

insights config check
insights checks list

Give the new critical severities a day. Five consumer checks moved from warning to critical, so a pager routed on critical may see traffic it did not before. server-065 is also new and reports a server with no varz reply in an epoch where a peer still routes to it.

Check that epochs are complete. The new completeness columns show whether a scrape reached the whole fleet, which is useful to establish as a baseline.

SELECT epoch, servers_expected, servers_observed, partial_endpoints
FROM hx.epochs
WHERE servers_observed < servers_expected
ORDER BY epoch DESC LIMIT 20;

Going back

The database moves forward on first start: the findings table is renamed and rewritten, JSON columns are unwrapped, enum values are rewritten, repeated account identity rows are deleted, the audit schema is dropped, and columns and a table are added. A 0.1.10 binary will not find hx.check_results in a 1.2.0 catalog.

Going back means restoring the backup from Before you upgrade into a fresh data directory, pointing the old binary at it, and letting it scrape forward again. Epochs indexed since the upgrade are not carried back. Keep the pre-upgrade config file alongside the backup, since the new check codes are not valid on 0.1.10 either.