Skip to main content

Insights v1.2.0

One change shapes this release: an Insights node is no longer tied to a single NATS system. A single process can own several monitored systems, each with its own database, indexer, scraper, checks and API subtree, and every client (the CLI, the web UI, the MCP server, the HTTP gateway) says which one it means. Around that: a DuckDB-free collector build for edge sites, a rewritten ops backup that builds the file on your machine, a backup viewer, a finding feed on NATS, grouping and labels across a fleet, one Prometheus endpoint for a whole deployment, and a long list of query, check and UI work.

Several things change behavior on upgrade. Read Considerations before you deploy: every check code is spelled differently, and a config naming an old code or parameter refuses to start, the flat $INS.* subjects are gone, a data directory now records the ids it was built under, and ops backup, the HTTP gateway's URLs, stored enum spellings and the container images' base all changed.

Features​

  • One node, several NATS systems: One node can now monitor more than one NATS system at a time. Each system listed under systems: gets its own DuckDB database (<data-dir>/systems/<id>.db), its own JetStream stream, its own indexer, its own check thresholds and its own API subtree at $INS.sys.<id>.*; nothing is shared or steered between them. A system is either scraped by this node (it has a scrape: block) or fed by a collector elsewhere (a NATS mirror fills its stream and this node only indexes it). A system whose catalog fails to open is degraded rather than fatal: it is logged, kept in the system set and skipped, and its scrape side keeps filling its stream so the epochs index once the catalog is back. The list is YAML-only, and a single-system deployment keeps the flat system:/sys:/scraper:/sink: config it has today, resolving to byte-identical values, so nothing needs to change to upgrade.

    # config.yaml
    node:
      id: central-1               # required once a node serves more than one system
    systems:
      # Scraped by this node.
      - id: core-prod
        scrape:
          nats:
            server: nats://core.internal:4222
            creds: /etc/insights/core-prod.creds
          interval: 20s
        retention: 24h              # stream buffer depth; the database keeps history
        db:
          memory-limit: "8GiB"      # per-system DuckDB pool
        metadata:
          env: prod
          region: us-east-1
        check-thresholds:
          server-003:
            cpu_percent: 80
      # Fed by an edge collector: the absence of a scrape block is the whole
      # declaration. This node creates the mirror and asks the collector for the
      # names, prefix, domain and cadence it actually uses.
      - id: edge-west
        retention: 6h

    The web UI follows the same model. Every page is path-scoped under /systems/<id>, each backed by its own querier, time-picker configuration and check parameters, so the URL rather than a stateful selection picks the backend. A node serving one system lands and reads exactly as it did before. A node serving several gets a Systems list, a Nodes inventory discovered over $INS.ping, and, where geo enrichment is configured, a Map page that merges geo data across every system in Go, since separate databases rule out a cross-system join. Systems appear in search results, so the search box is also how you move between them.

  • A collector build for edge sites (insights-collector): A separate binary that scrapes a NATS system and publishes to a stream, with no DuckDB, no web tier, no embedded NATS server and no Prometheus client linked in. It is a fraction of the size of the full binary, ships as an insights-collector archive and a -collector image tag, and is meant for places where the scrape has to happen but the indexing does not: a customer site, a leaf location, a different network zone. A full Insights node at a central site mirrors the stream, indexes it, and serves the query API and UI for it. Both halves bind disjoint parts of the same $INS.sys.<id>.* subtree, so clients address subjects and never need to know which process holds what. Give the two distinct node.id values: each defaults to the system id when it serves only that system, and a node whose id is already in use refuses to start.

    Insights now creates every stream it fills, the central node's mirror included, and asks the collector over $INS.sys.<id>.ops.sink.info for the origin stream's name, subject namespace, JetStream domain and scrape cadence. A central node that starts before its collector waits and retries per system instead of guessing from the naming convention. That removes the hand-typed mirror block, which JetStream refuses to edit once a stream exists, so a mirror aimed at the wrong origin, prefix or domain used to stay empty forever with nothing to read. An existing stream is still adopted as-is, with a per-field warning where it disagrees with what the collector reports. Only mirror-api-prefix survives, for an origin reached by a subject mapping or account import rather than a domain.

    The Federated Deployment guide covers the part operators get wrong: the leafnode bridge, the two JetStream domains, and the cross-domain mirror and permissions that carry scrape data from the site to the central node.

  • Naming a system or node from the CLI: Every client subcommand takes --system <id> (env INSIGHTS_SYSTEM, or system.id in a shared config file), and the process-diagnostic commands take --node <id> for the Insights process itself. With nothing named, the CLI runs one $INS.ping discovery round: one answering system is used automatically, and several produce an error listing them. insights system list and insights node list show what is reachable.

    insights system list
    insights db query --system core-prod "SELECT count(*) FROM hx.conns"
    insights ops pprof --node central-1 --type goroutine --debug 2

    The two long-running gateways serve every system from one process. insights http takes the system in the URL path, token for token with the subject it forwards to, so an ingress can route, rate-limit and scope a token per system (GET /systems lists them). insights mcp takes it as a required tool argument, and its systems tool carries the rule for resolving it: one result is the answer, several means ask. Neither holds routing state, so no request can be answered by a system other than the one it named.

    curl -X POST http://127.0.0.1:8080/sys/core-prod/db/query \
      -H 'content-type: application/json' \
      -d '{"sql":"SELECT name, connections FROM hx.servers LIMIT 5"}'

  • Backups you build, and read, on your own machine: insights ops backup save streams the database one table at a time over $INS.sys.<id>.db.backup.stream and inserts each Arrow stream into a DuckDB file it creates locally. The server reads and writes nothing, so it no longer needs room for a second copy of the whole database on its own disk before you have anything. The result is a complete database, schema and views and macros included, that opens directly as a --data-dir.

    The latest epoch is resolved once and every table filtered to it, so tables copied minutes apart stay consistent with each other. Version skew is refused before any bytes move, on column types as well as names, because an INSERT would cast a changed column rather than reject it. A retention sweep during a long copy warns and names the epoch below which the tables disagree rather than discarding the backup, since the rows in question are ones the server has just deleted anyway.

    insights ops backup view then opens that file read-only and serves the existing UI over it, so inspecting a backup no longer means hand-written SQL or moving the file into a data directory and starting a server that would migrate your only copy and prune it. Nothing that writes is started. A backup records no scrape interval, so the viewer reads the cadence off the spacing of the file's own epochs, which is what lets its sparklines draw.

    # Last seven days, prompting for confirmation:
    insights ops backup save --start 7d --output weekly-backup.db

    # Non-interactive full backup:
    insights ops backup save --force --output "insights-$(date +%Y%m%d).db"

    # Read it back in the UI on http://127.0.0.1:8080:
    insights ops backup view weekly-backup.db

  • Turning a check off: disabled-checks lists the check codes an operator has decided not to act on, at the top level for every system or inside a systems: entry for one of them. An entry's list is unioned with the top-level list, so it can add codes for its own system but never re-enable one. It is for a check that is structurally true of a topology rather than a symptom of anything, such as account-012 (High Gateway Traffic Ratio) on one account spanning a supercluster, where no threshold value would make it meaningful.

    A disabled check keeps running and keeps its history, and it still appears everywhere it otherwise would, marked disabled, with its real findings. What changes is that it is neither graded nor delivered: the category and overall health percentages and the health-timeline stat cards leave it out on both sides, and nothing it finds becomes an advisory, a webhook or an alert, with no spurious resolve when it is turned off and no re-fire when it is turned back on. insights checks list gains a Disabled column when any check is off, and the findings endpoint marks the rows too, so the CLI table and the MCP checks_findings tool no longer present them as open issues.

    disabled-checks:
      - account-012
    systems:
      - id: edge-west
        disabled-checks:
          - stream-025          # added to the top-level list for this system

  • Checks that know which servers can answer them: An absent JSON key decodes to its zero value, so a counter a server never sent was stored as though it had been reported as zero. server-012 and server-013 read counters that arrived in nats-server 2.12.0 and could never fire below it, indistinguishable in the UI from a pass; consumer-012 reads a field absent before 2.11.0, where the empty string satisfied its predicate and it fired on servers with no policy to set. A version-gated check is now measured against the entities it can actually evaluate, and where that is none, the row dims and names the nats-server release that would enable it. The version lives in the check registry and reaches the SQL as an argument, so the two cannot drift apart.

  • A finding feed on NATS: Every state change a check makes is published on $INS.sys.<id>.feed.finding.<status>.<severity>.<code>, with the status, severity and code in the subject so a subscriber filters server-side rather than receiving everything and discarding most of it. The payload names the entity, carries starts_at (when the finding first appeared, not when it last changed) and, on a resolve, ends_at. There is a third transition beyond fired and resolved: a check whose severity moves while it stays up reports changed with the severity it left behind, so a subscriber no longer keeps whichever severity it first saw. Severity is deliberately not part of the finding's identity, since putting it there would emit a resolve and a fire on every escalation and tell a receiver the problem went away.

    nats sub '$INS.sys.core-prod.feed.finding.firing.critical.>'
    nats sub '$INS.sys.core-prod.feed.finding.*.*.stream-002'

  • Alertmanager holds what is still firing: An Alertmanager endpoint receives every standing finding as firing once a minute, each with an endsAt four minutes out, alongside transitions as they happen. Alertmanager keeps an alert while it keeps arriving and resolves it on its own when Insights stops sending it. The re-send runs on a wall-clock ticker rather than on scrapes, so a stalled scrape does not let held alerts expire. A severity change resolves the alert at the former severity in the same delivery, so an escalation does not leave a duplicate behind. Webhook endpoints still receive transitions only.

  • Notification routing: Endpoint subscriptions and a new ignore list take check-code patterns: an exact code, a trailing-* prefix, or * for every code, with ignore applied after the subscription match so it carves exceptions out of a wildcard. Patterns are validated at startup against the check registry, so one matching no known code fails startup rather than silently delivering nothing, and the endpoint detail card expands each pattern into the concrete checks it resolves to. A systems: list scopes an endpoint to a subset of the systems a node owns (empty means all of them), and every payload carries a system label per alert and on the group labels, so a receiver can route and group by it.

    notifications:
      alertmanager:
        - name: "Alertmanager"
          url: "http://localhost:9093/api/v2/alerts"
          subscriptions:
            - "*"
          ignore:
            - consumer-*        # dropped after the wildcard match above
          systems:
            - core-prod         # empty would deliver for every system on this node

  • Labels, and grouping a fleet by them: node.metadata and system.metadata are operator-assigned labels reported on $INS.ping. An id says which system, never which kind, so a node discovering thirty of them had no way to say "the prod ones" or "everything in us-east-1". Labels are inert: nothing addresses a subject with them and no behavior keys off them. Metadata for a system is per (node, system), because collector mode splits a system across processes, and the summary unions them and reports the keys they disagree on rather than picking a winner silently.

    Every list page now runs the search DSL's own parser rather than imitating it, which adds | by <key> and | sort <attr> [desc] as stages of the filter expression. Grouping puts the aggregate on screen: a group header takes the worst value rather than the mean, since one unreachable system in a group of twenty is the fact the header exists to surface. Grouping rearranges and never filters, and a row with no value for the key bands under "no value", last. The DSL also learns metadata.<key>:<value>, resolved against the entity's JSON column, and a bare attr: asking whether a value exists at all; in global search, qualify it with an entity, as in server@metadata.env:prod.

    # on the Systems list
    metadata.env:prod | by region | sort health desc

  • Search says why a row matched: Matching servers on subscriptions:>50000 used to list names with no numbers, so the only way to learn why a row was there was to open it. Every attribute an expression names now travels with the query and renders under the result in written order, including under OR, where both values are what tell the two halves apart. The per-table filter box gains a preview of the rows its filter selects, with a count, so a subset can be read before Enter changes the table; it reads the entity's own view rather than the table's, which is what lets a filter on an attribute the table renders no column for still show its values. Both surfaces read the epoch the page is showing rather than the latest, so a pinned page finds what existed then.

  • Readable tables from every CLI command: db query gains --format text, which renders the result as a table sized to your terminal, with numeric columns right-aligned off the Arrow schema so a port or an account ID still reads as text. Every text table the CLI prints now goes through one renderer, including the collector build, so db schemas, checks findings and the deployment listings all lay out the same way: prose columns wrap at word boundaries, value columns keep their width, and output that is not a terminal is laid out to 100 columns with no color. csv is still the default for db query, so existing scripts are unaffected.

    insights db query --format text "SELECT name, cluster, connections, cpu FROM hx.servers WHERE epoch = (SELECT max(epoch) FROM hx.server_stats) ORDER BY name"

  • A data directory that records what it owns: Node and system ids name persistent state (the $INS.node.<id>.* subjects, the scrape stream, the catalog file), and nothing recorded which ids a data directory was built under. Changing one in config silently stranded everything named after the old one: a fresh catalog, an empty history, and months of data under a name no instance opens, discovered when a dashboard came up blank. identity.json now records the ids the last run owned and the resolved names those ids produced, and a start that no longer claims something warns and names the unclaimed state on every start rather than once.

    insights config check moves the decision ahead of the restart and validates the configuration (including reporting each system's resolved disabled-check set); insights config migrate performs it, moving the catalog and its write-ahead log onto the new names. Both run against a stopped instance, which moving a catalog requires.

    insights config check --config /etc/insights/config.yaml
    insights config migrate --config /etc/insights/config.yaml               # adopt the new naming scheme
    insights config migrate --rename system:west=north --config /etc/insights/config.yaml

  • One Prometheus endpoint for a whole deployment: Prometheus needs a route to every node and a target list that changes as nodes come and go, which a fleet behind a leafnode, NAT or a tailnet does not give it. With prometheus.aggregate set on one node, each scrape of that node's /metrics discovers the deployment's nodes over $INS.ping, asks every one for its metrics over NATS, and serves the merged set with a source label per node. insights_metrics_source_up reports which nodes answered. insights prometheus serves the same endpoint as a standalone process. Enable it on exactly one endpoint, since two aggregators report every series twice.

    prometheus:
      aggregate: true
      timeout: 2s        # shorter than the scrape timeout of whatever reads it

Improvements​

  • Query results stream instead of being materialized: The DuckDB driver runs a statement to completion and wraps a finished result, so both streaming endpoints spent memory proportional to the whole result before producing a single batch, and a seven-day backup window took a production instance down. Neither db.memory-limit nor temp_directory bounds that, because a materialized result is neither accounted against the buffer manager nor spillable to disk. The endpoints now execute as a DuckDB streaming result, so peak memory is a function of one batch, and the ack window, batch watchdog and row cap become bounds on work rather than pacing over an already-realized result.

  • Single-epoch stream and consumer queries no longer time out: The entity views resolved each row's configuration with a correlated lookup that expanded the retained stats against the whole options history before the epoch filter could apply, so a query for one epoch of hx.streams or hx.consumers could run past the query endpoint's 30-second limit. Configuration is now resolved through per-entity validity intervals, with the window running over the options history alone. No public column changes, and a full-history comparison against a production copy found no row differences.

  • Retention no longer blocks queries for minutes: The orphan-identity prune expressed its check as a correlated NOT EXISTS inside the DELETE, which planned as an anti-join over the largest unindexed table in the schema and held DuckDB's single write lock for the duration while every web query queued behind it. It now finds the doomed rows in a read and deletes them in primary-key batches, IP queue identities included, and each slow prune statement is logged so a recurrence names its own table. The sweep also runs on the indexer right after an epoch commits, at most once per db.retention.interval, instead of on a ticker of its own that could overlap a commit, stall it, and fail the post-prune checkpoint with "there are other write transactions active".

  • Throughput rates are computed per entity: Every metrics view took its deltas per entity and then divided the sum by the epoch interval. An entity that misses one scrape has its delta span two intervals, charted as one, so a fleet whose throughput never moved drew a dip where the entity was absent and a matching spike where it returned. Two variants were worse: raft metrics summed the committed index across groups before differencing, so a group missing from one scrape subtracted its whole log index and produced commit rates orders of magnitude off; and account metrics divided one epoch's delta by a whole bucket span, understating throughput at every window of three hours and wider. Rates are now computed per entity and then summed, and bucketing samples a rate rather than dividing a delta. Each entity's delta is divided by its own elapsed interval, in the aggregate views and in the cluster and account detail sparklines, so a server returning from a missed scrape spreads its delta over the gap instead of charting a spike. A missing point on a throughput card reads as a gap rather than pulling its range bar and last value toward zero.

  • An epoch that reached 17 of 18 servers no longer looks whole: A server that misses the scraper's discovery window is dropped with no error, and the epoch's system aggregates silently lose its routes, gateways and throughput. Each epoch's server set is now diffed against the last and what disappeared is named, and the scraper's own expected-versus-observed report is recorded on hx.epochs and exported as gauges. An epoch with no report stays NULL or NaN, which reads as unknown rather than complete.

    A server can also answer the discovery ping and STATSZ from queues of their own while its other monitoring endpoints stall, so it counted as observed while producing no rows, vanished from the UI and charted as a dip. The new server-065 (Server Not Reporting) reports a server with no varz reply in an epoch where another server did reply and a peer still routes to it, and the servers table and detail page keep it listed. Where it still answered STATSZ, those counters are stored as a hx.server_stats row marked statsz_fallback, with leafs, remotes and gateways NULL because STATSZ has no equivalent.

  • Every check's evidence panel shows what that check measured: A check is described in four places (the code that identifies it, the macro that computes it, the prose that explains it, and the panel showing a finding's supporting data) and those had drifted apart. Panels bound to a code's older meaning queried another check's table with this check's key, which resolved against an unrelated row and rendered it as evidence rather than failing; twenty-five rendered a generic table carrying no evidence at all; six codes were registered twice, so filename ordering decided which body survived. Every registered check now has a panel showing the quantity its own finding reports. Nineteen documented thresholds that sat on the wrong side of their macro's comparison are corrected, four upstream io.nats.monitor.* keys now use the same boundaries jsm.go compares them with, and the mechanizable half of that agreement is a test: the documented operator against the macro, the upstream boundaries per key, and each aggregation semantic against the number its own macro reports.

    Panels also read the columns their check declares it fired on rather than recomputing the number from source tables, which is where server-005 drew a server at 95% of max_memory as healthy and server-038 and server-039 printed byte counts with a MiB suffix. Threshold markers use the value configured for the system being viewed rather than the registry default, and an optimization check's detail is measured over the check's configured lookback, so changing the page's time range no longer changes the number a finding was made on.

    An event in the health timeline and the panel header beneath it stated the same finding twice, and are now one row: the severity, the code as a link to the other entities firing it, where the finding is in its life, and the check's own explanation behind an info icon. Each threshold there says where it came from and whether an operator moved it. Expanding an event evaluates it at its own epoch rather than wherever the page is pinned, which is what let a resolved event describe its condition as still true. Panels also draw from a fixed set of shapes, so a gauge names the ceiling it divided by and a bar means the same width in every panel rather than being scaled to the longest label beside it.

  • What a check says about itself now matches what it does: A check's name, description, remediation and trigger were written out in six places, and the copies had drifted: rows filed under the wrong group, triggers naming the wrong comparison, and conditions the SQL applies that no text mentioned. Each check is now described once, and the reference table, the web UI, insights checks info and the text on a webhook payload all derive from that one description. The wording is plainer, and it names the threshold that decides a finding rather than the default that threshold ships with, since an operator can change it and the panel shows whichever value is in force.

  • healthz conditions split by what an operator does about them: The server reports a dozen distinct conditions under one error type, and server-015 carried all of them under a single remediation, so a rejected configuration and a full disk read the same. They now split into a configuration the cluster refused, an asset assigned here and missing, a store that cannot write, and a Raft group the server cannot serve from, and the asset each error names is indexed alongside the message rather than parsed back out of prose. A new check reports a Raft group whose write-ahead log failed, which healthz can only describe and raftz can name.

  • connection-002 measures against the server's own limit: It compared pending bytes against a fixed 1 MiB, which is 1.6% of the 64 MiB default and half of a server tuned to 2 MiB, so it reported a different thing on every deployment and could sit above the point at which the server actually stops the connection. It now measures the fraction of that server's own max_pending.

  • Checks judge what NATS actually reports: Each check's predicate was compared against live nats-server behavior, and several changed what they report:

    • server-008 no longer reports a restart that a configuration change explains, comparing the config_digest now indexed from varz. Servers older than 2.11, which do not report a digest, keep the version-only rule.
    • server-014 and server-048 escalate only once a condition has held for a sustain window, so a booting server's recovering meta layer is a warning rather than an immediate critical, while a disabled JetStream stays critical.
    • account-013 could not see a tiered account JWT's JetStream limits, which leave the flat limit columns empty. Every limit set a JWT declares is now indexed in hx.account_js_tier_opts, backfilled from stored claims, and the check reads those.
    • server-061 counts the dedicated routes each pinned account adds, taken from routez, so a cluster pinning several accounts no longer hides a lost peer under its expected route count.
    • consumer-004, consumer-005 and stream-009 report at info. The sequences and message counts they compare are sampled at different moments, so a mismatch is evidence to look at rather than proof of loss.
    • system-007 stays critical when offline peers leave the meta group below quorum, and reports a warning when no meta leader has been seen across two scrapes, which missing monitoring replies can also cause.
    • stream-034 and consumer-020 are new and report io.nats.monitor.* metadata that is invalid and therefore cannot be evaluated.
    • account-005 and service-002 keep a confirmed finding through an epoch whose subscription data is incomplete, rather than resolving it and firing it again, and the finding shows when it was last confirmed.
    • server-049 (Leafnode Auto Compression with High Count) is retired: leafz reports negotiated compression, not the configured s2_auto mode, so the check could not establish what it claimed.

  • Consumer storage is shown as the consumer uses it: The consumer page showed the parent stream's storage, but a file-backed stream can host a consumer with memory storage. It now shows the consumer's effective storage, the stream's storage and the memory override separately.

  • One system's fault no longer stops the node: A node with several systems holds N independent DuckDB files, and any one failing to open (a corrupt file, a bad path, a full disk) failed startup for all of them, closing the catalogs that had built fine. The node an operator most needs running is the one where something is already wrong. That system is now degraded and skipped while the rest serve; no catalog opening at all still exits non-zero. The degraded catalog is retried every minute, so once the file, disk or mount is repaired its queries and indexing resume without restarting the node. Its finding feed and notifications resume only after a restart.

  • DuckDB recovery is confirmed, retried and bounded: The reconnect guard trusted PingContext, but the DuckDB driver implements no Pinger, so a ping returned nil even after the instance had been invalidated and every query kept failing until the last pre-fault idle connection aged out of the pool. It now confirms with a SELECT 1 on the raw pool, and recognizes FATAL as invalidating alongside INTERNAL. Web reads, which run on the streaming query endpoint outside the recovery wrapper, now report through it and re-run the query once. A recovery attempt that cannot return within 30 seconds exits the process so a supervisor can restart it, rather than leaving listeners serving an unusable database.

  • A scrape trigger answers as soon as the scrape starts: A scrape routinely outlasts the request asking for it, so waiting for it meant callers timed out on scrapes that had succeeded. The trigger now returns once the scrape is accepted; completion is observable without waiting, because the epoch lands on the system's feed.epoch broadcast once it is committed. The runner's loopback control service goes with it, which also takes four queue subscriptions per system off the sink account that a second scraper on a shared account could otherwise have answered.

  • Finding transitions are tracked whether or not you have an endpoint: The per-epoch diff of check results lived inside webhook delivery, so state changes were computed only when an endpoint was configured, and only for the first system on a node. A tracker per system now owns that state; the finding feed publishes the transitions and webhook delivery consumes them. Names are resolved for transitions only rather than for every finding in an epoch, which is what keeps the seven-way identity join off the hot path, and cached, since a name is immutable per key and a resolve can then still name an entity that has since been deleted.

  • Notification delivery no longer depends on a KV bucket: Endpoint config was mirrored into a JetStream KV bucket at boot and read back live, so an unavailable bucket (its single replica on a downed node, say) was indistinguishable from "no endpoints configured" and silently stopped delivery every epoch. Config now stays in memory. The store keeps only the delivery log, and a node with no JetStream loses delivery logging rather than delivery. The per-endpoint timeout and max-retries knobs are gone; each delivery makes up to 3 attempts with a 10s request timeout (one attempt when the receiver answers with a 4xx other than 429), and a failed delivery is retried for that endpoint at up to one-minute intervals until it succeeds. Old config files carrying the keys still parse.

    Delivery also left the ingest path. It ran inline in the epoch commit, one endpoint after another, so a receiver that accepted the connection and then hung could stall indexing long enough to lose epochs. Each endpoint now has a delivery worker of its own: a failed delivery is retried for that endpoint alone, on a backoff capped at one minute, until it succeeds, and transitions waiting for a slow receiver are coalesced to each finding's latest state, so the backlog is bounded by the number of findings. Folding new transitions into that backlog costs the same however large it has grown, so a receiver that keeps rejecting cannot slow indexing. Delivery is at-least-once.

  • Every metric carries a system label: All the metric families on /metrics are per-system, so each series names the monitored system it describes. Aggregate with sum by (system) (...); a node monitoring two systems would otherwise read as one blended series. The bundled Grafana dashboard is updated to match.

  • DuckDB 1.5.5: Takes a silent data-loss fix upstream, where ALTER TABLE ... ADD COLUMN IF NOT EXISTS <col> DEFAULT <non-constant> was not a no-op when the column already existed and could overwrite every row. Two Insights migrations use a non-constant default, so this was latent rather than live, but the idiom reads as safe. Also included: checkpoint failures stop being swallowed, a hash-aggregate segfault, a memory-manager deadlock, four out-of-bounds reads, and two window-frame correctness fixes.

  • Tested against the nats-server versions we claim to support: Recorded monitoring replies from 2.10.27, 2.11.17, 2.12.14 and 2.14.4 are now committed and replayed through the real decode path on every build, so a version whose fields Insights silently drops fails a test instead of quietly producing empty columns. A weekly job compares the pinned server's monitoring types against the newest release of each minor and reports when the pin has fallen behind. Two immediate results: the pin was moved to a patch that declares 11 JSON fields the previous one did not, all of which were being discarded, and the documented floor was corrected to 2.10.27, the first patch where every monitoring endpoint Insights uses is present.

  • Column comments describe what the data actually does: The query documentation tells agents and operators to trust the catalog, so a comment that describes the wrong thing propagates into queries. Several were wrong in ways that changed an answer: conn_stats.msgs_sent is the server's delivery to the client rather than the client's send; account_stats.total_conns is a gauge always equal to conns plus leafnodes, so nothing can derive connection churn from it; and server_stats.in_msgs and its siblings include route, gateway and leaf traffic, so summing them across servers is not client throughput (route traffic alone was 78% of the fleet delta on one production cluster). Also corrected: num_redelivered decreases, routes count connections while remotes count peers, cache_hit_rate is cumulative and can exceed 1.0, cpu is a one-second sample, and in the raft columns group_id identifies the reporting server while leader is a peer hash rather than a name. The replica tables needed more than wording: they hold three kinds of row per entity per epoch, and leader-reported peer rows carry zeros for every message and sequence column while self-reported rows hardcode is_current true and is_offline false, so a replica-health query has to restrict to peer_server_pk != 0.

  • Route pool size is recorded: NATS opens route_pool_size connections per peer, so a check counting routes against an expected total was wrong in a pooled cluster. The column and the code that fills it land together.

  • Requests work from accounts without _INBOX.>: A least-privilege account that is not granted _INBOX.>, a common setup on Synadia Cloud, could not complete a single request, because every reply landed on a subject it may not subscribe to. Every NATS connection block now takes an inbox-prefix, for example --sys.inbox-prefix / INSIGHTS_SYS_INBOX_PREFIX on the server side and --inbox-prefix / INSIGHTS_NATS_INBOX_PREFIX for a client command. Leave it empty unless the account requires it; when set, the account must allow subscribing to <prefix>.>.

  • Subsystem toggles are one spelling: --web, --indexer, --scraper, --simulator, --prometheus, --updater and --telemetry are the CLI form, and each now has a matching environment variable (INSIGHTS_UPDATER, INSIGHTS_WEB, and so on). The --<subsystem>.enabled forms still work but are hidden from --help. YAML is unchanged: the section is a map, so use its enabled: key.

  • Service search reads only the epoch it asks about: A free-text search over services decoded the subject of every retained subscription before applying the epoch filter. The epoch's own subscription rows now drive the join, so older subscriptions are skipped rather than decoded.

  • An idle federated link carries less: A remote web tier subscribed to the once-a-second disconnect feed for the life of the process, though only the real-time view renders it, and the web framework polled component metrics every second across every leaf. Both ends now carry the feed only when the real-time view is enabled, and the poller is off, so its two settings are gone (see Considerations). The scraper also sends raftz requests only for the accounts that can answer them.

  • An indexer whose stream consumer stops says so: Lost heartbeats, a deleted consumer and other errors the JetStream consumer reports only through its error handler are logged, and a subscription that ends for good logs "consumer stopped delivering" instead of looking like a stream with nothing new.

  • The toolbar rests on LIVE: The next-scrape countdown moved into a tooltip, a ring pulses on each epoch so a dead update stream no longer looks like a healthy one, pausing updates the indicator at once, and a pinned window reads "Pinned" rather than "Paused", since pinning does not pause the scraper.

  • Agent skills come from one place: The Insights query skill for AI agents is no longer rebuilt inside the binary and downloaded from the info dialog. The dialog links to the public synadia-io/insights repository, which carries the Claude plugin marketplace, the Gemini extension and the agent instruction files. A release opens a pull request there whenever its skill and extension files differ from what is published, rather than maintaining a second copy inside the binary.

Security​

  • The query API can no longer reach outside the database: The read-only query API constrained the DuckDB statement type but left enable_external_access at its default, so a plain SELECT could read local files (read_text, read_blob), enumerate the filesystem (glob), and issue outbound HTTP (read_csv over http, a server-side request forgery primitive), all while passing the statement-type guard. Every connection now disables external access, extension autoinstall and community extensions. This is not runtime-configurable: no config field, environment variable or flag turns it back on. Because the DuckDB setting is a one-way latch, the optional PostgreSQL-attach build opens permissive and latches every catalog off immediately after attaching, rather than only the attach target.

  • Compressed monitoring replies are bounded: The raftz parse path expanded an S2-compressed response with no decompressed-size limit, so a spoofed server could reply with a small payload that expands to gigabytes of Go maps and exhaust the heap before any downstream size check; the NATS max_payload bounds only the compressed bytes. The decompressed stream is now capped at 64 MiB, and hitting the ceiling is reported as its own error rather than as ordinary malformed JSON. The bundled scraper takes the matching fix for its two S2 paths.

  • Images and builds carry fewer vulnerabilities: The Insights image is built on distroless static, which holds CA certificates, time zone data and a handful of OS packages, and no shell or package manager, instead of a Debian base whose packages every scanner reported. The new collector image uses the same base. Releases are built with a patched Go toolchain rather than the first 1.26 release, and a release whose published images carry a fixable critical or high CVE fails. The latest tag and the suffixed latest-collector, latest-windows and latest-dynamic tags move only on a final release, where latest used to move on release candidates too.

Fixes​

  • Checks that could never fire, on data that was there: Three independent causes. The indexer handed pre-encoded strings to DuckDB's appender, which marshals whatever it is given, so every JSON column landed as a JSON string and every json_extract on stream and consumer metadata, backoff and subject transforms returned NULL, while priority_policy and replay_policy kept the quotes the server's own formatting puts on them. None of the io.nats.monitor.* checks could fire on scraped data. Separately, four checks filtered a write-on-change options table by exact epoch and so reported nothing at all across their entire retained history. And stream storage, retention and discard policies were stored in Go's display spelling (File, WorkQueue, DiscardNew) while stream-022, stream-023 and stream-031 compared against the NATS API spelling (memory, file, workqueue), so those three never matched. All three are fixed, and migrations unwrap the JSON rows and rewrite the enum values already stored.

  • Raft findings on states no server reports: Two Raft checks compared against state names in the wrong case, so they matched nothing on real wire data.

  • Raft groups scattered their history when one reply went missing: raftz is requested per account and answered per peer, so any single reply can go missing on its own. Pruning treated absence as deletion whatever the reason, so one lost epoch evicted every group and re-minted thousands of keys on the next, scattering each group's history across new rows. A group is now evictable only on the word of its own peer, for its own account.

  • Accounts that could never be named: Seven endpoints resolve an account by name and mint a key when it is new, but only the accountz handler wrote the account identity row, and only for a name its cache had missed, which the resolve had already filled. An account first seen anywhere else accumulated stats under a key no view or check macro can join to a name, and the cache, rebuilt from that table on restart, would later hand the key to someone else.

  • Pinned, sorted and paged views reverted on the next epoch: Every control that changed what a page showed fetched the page from its own element, and the page's own bootstrap stream stayed live and re-rendered its load-time state on every epoch. A pin snapped back to live, a sorted table flipped back, page 2 became page 1, and a relation tab's sort was overwritten by the page's default. Controls now set their signals and dispatch to the element that owns the page's one request, so each request replaces the stream before it.

  • An open check detail closed itself every epoch: Each epoch re-patched the block holding the health timeline, and its template still rendered the empty placeholder the detail had been fetched into, so the morph emptied whatever had just been opened. Groups and details are also keyed by epoch now, and the flash guard no longer writes display: none back onto shown elements, which used to collapse the page for a frame and clamp the scroll to the top.

  • Two browser tabs on one page cancelled each other: The connection tracker allowed one stream per session and route and cancelled the previous one, but every session shares a store name and carries no per-client id, so the key was the route alone: six concurrent requests for one page produced five 500s, and a client whose stream was cancelled mid-render stayed on skeletons.

  • Paginated tables reshuffled between refreshes: Every paginated table ordered by the chosen column alone, so tied rows landed on a page in whatever order the plan produced. The consumers table, with thousands of bulk consumers tied on health, changed on every refresh and rows moved between pages. Each paginated ORDER BY now ends with the row's key.

  • A time window with no data read as a healthy system: A window can outlive its data, whether retention pruned it or a session restored a pin from weeks ago. Every count then rendered zero and every check passed for want of an entity to fail on, and the overall grade reported 100%, with nothing on the page to tell that apart from a healthy idle system. The paused-offset badge had the same shape of problem, printing whatever unit divided the offset evenly, so a five-minute pin read "350s" and one months back read "109650m".

  • The toolbar said paused when nobody had paused it: A render that reached the time controls before the scraper had answered wrote the offline anchor into the pinned client signal, and that signal outlives the outage, so seconds later the scraper connected and the toolbar still read paused against an epoch nobody chose, until the page was reloaded.

  • A standing finding's age reset on every restart: The start time was stamped with whichever epoch the tracker happened to baseline on. It is now derived from the check history, as the first epoch of the current unbroken run, and the wall-clock re-send re-asserts it to every Alertmanager endpoint each minute so a receiver that has expired the alert relearns the real start.

  • A stored cross-site scripting hole in check-finding links: Detail links formatted resolved URLs carrying entity names the monitored servers report themselves into both an HTML attribute and a JavaScript string argument, while escaping only the visible label, and the anchor was emitted raw so the template's contextual escaping never ran. A crafted server, cluster or gateway name could break out of the attribute and inject markup or script into an operator's session. Both output contexts are now escaped at the single point every call site goes through.

  • Some detail pages rendered as if the entity were gone: The account key on leaf nodes, and the account and user keys on connections, are nullable, the latter for any connection with no authorized user. Scanned into non-pointer integers, each failed its page's core query, which the handler renders as a missing entity while still returning 200. Found by executing roughly 1,400 statements of hand-written SQL across the detail pages for the first time, which is now part of the test suite.

  • Filter attributes that could not be used: The attribute dropdown offered attributes the underlying queries could not bind: global search reached 2 of 44 account attributes, the per-table filter failed on 73 attribute/entity pairs, and clusters were absent from it entirely. Every view now selects what its entity advertises.

  • Findings rows and detail pages disagreed on five icons: Twice they were swapped outright: the person glyph meant account in a row and user one click later, and the key meant user in one place and KV store in another.

  • Long values collapsed table columns instead of scrolling: Nothing floored the width of a flex table column, so the table always shrank to its wrapper, horizontal scrolling never engaged, and identifier columns degraded to a few characters while headers painted over the next column.

  • A one-line CLI table wrapped the whole box apart: Nothing bounded table width, so a long schema comment drew db schemas a thousand columns wide.

  • Windowed backups dropped the identity their rows named: --start applied its lower bound to the identity and options tables too. Identity is written once at an entity's first-seen epoch, so a window dropped it for anything older, and every view and check macro inner-joins one: the extract opened, counted plausibly and reported no findings. Identity is now narrowed to what the window's rows reference rather than filtered by time, and the copy verifies this before saving. A related refusal is relaxed: where the source database has lost an identity row, a faithful copy carries that gap, and no re-run could do better, so the source is asked whether it still holds the orphaned keys and the backup is kept with a warning naming them.

  • insights web could not be configured from a config file: Its web:, prometheus: and updater: sections were read from the file for no command but serve and the collector, so the UI bound loopback and a published port answered nothing, with no error to point at.

  • Systems read as stale on every load: The fleet list bucketed freshness on a fixed wall-clock age, so any system whose scrape cadence exceeded 90s was always stale. Staleness now allows two missed intervals plus a processing buffer, relative to each system's own cadence.

  • Stopping a container skipped the database checkpoint: insights serve handled only an interrupt, so SIGTERM, which is how Docker, Kubernetes and most supervisors stop a process, ended Insights without its shutdown checkpoint and the next start replayed the write-ahead log. SIGTERM now runs the same clean shutdown as Ctrl-C.

  • Replicated streams and consumers were re-keyed every epoch: Each replica of an R>1 stream or consumer reports its own creation time, microseconds apart, and a follower can report a stale configuration mid-propagation. Every report was treated as authoritative, so a consumer's key was minted again on nearly every report, its findings resolved and fired again each epoch, and a follower could add a second options row that duplicated view rows until the configuration next changed. Identity and configuration now come from the leader alone.

  • Scrapes dropped replies on large or busy systems: routez asked each server for every route's subscription list, which grows with the whole subject space, so on a system with millions of subscriptions the replies overran the scraper's pending limit, were dropped, and stalled the endpoints queued behind them. It is now asked with the lists off. The paged connz, jsz and subsz walks stopped when one idle server answered last and skipped a record at every page boundary. A scrape that overran its interval forfeited a whole epoch slot, one undecodable reply discarded every server's reply for that endpoint, and a reply too large to republish under max_payload was rejected along with the rest of its batch; the scraper now splits such a reply and the indexer reassembles it.

  • One server that stopped answering cost the whole fleet its epochs: A server can stall on its monitoring endpoints while its peers still route to it, which is what a rolling upgrade looks like from the outside. Every broadcast and every page waited the full timeout on it, so a single mute server stretched one scrape across several intervals and every other server lost the epochs in between. A server that lets a broadcast run out its timeout is now not waited for again for the rest of that scrape, and its replies are still taken if they arrive before the rest of the fleet has answered. Only fleet-wide requests judge a server this way, since those are the ones every server answers, and a reply that failed to decode judges nobody, because it carries no server to judge.

  • Tables listed an account's entities more than once: hx.account_ident can hold the same key and name at more than one epoch, and nine table views joined it directly, so every stream, consumer, connection, user, leaf node, service, KV store and object store of such an account was listed once per identity row, and page totals grew with it. A migration also deletes the repeated rows, keeping each identity's first observation, so a query of your own that joins the table stops multiplying too.

  • A measurement nobody took was drawn as a measurement of zero: A rate needs the same server's sample in the previous epoch. A server that had just restarted, which mints a new key, or that missed that epoch, has none, and the servers table drew its throughput and slow-consumer rates as 0, which reads as a server carrying no traffic. Those cells now show as unavailable and say why. Sparklines had the matching problem: a point with a missing epoch on both sides had no neighbour to draw a line to and disappeared entirely, hiding the samples around a gap as well as the gap. Such a point now renders on its own, and a dashed line spans each gap so the break is visible. The dashed span is interpolated between the samples on either side, never a measurement, and the hover readout leaves it out.

  • A page past the end of a shrunken table showed nothing: An empty page beyond the first carried no total, so a table that shrank while you were on a later page, or a filter applied there, rendered no rows and a zero count. The table now falls back to its last page with the real total.

  • A check's stream count linked to a list missing some of them: The System page counts a stream check's findings on KV and object store backing streams too, but the streams page it linked to hid those streams. Under a check filter the streams page now lists backing streams as well, named with their kind and linked to their own detail page.

  • Detail page headers lost their parents or overflowed: The consumer page rendered with no account, stream or server breadcrumb, and the object store page lost its account and server, because the lookups ran on an already-cancelled context. A long name, such as a subject-shaped consumer name or an account key, pushed the header past the page edge rather than truncating, and the loading skeleton reserved no room for the header, so the page jumped when it arrived.

  • Selected and disabled states that never showed: Bindings written with a hyphen in the attribute name were never applied, so the relation cards' selected border and two buttons' disabled state had no effect.

Considerations​

Every check code is spelled differently, and a stale one refuses the boot. This is the largest upgrade hazard in the release. Two changes stack. First, codes moved off the grouping they were written under and onto the entity a finding is actionable at, which retires the CLUSTER_, JETSTREAM_, CHANGE_, META_ and OPT_* prefixes entirely (there were seven CLUSTER_ codes at v0.1.10 and there are none now). Then the whole set was respelled lowercase and hyphenated. ACCOUNTS_005 is now account-005; CLUSTER_007 is now server-021.

The same string is a contract on more than one surface, so budget for all of them:

  • check-thresholds and disabled-checks in your config. An unknown code here now fails startup rather than being warned about and ignored, in both directions: an unknown disabled-checks entry used to leave the check on and flood an operator who thought they had silenced it. The case-and-underscore change is suggested by name, so SERVER_001 fails with a message naming server-001. A code whose number or entity also changed is not, and fails with a plain "unknown check code"; the upgrade guide maps every old code.
  • Parameter names under check-thresholds. A name that no longer exists fails startup the same way and lists the valid ones (rtt_ms under connection-001 fails with valid: rtt), where it used to be skipped with a warning while the check ran at its default. Percentages are _percent on a 0 to 100 scale, and durations and sizes take typed values such as 100ms and 1 MiB.
  • Webhook and Alertmanager subscriptions (and the new ignore list). A pattern matching no known code also fails startup.
  • The Alertmanager alertname, and the finding feed's subject token.
  • hx.check_findings.code, which agents and hand-written SQL query directly (see the table rename below). A migration rewrites the stored findings in one transaction, so history is preserved and open findings keep their age. The seven codes whose check was retired are dropped rather than remapped (so are rows left under codes retired before v0.1.10), and one rescoped check starts fresh because its stored key referenced the wrong kind of entity.

server-049 is retired differently: its stored findings are kept, and a check-thresholds, disabled-checks, subscriptions or ignore entry naming exactly server-049 is accepted with a warning and has no effect.

hx.check_results is now hx.check_findings. Every row in the table is an entity that tripped a check: the macros emit only what fails, there is no pass row and no severity meaning "ok". The rest of the domain already said finding (the feed subject, the API, the UI), so the table now does too. An automatic migration renames it in place, which is a catalog-only operation and does not copy rows. Update any saved query, dashboard or agent prompt that names the old table; there is no view left behind under the old name.

Flat NATS subjects are gone. Every $INS subject carries a system or node id ($INS.sys.<id>.db.query, $INS.node.<id>.ops.pprof); $INS.ping is the only flat one left, where collision is the mechanism rather than the hazard. There is no compatibility knob. The gain is that a shared NATS account is safe by default: several single-system nodes on one account used to take the same flat namespace, where two JetStream streams sharing capture subjects each store the other's messages and duplicate micro endpoints queue-balance requests across different databases, both silently and both answering with another system's data. Federated deployments no longer need the deny_exports / deny_imports lines that worked around it. A v0.1.10 client speaking flat subjects to a v1.2.0 server will not resolve; every bundled client discovers an id first.

The HTTP gateway addresses a system in the URL path. The form is POST /sys/<id>/db/query, and likewise for columns, explain, macros, schemas and tables, matching the subject token for token. There is no default system and no cached topology, so nothing can be stale and no request is answered by a system other than the one it named. GET /systems lists them. A caller posting to the flat /db/* paths has to add the system.

Stream and consumer enum columns hold the NATS API spelling. The indexer stores what the NATS API sends rather than Go's display spelling: storage_type is file or memory, retention_policy is limits, interest or workqueue, discard_policy is old or new, store_compression is none or s2, and persist_mode is default or async; replay_policy and priority_policy carry no quotes. A migration rewrites the rows already stored. Update saved SQL that compares against the old spelling, which now matches nothing.

entity_name is an alert annotation, not a label. A name resolves from an identity row that can arrive after the finding, and as a label it changed the Alertmanager fingerprint and fired a stable alert again. Alertmanager routes, inhibitions and silences matching entity_name stop matching; match on entity_key or entity_pk, which remain labels. Every payload also carries a system label.

A data directory now records the ids it was built under. identity.json records the node and system ids of the last run and the names they resolved to. Renaming an id in config used to silently strand the catalog and start an empty history; it must now be declared. Run insights config check before the upgrade to see what a start would no longer claim, and insights config migrate (against a stopped instance) to move the catalog onto the new names, with --rename system:<old>=<new> where ids actually changed. Run with no flags it adopts the new naming scheme, which is the ordinary upgrade case, where the ids held still and the names beneath them moved.

ops backup changed shape. Saving is now insights ops backup save (still the default subcommand, so an existing invocation keeps working), it builds the file on the client, which means the disk it needs is yours rather than the server's, and it must run at the same Insights version as the server, since a schema difference is refused up front. Three things went with the old server-side path:

  • --output - no longer exists. The copy writes into a seekable DuckDB file, so the backup cannot be piped to another process until it is complete.
  • The $INS.db.backup endpoint is retired in favor of $INS.sys.<id>.db.backup.stream.
  • The backup.* config section is gone, including backup.object-store-ttl / INSIGHTS_BACKUP_OBJECT_STORE_TTL. Backups are no longer uploaded to an object store.

--force is required when running non-interactively, since the command prompts for confirmation at a terminal.

The Insights image has no shell. It is built on distroless static and still runs as UID 1000 with /var/lib/insights owned by it, so a data volume that worked with v0.1.10 needs nothing. Anything that runs a command inside the container through a shell or a package manager, such as docker exec <container> sh, an exec probe calling sh -c, or an init step that reuses the image to chown a volume, needs a different image for that step. The new collector image is built the same way and needs no data volume.

Web UI URLs are path-scoped. Every page lives under /systems/<id>/…. On a node serving one system, old flat paths redirect into it and preserve the query string, so bookmarks and deep links keep working; on a node serving several they open the system list. Any tooling that constructs Insights URLs should be updated to the new form.

query is now db query. Every other $INS.db.* front-end already sits under db. The old spelling still works as a hidden alias on the same command, so scripts do not break, but prefer db query in new work.

Trial builds report usage telemetry, on by default. A pulse every five minutes carries a random instance id generated locally, the release version and platform, a few configuration durations and counts, the size of the watched environment, how many checks have a finding, and usage counters: browser sessions, pages opened, table and search actions, and requests made by the CLI, coding agents, MCP and HTTP clients. A node fed by collectors also lists those collectors' instance ids. The telemetry reference lists every field. It carries nothing about the monitored system: no server, cluster, account, subject, stream or consumer names, no addresses, no message data, no configuration values, no query text. Other builds contain no reporter at all. insights web sends no pulse, but passes the page views it serves to the node holding each system's database over NATS. Set telemetry.enabled: false (or --telemetry=false / INSIGHTS_TELEMETRY=false) to switch it off, on a web node too, or telemetry.print: true to log every pulse and read exactly what would leave the process. There is no destination setting.

Notification endpoints lost two knobs, and their KV bucket. timeout and max-retries are removed from endpoint config; each delivery makes up to 3 attempts with a 10s request timeout (one attempt when the receiver answers with a 4xx other than 429), and a failed delivery is retried for that endpoint at up to one-minute intervals until it succeeds. Old config files carrying the keys still parse. Endpoint config is no longer mirrored into a JetStream KV bucket, and the delivery log moved to a stream named for the node. Nothing removes what v0.1.10 created: its webhook-endpoints KV bucket and webhook-deliveries stream stay behind, unused. Remove them with nats kv del webhook-endpoints and nats stream rm webhook-deliveries.

Two web settings are gone. web.metrics-interval and web.metrics-timeout (METRICS_INTERVAL, METRICS_TIMEOUT) configured a component metrics poller nothing in Insights displayed. Leftover YAML keys and environment variables are ignored, but the flags are rejected on the command line.

Database recovery can exit the process. A DuckDB recovery attempt that cannot return within 30 seconds now exits so a supervisor can restart the process, rather than leaving listeners serving an unusable database. Deployments should run under a restart policy (a Kubernetes Deployment, a systemd unit with Restart=, or --restart on a container).

Unnamed client commands cost a discovery round. With nothing naming a target, a client subcommand waits out the full two-second $INS.ping window so every responder can reply. Set --system (or INSIGHTS_SYSTEM, or system.id in a shared config file) for repeated invocations. A named target skips discovery entirely, so silence from it is reported as the error it is. Unnamed discovery also needs permission to publish $INS.ping and receive replies; a named call needs only its endpoint permissions.

The supported nats-server floor is 2.10.27. Earlier 2.10 patches are missing monitoring endpoints Insights relies on (raftz, ipqueuesz), which the previously documented "2.10" floor implied were present.

Agent skill downloads moved. The info dialog no longer builds and serves the per-agent skill bundles. Install them from synadia-io/insights instead, which is where the plugin marketplace and Gemini extension already lived.