Checks Reference

This document is the canonical reference for all automated checks. Each check runs against collected monitoring data to surface operational issues and optimization opportunities.

See Checks for the conceptual overview.

Conventions

• Codes are lowercase and hyphenated, such as server-001. That spelling is what you write under check-thresholds and disabled-checks, what a notification subscription matches, and what the API and stored findings carry.
• Severity levels: critical (immediate action required), warning (investigation needed), info (optimization opportunity). A check shown as warning/critical or info/warning sets its severity per finding.
• Operational checks detect current health and correctness issues at a single epoch. Results are computed during indexing and exposed through the checks API and web UI.
• Optimization checks analyze trends across a time range to surface waste, imbalance, and improvement opportunities. During indexing each one is evaluated over its lookback window (default: 6h, set per check under check-thresholds); querying its findings for a selected time range evaluates it over that range instead.
• Categories group checks by operational concern. Each check belongs to exactly one category.
• Configurable thresholds: a check whose Threshold names a parameter in backticks (such as cpu_percent) can be tuned under check-thresholds, keyed by check code and parameter name. See the check-thresholds section of the Configuration reference.
• Disabling a check: any check can be turned off by listing its code under disabled-checks. A disabled check keeps running and stays visible, marked disabled; it is left out of the health grades and never delivered as a notification.

Checks that read io.nats.monitor metadata follow the pinned jsm.go evaluator's directions, boundaries, and disabling rules. Malformed recognized metadata prevents that monitor option set from being evaluated. Mirror seen-critical controls mirror activity and peer-seen-critical controls peer activity; Insights preserves those documented meanings despite the jsm.go v0.3.0 parser assigning the mirror key to the peer option.

server-049 is retired: negotiated leafnode compression cannot establish the configured compression mode. The code is reserved and excluded from active checks. Stored findings remain available with a retirement explanation; original measurements are unavailable. Saved policy and webhook entries naming exactly this code are accepted with a warning and do not enable new evaluations.

Minimum Server Versions

A few checks read monitoring fields that older NATS Server releases do not report. Those servers are skipped rather than evaluated: an absent field arrives as a zero that is indistinguishable from a real reading, so any finding derived from it would be wrong: either silently missing or falsely raised.

The gate applies per server, so during a rolling upgrade these checks keep evaluating the nodes that have already moved. Servers below the minimum are excluded from the check's entity total rather than counted as passing.

A check with no eligible servers is shown on the system page dimmed, with a Requires NATS x.y.z+ label in place of its counts, rather than as a clean result. Upgrading past that release enables it with no configuration change. This is a different state from a disabled check: a disabled check still runs and still shows its counts, while an unsupported one has nothing to show. The minimum is also reported as min_server_version by the checks list and checks info API endpoints.

Health Indicator

Each entity in an entity table displays a health indicator derived from the findings recorded for it at the current epoch. Every recorded finding counts: from operational and optimization checks alike, and from checks listed under disabled-checks, which are left out of the health grades but not out of this indicator.

The indicator has three states. The icon carries the same meaning as the color, so the state reads without color.

Operational Checks

Server

server-001: Connection Readiness Failure

critical · Health

Flags servers reporting connection readiness failures via the healthz endpoint, or reporting unhealthy with no error detail.

Threshold. healthz status != "ok" with CONNECTION error type, or with no errors reported

Remediation. Check for listener port conflicts, TLS certificate errors and permission issues. Find the root cause before restarting.

server-002: Server Version Mismatch

info · Consistency

Identifies servers running a different software version than the cluster majority. Unclustered servers are compared as one group. On an even split, the lexically higher version string counts as the majority.

Threshold. Version differs from majority

Remediation. Complete the rolling upgrade so all servers run the same version. If intentional, verify compatibility between versions.

server-003: High CPU Usage

warning · Performance

Flags servers where per-core CPU usage exceeds the CPU threshold.

Threshold. cpu/cores > cpu_percent% (default: 90)

Remediation. Identify the workload driving CPU usage using the /pprof/profile debug endpoint. Common causes: high-fanout subjects, subscription matching overhead, JetStream write pressure. Scale out with additional servers, optimize subject hierarchies, or increase server capacity.

server-004: Slow Consumers

warning · Performance

Flags servers with new slow consumer events since the previous epoch.

Threshold. slow_consumers delta > 0

Remediation. Find the affected clients and speed up their message processing. A slow consumer is evicted when its pending outbound bytes exceed max_pending or a write to it times out. To buffer more, increase max_pending in the server config (default 64 MiB). Otherwise add more clients, reduce message rates, or have clients subscribe to fewer subjects.

server-005: JetStream Memory Pressure

warning/critical · Saturation

Flags servers whose JetStream memory usage is approaching the configured max_memory limit, where writes to memory-backed streams are rejected. Severity rises to critical near the limit.

Threshold. js_memory > warn_percent% (default: 90) of js_max_memory; critical above crit_percent% (default: 95)

Remediation. Reduce memory-backed stream usage, raise max_memory in the server config, or convert large memory-backed streams to file storage. Usage is measured against max_memory, the hard limit, which is a different number from js_reserved_memory (the sum of memory-backed stream MaxBytes reservations).

server-007: Authentication Not Required

info · Health

Flags servers that do not require client authentication. It misses a server with no_auth_user configured, which accepts clients without credentials as a default user while still reporting auth_required=true. That setting is not scraped, so this check cannot see it.

Threshold. auth_required = false

Remediation. Enable authentication in the server configuration. Options: NKey authentication, JWT-based operator mode with signed credentials, user/password, TLS certificate mapping, or auth callout for centralized policy enforcement. Avoid no_auth_user in production.

server-008: Unexpected Server Restart

warning/critical · Change

Flags servers that restarted without a version upgrade or configuration change, and escalates when restarts repeat. Start times are compared across epochs by cluster and server name, since a restart mints a new server ID. A boot whose version or configuration digest differs from what the previous run last reported is treated as planned, as is a rollback. A server that reports no configuration digest is judged on version alone. The digest covers the config after includes and environment variables are resolved, so a value that changes every boot (such as a pod IP in client_advertise) makes every restart look planned and silences this check for that server.

Threshold. start_time changed; version and config_digest unchanged (an unreported digest is unknown, so version alone decides); critical on every epoch while unplanned boot generations in the trailing hour exceed max_restarts (default: 2)

Remediation. Check the NATS server logs and system logs (journalctl, dmesg) for the exit reason. Common causes are an OOM kill, hardware failure, a process crash or an external signal. If restarts repeat, also check file descriptor limits (ulimit), JetStream store corruption on startup, port conflicts with another process, TLS certificate expiry and account resolver timeouts. Verify disk space and ulimit settings before restarting.

server-010: High Route RTT

info/warning · Performance

Flags a route whose round-trip time to a cluster peer is well above what the rest of the cluster sees. Routes are compared within their cluster, so a cluster whose routes are all slow is not reported. The average excludes the reporting server's own routes, so a server with uniformly slow routes does not raise its own baseline.

Threshold. RTT > multiplier (default: 2) x the other servers' route average and >= min_rtt (default: 10ms); warning above severe_multiplier (default: 3)

Remediation. Compare this route's network path and server load with the cluster's other routes. Different availability zones, routing paths, congestion, packet inspection or an overloaded server can explain the difference. Route latency slows cross-server request/reply and JetStream replication.

server-011: Connection Count High

warning/critical · Saturation

Flags servers where active connections approach the configured maximum. Critical means the server has almost no connection slots left, and at its cap it refuses new clients. max_connections defaults to 65536, so the check rarely fires unless a lower limit is set.

Threshold. connections > warn_percent% (default: 80) of max_connections; escalates to critical above crit_percent% (default: 99)

Remediation. Increase max_connections in the server config or per-account limits. Distribute clients across more servers using DNS round-robin or a load balancer. Identify and close unnecessary connections.

server-012: Stale Connections

info · Errors

Flags servers with new stale connection events since the previous epoch.

Threshold. stale_connections delta > 0 (NATS Server 2.12.0+)

Remediation. The server sent PING and the client did not answer with PONG after the configured number of unanswered pings (defaults: ping_interval 2 minutes, ping_max 2). Common causes are a network partition or firewall dropping idle connections, a client process that crashed without closing its TCP connection, or a blocked client event loop that cannot process PINGs. Check firewall idle timeout settings and client health.

server-013: Stalled Clients

warning · Performance

Flags servers with new stalled client events since the previous epoch.

Threshold. stalled_clients delta > 0 (NATS Server 2.12.0+)

Remediation. The server is throttling a fast producer because a downstream client cannot keep up. Improve client message processing speed, add more client instances, or increase max_pending to buffer more. This is normal flow control and an early warning before slow consumer eviction.

server-014: JetStream Subsystem Not Ready

warning/critical · Health

Flags a server that cannot serve JetStream because JetStream is disabled in its configuration or its meta layer is still recovering. Disabled is critical from the first scrape. Recovering is a warning, and turns critical once it has been reported on every scrape for longer than the sustain window. Other JetStream health problems are reported by checks that name the Raft group.

Threshold. healthz JETSTREAM error in the not_ready family; a recovering meta layer escalates to critical once held > sustain (default: 15m), JetStream disabled is critical at once

Remediation. A booting server's meta layer recovers and clears on its own. If it is still recovering past the sustain window, it is stuck. Check its Raft state and whether the meta group's other peers can reach it. If JetStream is disabled here but enabled on its peers, compare this server's JetStream block with theirs.

server-015: JetStream Asset Configuration Rejected

critical · Consistency

Flags a stream or consumer whose configuration the cluster refused. The asset does not exist on this server and will not appear until the configuration changes, so unlike most health findings this one does not clear on its own.

Threshold. healthz STREAM or CONSUMER error in the config family

Remediation. The cause gives the server's own objection, such as an ack wait that must be positive, subjects that overlap the JetStream API, or a limit the account cannot satisfy. nats stream info and nats consumer info report the asset as missing because it never came up. Fix the configuration and recreate the asset. Restarting the server does not help.

server-054: JetStream Asset Missing

critical · Consistency

Flags a stream or consumer that was assigned to this server and that the server cannot find. Unlike a rejected configuration, the asset is supposed to be here. A placement change, a failed catch-up or a delete that did not propagate can cause this.

Threshold. healthz STREAM or CONSUMER error in the missing family

Remediation. Check whether the asset exists elsewhere in the cluster with nats stream info or nats consumer info. If it does, this server missed it: check the Raft group's peer list and whether a catch-up is stuck. If it exists nowhere, a delete propagated partially and the assignment on this server is a leftover.

server-055: JetStream Store Write Error

critical · Health

Flags a stream whose message store refused a write or could not be recovered, so it cannot persist messages on this server. After a refused write, a replicated stream steps down and stays an observer here so its replicas elsewhere can take over. server-064 reports a failed write to a Raft group's own write-ahead log, naming the group.

Threshold. healthz error in the store family

Remediation. Check free space, inode exhaustion, permissions and filesystem health on the JetStream store directory. The cause includes the operating system's error. A replicated stream has already stepped down here, so confirm a replica on another server leads it before repairing this one.

server-056: JetStream Asset Raft Fault

critical · Health

Flags a stream or consumer whose Raft group is in a state this server cannot serve from, such as a node skew, a stopped monitor or a group reporting itself unhealthy. It also catches any cause the other health checks do not recognize, so a condition introduced by a newer server still gets reported.

Threshold. healthz asset or JetStream error in the raft family

Remediation. These are server-side faults, not problems with the asset's configuration. Check the group's peers and leader with nats server report jetstream, and whether this server is partitioned from them. A restart usually clears a skew or a stopped monitor, but find out why it happened first. Report recurring skew upstream as a bug.

server-064: Raft WAL Write Error

critical · Health

Flags a Raft group whose write-ahead log failed a write it could not tolerate. nats-server then shuts down the group's node, so every asset in the group is unavailable on this server until the fault is fixed.

Threshold. wal_error is set on a Raft group

Remediation. The write failed on the JetStream store directory, so check free space, inode exhaustion, permissions and filesystem health on the server. An out-of-space or permission error also disables JetStream server-wide, so other groups may be affected too. The node does not recover on its own. Restart the server once storage is fixed.

server-065: Server Not Reporting

warning · Health

Flags a server with no VARZ observation at an epoch when another server reported VARZ and a peer reported a route to it. Checks that use STATSZ values (CPU, traffic, storage and other server measurements) can still fire for it. Only servers with cluster routes are covered, so a solo server, a single-member cluster or a node reachable only over gateways is never reported. A server the scraper's server filter excluded after it was scraped can also match while peers still route to it. It fires only on epochs ROUTEZ also ran in.

Threshold. no VARZ observation while a peer reports a route to the server and another server reports VARZ at the same epoch

Remediation. Check the scraper's server filter, system account permissions and request errors, then request VARZ directly over the scraper's connection. Compare which other endpoints and STATSZ values still respond. If monitoring requests hang while STATSZ responds, take a goroutine profile with PROFILEZ and look for blocked monitoring handlers or stream operations, since a slow shared handler can delay VARZ. This finding does not identify the cause.

server-016: Account Resolution Failure

warning · Consistency

Flags servers that could not resolve a JetStream account. The finding names the account, which does not appear in the account inventory because it was never resolved.

Threshold. healthz ACCOUNT error type

Remediation. Read the cause. A validation failure means the resolver returned a JWT the server rejected, so check the account JWT's signature and issuer against the operator it claims. An expired account needs its JWT reissued. A lookup failure points at the resolver. Check connectivity to it and whether it holds this account.

server-018: High Gateway RTT

warning · Performance

Flags gateway connections whose round-trip time is far worse than other servers' links from the same cluster to the same remote cluster. A cluster pair whose links are all slow is not reported. A link is only compared when another server in the same cluster has a link to the same remote cluster, and servers without a cluster name are skipped.

Threshold. RTT > multiplier (default: 2) x the average RTT of other servers' links from the same cluster to the same remote cluster, and >= min_rtt (default: 25ms)

Remediation. Compare this server's path to the remote cluster with its peers' paths. Check availability zones, NAT hops, network interfaces, packet inspection and load on both ends, since servers in the same cluster can differ in distance and routing.

server-019: JetStream Storage vs Configured Limit

warning/critical · Saturation

Flags servers where JetStream storage usage approaches the configured max_store limit, escalating to critical near the configured limit.

Threshold. storage > warn_percent% (default: 90) of max_store; critical above crit_percent% (default: 95)

Remediation. New message writes can be rejected once usage reaches max_store. This compares stream storage with that configured ceiling, which is a different number from js_reserved_storage (the sum of stream reservations) and from free filesystem space. Increase max_store in the server config, add disk capacity, or reduce stream storage by purging data or adding retention limits.

server-053: Meta Raft Group Closed

critical · Health

Flags servers whose meta Raft group has shut down while the server is still reporting. Without it the server cannot take part in JetStream control-plane decisions, such as stream and consumer creation, deletion, placement and leader election. Only the meta group is checked, since stream and consumer groups close normally when their asset is deleted.

Threshold. meta Raft group closed while the server still reports

Remediation. Check the server logs for the shutdown reason. A meta group closes on a write error, a failed store recovery or an explicit shutdown. Confirm JetStream storage is readable and has free space, then restart the server to rejoin the meta group.

Cluster

server-057: Memory Usage Outlier

warning · Saturation

Flags servers whose memory usage exceeds the cluster average by more than the configured multiplier.

Threshold. memory > multiplierx (default: 1.5) cluster average

Remediation. Investigate what is consuming memory on the outlier server. Common causes include large Raft state, many subscriptions, or memory-backed streams. Consider rebalancing workload.

server-059: High HA Assets

warning · Saturation

Flags servers with more highly-available JetStream assets than the warning count. Each replicated asset carries Raft memory, disk and network overhead, so a restart or failure of the server is costly and its load is hard to rebalance.

Threshold. ha_assets > warn_count (default: 1000)

Remediation. Distribute streams and consumers across more servers. Consider reducing replica counts on low-priority streams or consolidating small streams.

server-060: Cluster Name Whitespace

warning · Consistency

Flags servers whose configured cluster name contains whitespace. nats-server rejects a literal space at startup but does not check for tabs, newlines or carriage returns, which leave an error-prone name.

Threshold. Whitespace in cluster name

Remediation. Remove whitespace from the cluster name in the server configuration and restart the affected servers.

server-061: Route Count Low

warning · Health

Flags a server with fewer routes than its cluster peers require. For each peer where both sides pool routes, NATS opens the larger pool size plus a dedicated route for each account pinned on both ends, with a floor of one for the system account. A pair with pooling disabled needs one route. Pinned accounts are counted from observed routes, so the expected count is a lower bound.

Threshold. Routes < sum of negotiated pool plus inferred pinned routes per peer; one route when either peer disables pooling

Remediation. Check cluster connectivity, route URLs, pool sizes and each peer's pinned-account configuration. A dedicated account observed on both ends should have a route between them. Pinned-account configuration is not scraped, so an account whose routes have all disappeared is not counted, and a clear result does not prove every dedicated route is present.

server-020: Connection Count Change

warning · Errors

Flags servers whose connected-client count rose or fell sharply between epochs. A restarted server starts a new baseline, so the connections it drops on restart are not flagged on that server.

Threshold. abs(connection delta) > churn_percent% (default: 50) of the previous count, on a baseline of at least min_conns (default: 100)

Remediation. First rule out a planned drain or rolling restart. A server in lame duck mode sheds its clients on purpose, and this check cannot tell that apart from churn because lame duck state (the INFO frame and the $SYS.SERVER.*.LAMEDUCK event) is not scraped. Otherwise check, in order: misconfigured client reconnect backoff (the default is randomized jitter, not zero), authentication failures that disconnect clients right after connecting, slow consumer evictions causing reconnect loops, load balancer health checks opening short-lived connections, and network instability. Review client connection URLs and reconnect settings.

server-021: Gateway Disconnection

warning · Health

Flags servers that lost their connection to a remote cluster since the previous epoch. Links are tracked by server and remote cluster name, so a reconnect that gets a new gateway ID does not count as a loss. A server whose monitoring endpoints did not answer in the current epoch is not flagged.

Threshold. (server, remote cluster) link present at prev epoch but missing at current

Remediation. Gateways reconnect automatically with randomized jitter, so a brief network issue clears on its own. If the disconnection persists, check TLS certificate validity (including OCSP stapling if enabled, where stale OCSP responses are a common cause), confirm firewall rules between clusters allow the gateway port, and confirm gateway names are consistent across all clusters.

server-022: Gateway Config Mismatch

warning · Consistency

Flags servers whose set of gateway connections differs from the cluster majority. The majority set must be held by more than half of the cluster's servers with gateways, and at least 3 must have gateways. A server with no gateway connections is not compared.

Threshold. Gateway set differs from the set held by a strict majority of the cluster's servers with gateways (at least 3)

Remediation. Give every server in the cluster the same gateway configuration, since asymmetric gateways cause routing failures. With reject_unknown_cluster enabled, unlisted gateways are rejected. With it disabled (the default), gateways can be discovered through gossip. Check that every intended gateway is listed and that TLS configuration matches on both sides.

Account

account-001: Account Connection Limit

warning · Saturation

Flags accounts whose connections, summed across every server that reports them, exceed a configured share of the limit. The limit applies to the account as a whole, so an account spread over many servers is judged on its total.

Threshold. connections summed across servers > warn_percent% (default: 90) of the account limit

Remediation. Raise max_conn in the account JWT, or reduce the account's connections. Moving clients to other servers doesn't help, because clients on every server count toward the same total. Each account has its own limit.

account-002: Slow Consumers

warning · Errors

Flags accounts with new slow consumer events since each server's previous observation, aggregated across servers. A counter reset or a server's first observation counts as no new events.

Threshold. slow_consumers delta > 0

Remediation. Identify the affected consumers within this account and increase their processing throughput. Consider adding more consumer instances or reducing message rates.

account-003: Inactive or Unresolved JWT Import

warning · Consistency

Flags inactive JWT imports, and duplicate import declarations where the scrape can't tell which one activated the import. Each finding names the likely cause, such as a missing or expired activation token, a token signed by a rotated signing key, or a missing source export.

Threshold. JWT import inactive, or its runtime attribution ambiguous

Remediation. Act on the reason shown for each import. For 'Runtime import attribution is ambiguous', remove or disambiguate the duplicate JWT import declarations. For 'Import subject does not satisfy export account-token position', put the importing account's identity in the configured subject token. For 'Missing activation token', request one from the exporting account's operator. For 'Expired activation token', renew it with the exporting account. For 'Activation token signed by rotated signing key', re-issue it with the current signing key. For 'Source account not found', check the source account in the import and confirm that account is loaded on the servers. For 'Source export not found', verify the export exists in the exporting account's JWT.

account-004: Orphaned Export

info · Consistency

Flags exports in non-system accounts with no matching importer in any account. An import matches when it names this account as its source, has the same type, and its subject is fully covered by the export subject, using NATS wildcard matching.

Threshold. No matching importer in any account

Remediation. Remove the unused export from the account JWT, or create the intended import in the consuming account.

account-005: No Subscription Interest

info · Consistency

Flags active stream imports into non-system accounts that no current subscription in the importing account matches, using NATS wildcard subject matching. A remapped import is matched on its local subject. Inactive imports are reported by account-003. Accounts without a complete current subscription snapshot are skipped, so partial or unknown subscription data never counts as no interest.

Threshold. No subscriptions on imported subject

Remediation. Verify clients are subscribing to the correct subject. Remove the import if it is no longer needed.

account-006: Account Subscription Limit

warning · Saturation

Flags accounts with an open client or leafnode connection near the account's subscription limit. NATS applies this limit to each connection separately, so subscriptions on different connections are never added together. A lower user or server subscription limit can cap a connection before it reaches the account limit.

Threshold. most saturated open connection > warn_percent% (default: 90) of the per-connection limit

Remediation. Raise max_subs in the account JWT, or cut the subscriptions on the busiest connection. Spreading subscriptions over more connections helps, and consolidating clients onto fewer connections makes it worse. Removing unused subscriptions helps on any connection near the limit.

JetStream

stream-002: Stream Replica Lag

warning · Consistency

Flags stream replicas whose last sequence number trails the leader's by more than the lag percentage of the leader's last sequence. Replicas reporting sequence 0, such as during a snapshot sync, are skipped.

Threshold. Replica > lag_percent% (default: 10) behind leader

Remediation. Check the lagging replica's server for CPU, disk I/O or network contention. Replicas catch up on their own through Raft. If catchup stalls, force a full snapshot sync as a last resort by removing the peer with nats stream cluster peer-remove and adding it back.

stream-003: High Subject Cardinality

warning · Saturation

Flags streams with more unique subjects than the configured maximum.

Threshold. num_subjects > max_subjects (default: 1,000,000)

Remediation. Review the subject naming scheme. Consider partitioning high-cardinality data across multiple streams or using a flatter subject hierarchy.

stream-004: Stream Message Limit

warning · Saturation

Flags streams where message count is above the warning percentage of the limit.

Threshold. messages > warn_percent% (default: 90) of limit

Remediation. Set max_age to expire old messages, increase max_msgs, or set max_msgs_per_subject to spread the limit across subjects.

server-062: JS API Request Rate High

warning · Performance

Flags servers where the JetStream API request rate exceeds the threshold. The rate is measured between consecutive epochs, so a server missing from the previous epoch is not evaluated.

Threshold. Per-server API request rate > max_rps (default: 50) req/s

Remediation. Identify the source of API requests. Common causes: rapid stream/consumer creation, excessive info lookups, or tight-loop API calls. Reduce concurrent API call volume. The server queues requests internally and publishes an advisory when the queue saturates. Cache JetStream info responses client-side where possible.

server-025: JS API Pending High

warning · Performance

Flags servers where JetStream API inflight requests exceed the threshold.

Threshold. Inflight requests > max_pending (default: 1000)

Remediation. Reduce the rate of concurrent JetStream API calls. Check for clients making synchronous API calls in tight loops.

system-003: Consumer Count Change

warning · Errors

Flags when the total consumer count rises or falls by more than the threshold between consecutive epochs.

Threshold. Consumer count delta > max_delta (default: 5000) per epoch

Remediation. Find what is creating and deleting consumers so quickly. Use durable consumers so clients reuse them instead of recreating them. If ephemeral consumers are deleted during brief pauses in activity, raise inactive_threshold (default 5s).

stream-007: Stream Quorum Lost

critical · Health

Flags replicated streams where enough replicas are offline to lose quorum. Offline replicas are counted from the stream leader's report, so a stream left with no leader is not flagged here, and server-044 reports its leaderless Raft group.

Threshold. Online replicas below a strict majority of num_replicas (R > 1)

Remediation. Bring the offline replicas' servers back online. If a server is permanently lost, remove its peer with nats stream cluster peer-remove to lower the quorum requirement so the remaining peers can elect a leader.

server-028: JS API Error Rate High

warning · Errors

Flags servers where JetStream API errors exceed a percentage of total requests.

Threshold. error_delta / total_delta > error_percent% (default: 1) with total_delta >= min_requests (default: 100)

Remediation. Check server logs for the API error types. Common ones are permission denials (403), stream or consumer not found (404) and resource exhaustion (503). High error rates usually come from client misconfiguration, such as wrong stream names or missing permissions, more often than from the server.

stream-005: Stream Byte Limit

warning · Saturation

Flags streams where byte usage is above the warning percentage of the limit.

Threshold. bytes > warn_percent% (default: 90) of limit

Remediation. Review the retention and discard policies, then raise max_bytes or set max_age to expire messages you don't need. S2 compression reduces disk use but not the logical bytes counted against this limit.

stream-006: Stream Consumer Limit

warning · Saturation

Flags streams where consumer count is above the warning percentage of the limit.

Threshold. consumers > warn_percent% (default: 90) of limit

Remediation. Remove unused or inactive consumers, increase max_consumers, or consolidate consumers that read overlapping subject filters.

stream-008: Stream Subject/Message Count Inconsistency

warning · Consistency

Flags streams that report more unique subjects than total messages, which is an inconsistent stream state. Streams with no messages are skipped.

Threshold. num_subjects > msgs > 0 (invariant violation)

Remediation. Recheck the stream's state and the server logs, and contact support if the inconsistency persists. The finding doesn't show the cause or whether stored messages are damaged.

stream-009: Stream Replica Message Count Divergence

info · Consistency

Flags replicated streams whose replica message counts differ by at least the absolute threshold and by more than the proportional one. Only the leader and replicas it reports as current and online are compared.

Threshold. Spread among sampled, leader-confirmed current online replicas > divergence_percent (default: 5) and absolute spread >= min_divergence (default: 1000)

Remediation. Compare replica counts and sequences with nats stream info --all, and check again once publishing and retention settle. Counts are sampled independently, so one spread doesn't mean corruption or lost data. If the difference persists, check peer health, Raft progress and server storage errors.

stream-010: Mirror Last Seen Staleness

warning · Consistency

Flags zero-lag mirrors that have been inactive longer than the stale threshold while their local source stream advanced during that time. Mirrors of external sources or of themselves, and ambiguous source reports, are skipped.

Threshold. inactive longer than stale (default: 5m), with zero lag and local source advancement during that time

Remediation. Compare the current state of the mirror and source leaders. Check mirror errors and connectivity before choosing a recovery action.

stream-011: Mirror Lag Critical

warning · Consistency

Flags mirror streams where mirror lag is at or above the operator-defined io.nats.monitor.lag-critical threshold. Requires io.nats.monitor.enabled set to true and no invalid monitor metadata (stream-034).

Threshold. mirror_lag >= operator-defined io.nats.monitor.lag-critical metadata threshold (a threshold of 0 disables)

Remediation. Check network connectivity to the mirror source and resource contention on the mirror server.

stream-012: Mirror Seen Critical

critical · Consistency

Flags mirror streams where the time since the mirror was last active is at or above the operator-defined io.nats.monitor.seen-critical threshold. Requires io.nats.monitor.enabled set to true and no invalid monitor metadata (stream-034).

Threshold. mirror_active >= operator-defined io.nats.monitor.seen-critical duration (threshold of 0 or less disables)

Remediation. Verify the source stream is active and network connectivity is healthy.

stream-013: Min Sources

warning · Health

Flags streams where the source count is below the operator-defined io.nats.monitor.min-sources threshold. Requires io.nats.monitor.enabled set to true and no invalid monitor metadata (stream-034).

Threshold. num_sources < operator-defined io.nats.monitor.min-sources metadata threshold (threshold of 0 or less disables)

Remediation. Verify that all expected source streams exist and are configured correctly.

stream-014: Max Sources

warning · Health

Flags streams where the source count exceeds the operator-defined io.nats.monitor.max-sources threshold. Requires io.nats.monitor.enabled set to true and no invalid monitor metadata (stream-034).

Threshold. num_sources > operator-defined io.nats.monitor.max-sources metadata threshold (threshold of 0 or less disables)

Remediation. Remove unexpected sources or update the threshold.

stream-015: Peer Expect

warning · Health

Flags streams whose peer count, from the stream leader's report, differs from the positive value set in io.nats.monitor.peer-expect. Zero and negative values are invalid and reported by stream-034. Requires io.nats.monitor.enabled set to true and no invalid monitor metadata.

Threshold. distinct peers in the leader's report != operator-defined io.nats.monitor.peer-expect metadata value

Remediation. Check for offline replicas or verify the num_replicas configuration.

stream-016: Peer Lag Critical

warning · Consistency

Flags stream replicas whose lag, as reported by the stream leader, exceeds the operator-defined io.nats.monitor.peer-lag-critical threshold. Requires io.nats.monitor.enabled set to true and no invalid monitor metadata (stream-034).

Threshold. leader-reported peer lag > operator-defined io.nats.monitor.peer-lag-critical metadata threshold (a threshold of 0 disables)

Remediation. Check the replica's server for resource contention.

stream-017: Peer Seen Critical

warning · Consistency

Flags stream replicas where the time since the replica was last active, as reported by the stream leader, exceeds the operator-defined io.nats.monitor.peer-seen-critical threshold. Requires io.nats.monitor.enabled set to true and no invalid monitor metadata (stream-034).

Threshold. leader-reported peer active time > operator-defined io.nats.monitor.peer-seen-critical duration (threshold of 0 or less disables)

Remediation. Check whether the replica's server is offline or has network issues.

stream-018: Message Count Threshold

warning/critical · Saturation

Flags streams whose message count is at or below a positive io.nats.monitor.msgs-warn or msgs-critical minimum. Critical is checked first whatever order the two values are in, and zero disables a bound. Matches jsm.go. Requires io.nats.monitor.enabled set to true and no invalid monitor metadata (stream-034).

Threshold. msgs <= either positive metadata minimum; critical checked first; zero disables a bound

Remediation. Check publishers, retention settings and consumers that remove messages. If a small retained message count is expected, lower the minimum.

stream-019: Subject Count Threshold

warning/critical · Saturation

Flags streams whose subject count crosses the io.nats.monitor.subjects-warn or subjects-critical threshold. When warn is below critical, the thresholds are ceilings. Otherwise they are floors. A missing value counts as zero, and if both are zero or negative the check is disabled, matching jsm.go. Requires io.nats.monitor.enabled set to true and no invalid monitor metadata (stream-034).

Threshold. inclusive metadata thresholds; ascending = ceiling, equal/descending = floor; missing = 0

Remediation. Set both subject thresholds. With only subjects-critical set, warn defaults to zero, so every count below critical raises a warning. If the stream has too many subjects, review the naming scheme and consider partitioning high-cardinality data. If it has too few, check whether the publishers for those subjects have stopped.

stream-034: Invalid Stream Monitor Metadata

warning · Consistency

Flags streams with invalid values for recognized io.nats.monitor metadata keys. While any value is invalid, none of the stream's monitor thresholds are evaluated.

Threshold. Invalid recognized monitor metadata

Remediation. Correct or remove the listed metadata values. Each value must match its key's type (boolean, integer or duration), and peer-expect must be positive. Values are validated even when monitoring is disabled.

Meta Cluster

server-058: Offline Replica

warning · Health

Flags meta cluster replicas that the reporters agree are offline. A peer is not flagged while another reporter, or the peer itself, still sees it online. An offline peer stays a group member because NATS never evicts one. system-009 reports a peer leaving the group.

Threshold. Meta peer offline by reporter consensus

Remediation. Bring the offline server back online. Check server logs and network connectivity. If the server is permanently lost, remove it with nats server cluster peer-remove, which requires the remaining members to have quorum.

system-004: Meta Leader Flapping

warning · Errors

Flags when the meta cluster leader changes more times than allowed within the lookback window. Changes are counted between consecutive scrapes, so a leader that changes and changes back between two scrapes is missed.

Threshold. Leader changes > max_changes (default: 1) within window (default: 10m)

Remediation. Raft sends heartbeats every second and the election timeout is 4-9 seconds, so any disruption longer than 4 seconds starts a new election. Look for network instability between meta cluster peers (even brief packet loss can trigger elections), CPU saturation delaying heartbeats, or disk I/O stalls blocking Raft WAL writes. Run nats server report jetstream to see the current meta state.

server-063: Meta Snapshot Slow

warning/critical · Performance

Flags a server whose last local meta Raft snapshot took longer than the warning or critical threshold. A slow snapshot delays meta log compaction.

Threshold. Snapshot duration > warn_duration (default: 5s), critical > crit_duration (default: 30s)

Remediation. Reduce the number of JetStream assets (streams, consumers) to shrink snapshot size. Check disk I/O performance on meta cluster servers.

system-006: Meta Asset Replica Count High

warning · Saturation

Flags when the total number of JetStream asset replicas exceeds the threshold.

Threshold. Total replicas > max_assets (default: 5000)

Remediation. Remove unused streams and consumers. Consider reducing replica counts or consolidating small streams to reduce the total Raft group count.

system-007: Meta Quorum Lost

warning/critical · Health

Reports critical when peers reported offline leave the meta group below quorum. Reports warning when the group had a leader earlier but none appears in the two most recent scrapes. Missing JSZ responses can also cause the warning, and it doesn't mean every peer is offline.

Threshold. Observed online peers < floor(cluster_size / 2) + 1; warning if no leader is observed across two scrapes after earlier leadership

Remediation. Check server health, storage and network connectivity, and bring enough of the original meta members back online to form quorum. JetStream operations that change metadata need the meta group, while existing stream groups keep their own quorum. If members are permanently lost, follow a supported disaster-recovery procedure with backups and support. Deleting Raft state is not routine peer removal.

system-008: Even Cluster Size

info · Consistency

Flags a meta cluster with an even number of peers. An even size tolerates no more failures than the odd size one below it.

Threshold. cluster_size is even and > 1

Remediation. Use an odd meta cluster size when practical. Quorum is floor(N/2) + 1, so three and four members both tolerate one failure and five tolerate two. If an even cluster splits in half, neither side has quorum and the group loses availability. It still can't produce two leaders that commit independently.

server-030: Meta Pending High

warning · Performance

Flags when a server queues more JetStream CRUD and info requests than the configured threshold.

Threshold. pending > max_pending (default: 500)

Remediation. Check JetStream API request volume and server CPU, disk I/O, and request processing latency. Reduce excessive polling or asset churn if requests arrive faster than the server can process them. The count includes routed CRUD and info requests, not unapplied Raft entries.

system-009: Meta Cluster Size Decreased

warning · Health

Flags when the meta cluster's distinct peer count drops below its recent peak and stays down. NATS never evicts an unresponsive peer, so an offline peer still counts here. server-058 reports offline peers. The drop must last longer than the sustain time, so a single missed leader scrape doesn't raise a false alarm.

Threshold. Distinct peer count below its peak over the last baseline (default: 30m) for longer than sustain (default: 5m)

Remediation. If the peer was removed on purpose (nats server cluster peer-remove), make sure the remaining cluster has an odd number of peers. Otherwise, investigate the lost peer right away. Check its server logs, network connectivity and disk health, since a smaller meta cluster tolerates fewer failures.

system-011: Leader Disagreement

warning · Health

Flags when multiple servers report themselves as the meta cluster leader.

Threshold. more than one server reporting meta leadership at the epoch

Remediation. Several servers can briefly report leadership while an election changes terms. Raft quorum rules prevent split-brain. If it persists, check network connectivity between cluster peers and look for an asymmetric partition, then restart the lower-term leader to force convergence.

Service

service-001: Service Version Mismatch

warning · Consistency

Flags services where instances report different client versions or languages.

Threshold. Instances differ in lang/version

Remediation. Complete the rolling deployment so all instances run the same version. If intentional, verify compatibility between versions.

service-002: Service Down

warning · Health

Flags services that were present in an account's previous complete subscription snapshot and missing from the next complete one. Partial or unknown snapshots are skipped.

Threshold. Instances at the account's previous complete subscription snapshot and none at the next complete one

Remediation. Restart the service instances. Check application logs and orchestration platform (Kubernetes, systemd) for failure reasons.

Leafnode

leafnode-001: Leafnode Name Whitespace

info · Consistency

Flags leafnode connections whose remote server name contains whitespace. Whitespace makes the name hard to use in commands and to match across logs.

Threshold. Whitespace in leaf name

Remediation. Remove whitespace from the server name in the leafnode's configuration and restart it.

leafnode-002: High Leaf RTT

warning · Performance

Flags leafnode connections with round-trip time exceeding the threshold.

Threshold. rtt > rtt (default: 100ms)

Remediation. Check network latency and congestion between the leaf server and the hub. Consider moving leaf servers closer to their hubs.

leafnode-003: Leafnode Subscription Count High

warning/critical · Saturation

Flags leafnode connections carrying more subscriptions than the warning threshold. The hub processes those subscriptions when the leaf connects, and if that takes longer than the 2-second leafnode connect timeout, the connection drops and retries in a loop.

Threshold. subscriptions > warn_subs (default: 20,000) / crit_subs (default: 50,000)

Remediation. Reduce the subscriptions propagated across the leafnode. Use explicit exports and imports instead of wildcard subscriptions, or consolidate subscriber applications.

Connection

connection-001: High Client RTT

warning · Performance

Flags open client connections whose round-trip time exceeds the RTT threshold.

Threshold. rtt > rtt (default: 100ms)

Remediation. Move the client closer to the NATS server or use a leafnode to bridge the distance. Check for network congestion.

connection-002: Client Pending Pressure

warning · Performance

Flags client connections whose queued outbound bytes exceed a percentage of the server's max_pending. The server closes a connection as a slow consumer once its queue reaches max_pending, and past three quarters of it the server stalls publishers sending to that client.

Threshold. pending_bytes > warn_percent% (default: 75) of the server's max_pending

Remediation. Speed up the client's message processing or reduce the publish rate to this subscriber. Raising max_pending buys time but holds a larger backlog in server memory. Pending bytes are sampled once per scrape, so a backlog that fills between scrapes can hit the limit without this check seeing it. server-013 (stalled clients) and server-004 (slow consumers) keep cumulative counts of those events and miss none.

connection-003: Connection Stopped

info · Errors

Flags connections closed for an abnormal reason. Graceful client closes ('Client Closed') and planned server shutdowns ('Server Shutdown') are excluded.

Threshold. stop_time set with a reason other than 'Client Closed' or 'Server Shutdown'

Remediation. Check the stop reason in the detail column. 'Slow Consumer (Pending Bytes)' and 'Slow Consumer (Write Deadline)' mean the client couldn't keep up with the message rate. 'Authentication Failure' means invalid or expired credentials. 'Maximum Connections Exceeded' means a server or account limit was reached. 'Stale Connection' means the client missed ping/pong keepalives.

Consumer

consumer-001: Consumer Replica Offline

warning · Health

Flags consumer replicas that are reported as offline.

Threshold. is_offline = true

Remediation. Bring the offline server back online. Check server logs for the reason the replica went offline.

consumer-002: Consumer Replica Lag

warning · Consistency

Flags online consumer replicas lagging behind the leader by more than the maximum allowed number of operations. Offline replicas are reported by consumer-001.

Threshold. online replica lag > max_lag (default: 1000) operations

Remediation. Check for resource contention on the lagging server. If persistent, consider removing and re-adding the consumer replica.

consumer-003: Consumer Quorum Lost

critical · Health

Flags replicated consumers with enough replicas offline to lose quorum. The replica count comes from the reported peer set, so consumers that inherit their replica count from the stream (num_replicas 0) are included. When the consumer leader is offline no replica rows are reported, so that quorum loss is not flagged.

Threshold. Online replicas < quorum needed (R > 1)

Remediation. Restore offline replicas by bringing their servers back online. Without quorum the consumer cannot make progress.

consumer-004: Consumer Delivered Below Stream First Sequence

info · Consistency

Reports consumers with delivery history whose reported stream position is more than one sequence below the stream's current first sequence.

Threshold. delivered_consumer_seq > 0 and 0 < delivered_stream_seq < stream first_seq - 1

Remediation. Inspect the stream's origin, filters, pending messages and application activity. Retention, expiry, message deletion and some filtered purges can open a gap, while a full purge normally moves the consumer to the adjacent boundary. Consumer filters and holes in a filtered mirror can also skip positions the consumer never needed to receive, even when the consumer has no filter itself. The size of the gap doesn't count missed messages or show that the consumer is stuck.

consumer-005: Consumer Sequence Ahead of Stream Sequence

info · Consistency

Flags consumers with delivery history whose reported stream position is ahead of the parent stream's observed last sequence.

Threshold. delivered_consumer_seq > 0 and delivered_stream_seq > stream last_seq > 0

Remediation. Compare fresh stream and consumer state before considering recovery. The two are sampled separately, even within one server's JSZ response, so normal publishing can move the consumer past an older stream snapshot, and this can recur across scrapes. The finding doesn't indicate lost data or corruption.

consumer-006: Outstanding Ack Critical

critical · Health

Flags consumers with io.nats.monitor.enabled set to true where num_ack_pending on the leader is at or above the operator-defined io.nats.monitor.outstanding-ack-critical threshold. Invalid monitor metadata skips the consumer (see consumer-020).

Threshold. num_ack_pending >= operator-defined io.nats.monitor.outstanding-ack-critical metadata threshold (monitoring enabled, threshold of 0 or less disables)

Remediation. Increase consumer throughput, add consumer instances, or raise the threshold in the consumer metadata.

consumer-007: Waiting Critical

critical · Health

Flags consumers with io.nats.monitor.enabled set to true where num_waiting on the leader is at or above the operator-defined io.nats.monitor.waiting-critical threshold. Invalid monitor metadata skips the consumer (see consumer-020).

Threshold. num_waiting >= operator-defined io.nats.monitor.waiting-critical metadata threshold (monitoring enabled, threshold of 0 or less disables)

Remediation. Inspect outstanding fetches and message demand, and avoid redundant or overly long-lived pull requests. If this many waiting requests is intentional, raise the threshold.

consumer-008: Unprocessed Critical

critical · Health

Flags consumers with io.nats.monitor.enabled set to true where num_pending on the leader is at or above the operator-defined io.nats.monitor.unprocessed-critical threshold. Invalid monitor metadata skips the consumer (see consumer-020).

Threshold. num_pending >= operator-defined io.nats.monitor.unprocessed-critical metadata threshold (monitoring enabled, threshold of 0 or less disables)

Remediation. Add consumer processing capacity or look for a stalled consumer.

consumer-009: Last Delivery Critical

critical · Health

Flags consumers with io.nats.monitor.enabled set to true whose last delivery, as reported by the leader, is at least as old as the io.nats.monitor.last-delivery-critical threshold, or that report no delivery timestamp. The threshold is a duration such as 30s or 5m. Zero or less disables the check. Invalid monitor metadata skips the consumer (see consumer-020).

Threshold. no recorded timestamp or time since last delivery >= operator-defined io.nats.monitor.last-delivery-critical metadata threshold (monitoring enabled, threshold of 0 or less disables)

Remediation. Check whether processing is stalled or the stream is idle. A missing delivery timestamp can also follow a server restart or leader change, so check consumer sequences and application activity before concluding nothing was delivered.

consumer-010: Last Ack Critical

critical · Health

Flags consumers with io.nats.monitor.enabled set to true whose last acknowledgement, as reported by the leader, is at least as old as the io.nats.monitor.last-ack-critical threshold, or that report no acknowledgement timestamp. The threshold is a duration such as 30s or 5m. Zero or less disables the check. Invalid monitor metadata skips the consumer (see consumer-020).

Threshold. no recorded timestamp or time since last ack >= operator-defined io.nats.monitor.last-ack-critical metadata threshold (monitoring enabled, threshold of 0 or less disables)

Remediation. Check whether processing is stalled or the stream is idle. A missing acknowledgement timestamp can also follow a server restart or leader change, so check consumer sequences and application activity before concluding nothing was acknowledged.

consumer-011: Redelivery Critical

critical · Errors

Flags consumers with io.nats.monitor.enabled set to true where num_redelivered on the leader is at or above the operator-defined io.nats.monitor.redelivery-critical threshold. Invalid monitor metadata skips the consumer (see consumer-020).

Threshold. num_redelivered >= operator-defined io.nats.monitor.redelivery-critical metadata threshold (monitoring enabled, threshold of 0 or less disables)

Remediation. Investigate processing failures and acknowledgement timeouts, and tune retry timing. The count can include messages that exhausted max_deliver, so arrange recovery for those.

consumer-012: Pinned Consumer Policy Mismatch

warning · Consistency

Flags consumers with io.nats.monitor.enabled and io.nats.monitor.pinned set to true that are not using the pinned_client priority policy, so the declared client pinning is not in effect. Invalid monitor metadata skips the consumer (see consumer-020).

Threshold. io.nats.monitor.enabled and io.nats.monitor.pinned metadata true but priority_policy != pinned_client (NATS Server 2.11.0+)

Remediation. Recreate the consumer with priority_policy=pinned_client (e.g. nats consumer add --pinned-groups), or edit its configuration with nats consumer edit --config <file>.

consumer-020: Invalid Consumer Monitor Metadata

warning · Consistency

Flags consumers with invalid values for recognized io.nats.monitor metadata keys. Any invalid key stops every monitor threshold check on that consumer, so this reports a configuration error rather than a threshold breach.

Threshold. Invalid recognized monitor metadata

Remediation. Correct or remove the listed metadata values so each boolean, integer or duration value matches its key's type. Values are validated even when monitoring is disabled.

User

user-001: Bearer Token User

info · Errors

Flags bearer token users with active connections.

Threshold. bearer = true with active connections

Remediation. Migrate the user to NKey-based authentication where possible. Bearer tokens skip nonce signature verification during CONNECT, relying on JWT validity alone. They are appropriate for WebSocket and HTTP contexts where NKey signing is impractical, but should not be used for long-lived server-to-server connections.

user-002: Excessive User Connections

info · Errors

Flags users with more active connections than the configured maximum.

Threshold. connections > max_connections (default: 100)

Remediation. Investigate why a single user has so many connections. Consider using connection pooling or separate user credentials per service instance.

Change

server-023: Config Reload Detected

info · Change

Detects servers whose configuration was reloaded by comparing config_load_time with the server's own previous value. A restart is not a reload and is covered by server-008.

Threshold. config_load_time differs from the server's previous recorded value

Remediation. Verify the configuration change was intentional. Review server logs to confirm the reload was successful and no errors occurred.

server-024: JetStream Domain Changed

warning · Change

Flags a server whose JetStream domain differs from the value it last reported. Changing the domain requires a restart, which gives the server a new pk, so the comparison is matched by server name.

Threshold. js_domain differs from the value last reported under the same server name

Remediation. Confirm the domain change was intended and the new domain is correct. A different JetStream domain changes how streams and consumers are addressed across clusters, so check that dependent clients and mirrors were updated.

stream-001: Stream Configuration Changed

info · Change

Flags streams whose num_replicas, retention_policy, max_msgs, max_bytes, max_age or max_consumers changed from the stream's previous reported configuration. Changes to other settings are not flagged.

Threshold. num_replicas, retention_policy, max_msgs, max_bytes, max_age or max_consumers differs from the stream's previous config

Remediation. Verify the stream configuration change was intentional and monitor for downstream effects on consumers.

Optimization Checks

Placement

account-011: Cross-Cluster Stream Access

info · Performance

Flags accounts with clients in a cluster that leads none of the account's streams, when the account does lead streams in another cluster. Both floors keep a few stray clients from turning into placement advice.

Threshold. >= min_conn_count (default: 5) clients and >= min_conn_percent% (default: 10) of the account's connections in a cluster with no local stream leaders

Remediation. First check whether these clients use the account's streams at all, since clients that only use core NATS need no local stream leader. A stream's replicas all live in one cluster, so where clients reach it across a gateway by accident, consider moving the stream, adding a local mirror, or moving the clients closer to it.

account-012: High Gateway Traffic Ratio

info · Performance

Reports the share of an account's bytes that crossed gateways in the selected window. Each server measures from its own previous scrape, which can fall before the window starts. A counter reset adds no traffic for that interval.

Threshold. Gateway traffic > gateway_percent% (default: 30) of account traffic within the selected time range, with >= min_traffic (default: 1 MiB) moved

Remediation. Compare the gateway share with where publishers, subscribers and streams are meant to run. An account spread across clusters can still keep its traffic local, so client placement does not predict the ratio. Look for clients reaching remote clusters by accident, and consider moving a stream or running a responder closer to its clients. If this share is expected for your deployment, disable account-012.

account-014: Gateway Interest Mode

info · Performance

Flags accounts still using optimistic interest mode on a gateway to a remote cluster. Reports from every local server are combined into one finding per account and remote cluster.

Threshold. interest_mode = 'Optimistic'

Remediation. In optimistic mode a gateway forwards messages before it knows the remote side has interest, then stops sending subjects it learns have none. NATS 2.9+ normally switches accounts to interest-only mode when gateways connect or accounts load, so check both ends of the gateway and whether this is a brief transition. In the legacy optimistic path, the switch happens once enough distinct no-interest subjects build up. The subscription count shown is not that threshold.

stream-033: Stream Leader Not Co-located

info · Performance

Flags streams whose leader is in a different cluster from most of the account's client connections. Clients in other clusters reach the leader through gateways. Consumer Raft groups follow their stream's placement, so the finding is reported on the stream, where placement can be changed.

Threshold. stream leader in a different cluster than the account's dominant client connection cluster, with at least min_conns (default: 5) total connections and the dominant cluster holding at least min_dominance_percent% (default: 60) of them

Remediation. Use preferred placement tags to co-locate the stream leader with the majority of its clients. To force a leader election, use nats stream cluster step-down, which may relocate the leader to a better-positioned replica.

Cost

stream-021: Over-Replicated Inactive Stream

info · Consistency

Flags R3+ streams that received no new messages across the selected time range. The stream must be observed at or before the range's start and at its end. Sealed streams are skipped.

Threshold. R3+ stream with no new messages across time range

Remediation. Extra replicas still use storage and Raft resources. Reduce the replica count to R1 for inactive streams, or delete the stream if it is no longer needed.

stream-022: Memory Storage Large Stream

info · Saturation

Flags memory-backed streams using more memory than the configured maximum.

Threshold. Memory-backed stream > max_memory (default: 100 MiB)

Remediation. Convert the stream to file-backed storage if low-latency access is not required. Memory-backed streams consume server RAM directly.

server-038: Wasted JetStream Memory Reservation

info · Consistency

Flags servers where JetStream memory usage is below the minimum utilization of reserved capacity.

Threshold. Memory usage < min_utilization_percent% (default: 20) of reserved

Remediation. Reduce the JetStream memory reservation to match actual usage, or migrate memory-backed streams to this server to improve utilization.

stream-023: Uncompressed Large Stream

info · Consistency

Flags file-backed streams larger than the uncompressed size threshold with no compression enabled.

Threshold. File stream > max_uncompressed (default: 1 GiB) with compression disabled

Remediation. Enable S2 compression on the stream configuration to reduce disk usage and I/O costs.

server-039: Wasted JetStream Storage Reservation

info · Consistency

Flags servers where JetStream storage usage is below the minimum utilization of reserved capacity.

Threshold. Storage usage < min_utilization_percent% (default: 20) of reserved

Remediation. Reduce the JetStream storage reservation to match actual usage, or migrate file-backed streams to this server to improve utilization.

Balance

server-031: Uneven Leader Distribution

info · Saturation

Flags servers hosting disproportionately many stream and consumer leaders. Only JetStream servers in a named cluster are counted, and a server that answered only STATSZ is left out of the comparison entirely.

Threshold. Leaders > 1.5x cluster average (min 3 servers)

Remediation. Use nats stream cluster step-down and nats consumer cluster step-down to redistribute leaders across the cluster. Target servers with the highest leader counts first.

server-032: Connection Hotspot

info · Saturation

Flags servers with more than double the cluster average connections.

Threshold. Connections > 2x cluster average (min 100 absolute)

Remediation. Review client connection configuration. Use DNS round-robin or load balancer to distribute connections more evenly across cluster servers.

server-033: Subscription Hotspot

info · Saturation

Flags servers with more than double the cluster average subscriptions.

Threshold. Subscriptions > 2x cluster average (min 100 absolute)

Remediation. Redistribute client connections to balance subscription load. Check for clients with excessive subscriptions.

server-034: Stream Replica Count Imbalance

info · Saturation

Flags servers hosting disproportionately many stream replicas, which concentrates storage I/O and memory on a few servers. Only JetStream servers in a named cluster are counted, and a server that answered only STATSZ is left out of the comparison entirely.

Threshold. Replicas > 1.5x cluster average (min 3 servers, min 10 replicas)

Remediation. Use placement tags to distribute new streams more evenly. Consider removing and re-adding replicas to rebalance.

server-035: JetStream Storage Skew

info · Saturation

Flags servers whose JetStream storage exceeds double the cluster average.

Threshold. Storage > 2x cluster average (min 1 GiB absolute)

Remediation. Migrate large streams to other cluster servers or add storage capacity to balance disk usage.

server-036: Account Connection Concentration

info · Saturation

Flags servers hosting more than 70% of an account's open client connections within their cluster. An account connected through fewer than 3 of the cluster's servers is not evaluated.

Threshold. > 70% of the account's client connections in a cluster on one server (min 3 servers, min 10 connections)

Remediation. Configure client connection URLs to include multiple servers. Use a load balancer to spread connections across the cluster.

stream-020: Stream-Consumer Leader Co-location

info · Saturation

Flags streams where the stream leader's server also hosts a disproportionate share of replicated consumer leaders. R1 consumers are excluded because their leader cannot be moved off its only replica's server. Streams with fewer than 3 replicated consumer leaders are skipped.

Threshold. Replicated consumer leaders on stream leader's server > consumer_leader_percent% (default: 50) (min 3 replicated consumer leaders)

Remediation. Run nats consumer cluster step-down on consumers led from the stream leader's server to spread their leaders across the cluster. This moves acknowledgement and delivery work, and the I/O and CPU load that comes with it, off a single node.

server-037: JetStream Storage Saturation with Skew

warning · Saturation

Flags servers near their configured max_store JetStream storage limit in a cluster whose storage is also unevenly spread across servers.

Threshold. storage > saturation_percent% (default: 90) of max_store AND cluster skew (max - min utilization) > skew_pp (default: 30) percentage points

Remediation. New writes fail on this server once it reaches max_store. Move streams from it to less-used peers, or raise max_store or add disk capacity on this server. Use placement tags to guide future stream placement.

Account

account-007: Account Storage Quota Approaching Limit

warning · Saturation

Flags accounts whose file-stream JetStream reservations (max_bytes) are near the disk quota in any limit set. Untiered quotas count max_bytes once per replica, and an Rn tier counts max_bytes once for each stream in that tier. A stream without a leader still holds its reservation. Actual storage use is covered by a separate check.

Threshold. File storage reservations > warn_percent% (default: 90) of the applicable JWT limit set; most saturated set per account

Remediation. Reduce unused max_bytes reservations or raise the account's disk quota for that limit set. Until then, new reservations can be rejected even while existing streams still have room to write.

account-008: Excessive Account Claims Size

info · Consistency

Flags accounts whose decoded claims JSON exceeds the configured byte threshold. The measured value is the stored JSON representation, not the signed JWT's wire size.

Threshold. claims_size > max_size (default: 64 KiB)

Remediation. Review large imports, exports, signing key lists, revocations and metadata in the account claims, and remove entries that are no longer needed. Large claims add parsing work each time the account JWT is fetched or changed. Servers cache resolved accounts, so the cost isn't paid on every connection.

Idle Resources

server-040: Underutilized Server

info · Health

Flags servers that remained nearly idle across the selected time range.

Threshold. Max per-core CPU < max_cpu_percent% (default: 5) AND max connections < max_connections (default: 10) across range (min 5 samples, server reporting at current epoch)

Remediation. Consider decommissioning the server or migrating workload to it from busier servers.

stream-024: Inactive Stream

info · Health

Flags unsealed streams that received no new messages across the selected time range. The stream must be observed at or before the range's start and at its end.

Threshold. last_seq unchanged across time range (excludes sealed)

Remediation. Delete the stream if it is no longer needed, or seal it to prevent accidental writes. If temporarily inactive, no action is needed.

consumer-013: Inactive Consumer

info · Consistency

Flags consumers whose last delivered stream sequence is unchanged between the earliest and latest observations in the time range. Requires at least two samples in the range, the latest at the current epoch.

Threshold. delivered_stream_seq unchanged between earliest and latest samples in range (min 2 samples, consumer reporting at current epoch)

Remediation. Inspect pending messages and application activity. Redelivery attempts can continue while the stream position remains unchanged. Remove the consumer only after confirming it is no longer needed.

consumer-014: Drained Consumer

info · Consistency

Flags consumers with nothing pending delivery or acknowledgement at the current epoch, on a stream whose last sequence didn't change across at least two samples in the window.

Threshold. num_pending = 0 AND num_ack_pending = 0 at current epoch, stream last_seq unchanged between earliest and latest samples in range (min 2 samples)

Remediation. Confirm the consumer is no longer needed before removing it. Both counters can be zero while messages that exhausted max_deliver stay unprocessed in the stream, so check application ownership, retained messages and delivery exhaustion advisories.

account-009: Inactive Account

info · Health

Flags non-system accounts that are currently reported but have had no client connections, leaf connections or byte counter increases for longer than the inactivity threshold. A byte counter that decreases also counts as activity, since the account may have been removed and re-registered. An account must have been observed for at least the threshold before it can be flagged.

Threshold. Zero connections and zero throughput for threshold (default: 24h), excludes system

Remediation. Confirm the account is no longer needed, then remove or disable it.

connection-004: Idle Client Connections

info · Consistency

Flags client connections idle for longer than the idle threshold that have never sent or received a message.

Threshold. Idle > idle (default: 5m) with zero lifetime messages

Remediation. Review the client's purpose, subscriptions and expected traffic pattern. A subscriber waiting for rare messages, an idle request/reply client or a health connection can be intentionally quiet. Close the connection only after establishing that it is unused or leaked.

System Improvement

stream-025: Streams Without Limits

info · Consistency

Flags unsealed streams with no message, byte, age or per-subject limit.

Threshold. max_msgs = -1 AND max_bytes = -1 AND max_age = 0 AND max_msgs_per_subject = -1 (excludes sealed)

Remediation. Configure at least one retention limit (max_msgs, max_bytes, max_age, or max_msgs_per_subject) to prevent unbounded disk growth.

consumer-016: High Consumer Redelivery

warning · Errors

Flags consumers whose current count of messages tracked for redelivery is high compared with delivery attempts in the window. Tracked messages can predate the window or have exhausted max_deliver, so the ratio is not the share of attempts that were redeliveries. Requires at least 100 delivery attempts in the window.

Threshold. current num_redelivered > redelivery_percent% (default: 10) of delivery attempts within the check window, with at least 100 attempts

Remediation. Investigate acknowledgement timeouts and processing failures, choose retry timing for the workload, and arrange recovery for messages that exhausted max_deliver.

consumer-017: Ack Pending Buildup

warning · Errors

Flags consumers approaching their maximum ack pending limit. At the limit, the server stops delivering until acks catch up. Consumers with unlimited max_ack_pending are skipped.

Threshold. num_ack_pending > warn_percent% (default: 90) of max_ack_pending

Remediation. Scale out consumer instances to process messages faster, increase max_ack_pending (default 1,000), or investigate why messages are not being acknowledged. Ack pending can also be limited at the stream level via consumer limits and at the account level.

consumer-018: Unbound Push Consumer

info · Errors

Flags push consumers with no subscriber currently bound.

Threshold. Push consumer with no bound subscriber

Remediation. If delivery is expected, start the subscribing application. The server holds push deliveries until a subscriber binds, so messages stay pending without using up delivery attempts. Check pending messages and application ownership before removing an unused consumer or replacing it with a pull consumer.

server-041: Route Pending Pressure

info · Performance

Flags route connections with more pending data than the pending threshold.

Threshold. pending_size > pending (default: 1 MiB)

Remediation. Investigate network bandwidth between cluster peers and load on the remote peer. Reduce message rates on high-volume intra-cluster subjects or upgrade network capacity between peers.

leafnode-004: Leaf Compression Disabled

info · Consistency

Flags leaf connections reporting compression off after negotiation. Compression reduces bandwidth between leaf and hub.

Threshold. compression = 'off'

Remediation. Enable S2 compression in the leafnode configuration (compression: s2_auto adapts compression to RTT). Check both sides, since the negotiated mode doesn't show which one turned compression off.

server-043: Raft Apply Lag

warning · Performance

Flags Raft groups where applied entries trail committed entries by more than the maximum allowed lag. Committed entries take effect only once applied. Members that are catching up are excluded, and server-048 reports sustained catch-up.

Threshold. committed - applied > max_lag (default: 100) AND catching_up = false

Remediation. Check disk I/O and CPU on the affected server, which is falling behind in applying committed Raft entries.

account-013: Unlimited JetStream Account

info · Consistency

Flags non-system JetStream accounts whose JWT leaves memory or disk storage unlimited in any limit set, for a storage class at least one server offers. Each tier of a tiered JWT is checked separately, and a class limited to zero counts as disabled. Accounts without a JWT are not checked.

Threshold. any limit set with js_mem_storage = -1 OR js_disk_storage = -1, for a class a server offers (excludes system)

Remediation. Set explicit JetStream memory and disk storage limits in the account JWT, in every tier for a tiered JWT, so one account cannot reserve capacity until the cluster is exhausted.

server-044: Leaderless Raft Group

critical · Health

Flags a Raft group that has had a leader before and now reports none. A leaderless group cannot accept writes. This happens during an election or a sustained loss of quorum.

Threshold. leader = '' AND ever_had_leader = true

Remediation. Check whether later scrapes show a completed election. If there is still no leader, confirm that a majority of the group's peers are online and reachable, and check server logs for storage or network failures.

server-045: Raft IPQ Backpressure

warning · Performance

Flags a Raft group whose internal queue lengths exceed the threshold, a sign of a processing backlog.

Threshold. GREATEST(ipq_prop_len, ipq_entry_len, ipq_resp_len, ipq_apply_len) > max_ipq (default: 1000)

Remediation. Raft's internal queues (proposals, append entries, apply, responses) are backing up. The apply queue matters most, since a long one means JetStream cannot consume committed entries fast enough. Check server CPU, disk I/O and network latency.

server-046: Subscription Fanout Anomaly

info · Consistency

Flags servers where max fanout is disproportionately higher than average fanout.

Threshold. max_fanout > multiplier (default: 10) x avg_fanout AND avg_fanout > 1

Remediation. Investigate subjects with high subscriber counts. A large max-to-average fanout ratio indicates one or more subjects with excessive subscribers, which can create hot spots.

server-047: Subscription Churn

info · Errors

Flags servers with excessive subscription insert and remove operations since their own previous observation.

Threshold. churn delta > max_churn (default: 10,000) between adjacent server observations

Remediation. If one client is responsible, it is likely subscribing and unsubscribing in a loop. Find it by connection name or IP and fix the client code. If many clients are responsible, it is likely a reconnection storm, with clients re-subscribing all at once. Look for a network event or server restart just before it.

server-048: Raft Sustained Catching Up

info/warning · Health

Flags a Raft group member catching up to its leader. Catch-up usually finishes between scrapes, so a single sighting is info. A member still catching up on every scrape for longer than the sustain window is a warning, since it cannot serve as a failover target and may not keep pace with the group.

Threshold. catching_up = true; escalates to warning once held > sustain (default: 30m)

Remediation. Check disk I/O, network bandwidth and CPU on the catching-up server. A short catch-up is usually a burst being replayed. A sustained one means the server needs more resources or a re-sync.

server-042: Gateway Pending Pressure

info · Performance

Flags gateway connections with more pending data than the pending threshold.

Threshold. pending_size > pending (default: 1 MiB)

Remediation. Investigate network bandwidth between clusters and load on the remote cluster. Reduce inter-cluster message rates by improving stream/consumer placement, or upgrade inter-cluster network capacity.

consumer-019: Consumer ACK Floor Divergence

info/warning · Errors

Flags consumers whose consumer-sequence gap between delivered position and ACK floor is a large multiple of max_ack_pending, or exceeds an absolute threshold. The gap includes redeliveries and messages already acknowledged above the floor, so it is not the outstanding message count.

Threshold. gap > elevated_multiplierx (default: 2) max_ack_pending or elevated_abs (default: 100,000); escalates to warning above severe_multiplierx (default: 5) or severe_abs (default: 1,000,000)

Remediation. Compare num_ack_pending and num_redelivered, then investigate slow messages, out-of-order processing and retry patterns. Interest and workqueue streams can remove acknowledged messages above the floor, so a large gap doesn't mean storage is retained. Acknowledge each message only after its processing is complete.

stream-026: Direct Gets Disabled

info · Performance

Flags replicated streams with allow_direct disabled, so individual message-get requests must be served by the stream leader. R1 streams are skipped, since their single server serves every read either way.

Threshold. allow_direct = false AND num_replicas > 1

Remediation. Enable allow_direct so clients that use the direct-get API can read from replicas and spread read load. Before enabling it, check whether replica freshness meets the clients' read requirements.

stream-027: High Interior Deletes on Stream

info · Saturation

Flags streams with many interior sequence holes or a high share of deleted positions. Deleted ranges are stored compactly, so the count does not measure memory use.

Threshold. Interior deletes > max_deleted (default: 100,000,000) or holes exceed 90% of retained sequence positions

Remediation. Check measured memory use and recovery time, and review retention and update patterns. If expiring retained messages is acceptable, setting max_age can reduce old sequence holes.

stream-028: Large Deduplication Window

info/warning · Saturation

Flags streams with a long deduplication window and active publishing, estimated from sequence advancement. The window can use a lot of memory when publishers set message IDs. Needs a previous observation in the selected range.

Threshold. window > elevated_window (default: 1h) or severe_window (default: 6h) AND last-sequence advancement/sec >= min_rate_per_sec (default: 10)

Remediation. The server keeps a deduplication entry in memory for each distinct Nats-Msg-Id published within the window. The publish rate is estimated from sequence advancement, which can't tell whether publishers set IDs and can overstate the rate after sequence jumps. Confirm publishers set Nats-Msg-Id, then size the window to their retry interval.

stream-029: KV Buckets Without max_age

info · Saturation

Flags KV buckets with no max_age configured that have accumulated a large number of interior deletes.

Threshold. max_age = 0 AND deleted >= min_deleted (default: 100,000)

Remediation. Key overwrites and history limits create these holes without any key being deleted, so the count is not the number of KV delete markers. Check memory use and recovery time before changing retention. Setting max_age also expires live key values and history, so use it only if that TTL suits the application.

stream-030: R1 Streams in Multi-Node Clusters

info · Health

Flags R1 (single-replica) streams, which have no redundancy, in clusters with more than one JetStream node. Mirrors are excluded.

Threshold. num_replicas = 1 in multi-node cluster

Remediation. If the hosting node goes down, the stream is offline until that node recovers. Consider R3, which needs three eligible nodes with enough capacity (placement tags and resource limits can rule nodes out). R1 is fine for ephemeral, cacheable or easily reproducible data.

server-050: Subscription Count Growth

info · Errors

Flags servers whose subscription count grows steadily while the connection count ends within 5% of where it started, a sign of a subscription leak. Most reliable on a single server, because on clustered or leaf node servers the count also includes remote interest.

Threshold. Growth over min_epochs (default: 10) observations with >= min_growth_percent% (default: 20) increase, at least min_mono_percent% (default: 80) of adjacent intervals non-decreasing, and connections changed < 5%

Remediation. Find the client creating subscriptions without unsubscribing by comparing per-connection subscription counts, then fix it to unsubscribe when done. On clustered or leaf node servers the subscription count includes interest propagated over routes and leaf connections, while connections count only local clients, so the growth may be cluster-wide interest instead of a local leak. Confirm with per-connection counts before calling it a leak.

server-051: Raft WAL Size Excessive

warning/critical · Saturation

Flags Raft groups with an excessively large write-ahead log, risking disk exhaustion and cascading OOM failures. Triggers on absolute size or relative to js_max_store. The relative thresholds apply only when js_max_store is set.

Threshold. WAL > warn_size (default: 10 GiB) or warn_percent% (default: 50) of js_max_store; critical above crit_size (default: 50 GiB) or crit_percent% (default: 80)

Remediation. Check Raft catch-up, snapshot creation and log compaction on the affected server, along with free disk space and I/O latency. Raft commits depend on replication quorum, so unacknowledged consumer messages do not hold the log back. Contact support if the WAL keeps growing or snapshots fail.

stream-031: WorkQueue Discard New with Aggressive Consumer Settings

warning · Consistency

Flags WorkQueue streams with discard_policy=new that have a consumer using timed acknowledgements with a short ack_wait or low max_deliver. Fast retries can exhaust delivery attempts and leave work retained until the stream fills.

Threshold. discard_policy=new AND ack_policy != flow_control AND (ack_wait < min_ack_wait (default: 30s) OR 1 <= max_deliver < min_max_deliver (default: 10))

Remediation. Set ack_wait longer than processing time, allow enough deliveries for retries, and plan how to recover messages that exhaust max_deliver. Those messages stay in the stream until acknowledged or removed, and once a limit is reached the stream rejects new publishes, so monitor rejected publishes and make sure publishers handle the error. Switching to discard-old can evict unprocessed work and does not fix retries.

stream-032: Sustained Consumer Growth on Stream

warning · Errors

Flags streams where consumer count has been growing steadily, which can indicate a consumer leak.

Threshold. At least min_growth_epochs (default: 5) observations, net growth >= min_total_growth (default: 500), and at least min_mono_percent% (default: 80) of adjacent intervals non-decreasing

Remediation. Find what creates the consumers. Set inactive_threshold on ephemeral consumers so unused ones expire (it may be set too high), or use durable consumers and delete them when they're no longer needed.

server-052: Raft Group Peer Count Mismatch

warning · Consistency

Flags Raft groups where the observed peer count exceeds the expected replica count from stream or consumer configuration. A consumer with num_replicas 0 inherits its stream's count. Each group is reported once, against its leader when the leader reports it.

Threshold. Observed peers > configured num_replicas

Remediation. This usually follows a peer-remove then peer-add where the old peer was not fully removed, or a replica count decrease that did not fully propagate. Remove the extra peer with nats stream cluster peer-remove, or update num_replicas to the desired count.