Skip to main content

Checks Reference

This document is the canonical reference for all automated checks. Each check runs against collected monitoring data to surface operational issues and optimization opportunities.

Conventions​

  • Codes are lowercase and hyphenated, such as server-001. That spelling is what you write under check-thresholds and disabled-checks, what a notification subscription matches, and what the API and stored findings carry
  • Severity levels: critical (immediate action required), warning (investigation needed), info (optimization opportunity). A check shown as warning/critical or info/warning sets its severity per finding
  • Operational checks detect current health and correctness issues at a single epoch. Results are computed during indexing and exposed through the checks API and web UI
  • Optimization checks analyze trends across a time range to surface waste, imbalance, and improvement opportunities. During indexing each one is evaluated over its lookback window (default: 6h, set per check under check-thresholds); querying its findings for a selected time range evaluates it over that range instead
  • Framework categories group checks by operational concern. Each check belongs to exactly one category. See Checks for the conceptual overview
  • Configurable thresholds: Checks whose Threshold column contains backtick-quoted parameter names (e.g. `cpu_percent`) can be tuned via the check-thresholds config section. See Configuration → Check thresholds for the override syntax and config.example.yaml for an annotated example
  • Disabling a check: Any check can be turned off by listing its code under disabled-checks. A disabled check keeps running and stays visible, marked disabled; it is left out of the health grades and never delivered as a notification. See Configuration → disabled-checks

Monitor thresholds follow the pinned jsm.go evaluator's directions, boundaries, and disabling rules. Malformed recognized metadata prevents that monitor option set from being evaluated. Mirror seen-critical controls mirror activity and peer-seen-critical controls peer activity; Insights preserves those documented meanings despite the jsm.go v0.3.0 parser assigning the mirror key to the peer option.

server-049 is retired: negotiated leafnode compression cannot establish the configured compression mode. The code is reserved and excluded from active checks. Stored findings remain available with a retirement explanation; original measurements are unavailable. Exact saved policy and webhook entries for this code are accepted with a warning and do not enable new evaluations.

Minimum Server Versions​

A few checks read monitoring fields that older NATS Server releases do not report. Those servers are skipped rather than evaluated: an absent field arrives as a zero that is indistinguishable from a real reading, so any finding derived from it would be wrong — either silently missing or falsely raised.

CheckRequiresField it reads
server-0122.12.0stale_connections
server-0132.12.0stalled_clients
consumer-0122.11.0priority_policy

The gate applies per server, so during a rolling upgrade these checks keep evaluating the nodes that have already moved. Servers below the minimum are excluded from the check's entity total rather than counted as passing.

A check with no eligible servers is shown on the system page dimmed, with a Requires NATS x.y.z+ label in place of its counts, rather than as a clean result. Upgrading past that release enables it with no configuration change. This is a different state from a check disabled in configuration: a disabled check still runs and still shows its counts, while an unsupported one has nothing to show. The minimum is also reported as min_server_version by the checks list and checks info API endpoints.

Health Indicator​

Each entity in an entity table displays a health indicator derived from the findings recorded for it at the current epoch. Every recorded finding counts: from operational and optimization checks alike, and from checks listed under disabled-checks, which are left out of the health grades but not out of this indicator.

The indicator has three states. The icon carries the same meaning as the color, so the state reads without color.

ConditionColorIcon
No findingsGreenCheck circle
One or more findings, none criticalYellowTriangle
Any critical findingRedOctagon

Operational Checks​

Server​

CodeCheck NameSeverityCategoryThresholdDescriptionRemediation
server-001Connection Readiness FailurecriticalHealthhealthz status != "ok" with CONNECTION error type, or with no errors reportedFlags servers reporting connection readiness failures via the healthz endpoint, or reporting unhealthy with no error detail.Check for listener port conflicts, TLS certificate errors and permission issues. Find the root cause before restarting.
server-002Server Version MismatchinfoConsistencyVersion differs from majorityIdentifies servers running a different software version than the cluster majority. Unclustered servers are compared as one group. On an even split, the lexically higher version string counts as the majority.Complete the rolling upgrade so all servers run the same version. If intentional, verify compatibility between versions.
server-003High CPU UsagewarningPerformancecpu/cores > cpu_percent% (default: 90)Flags servers where per-core CPU usage exceeds the CPU threshold.Identify the workload driving CPU usage using the /pprof/profile debug endpoint. Common causes: high-fanout subjects, subscription matching overhead, JetStream write pressure. Scale out with additional servers, optimize subject hierarchies, or increase server capacity.
server-004Slow ConsumerswarningPerformanceslow_consumers delta > 0Flags servers with new slow consumer events since the previous epoch.Find the affected clients and speed up their message processing. A slow consumer is evicted when its pending outbound bytes exceed max_pending or a write to it times out. To buffer more, increase max_pending in the server config (default 64 MiB). Otherwise add more clients, reduce message rates, or have clients subscribe to fewer subjects.
server-005JetStream Memory Pressurewarning/criticalSaturationjs_memory > warn_percent% (default: 90) of js_max_memory; critical above crit_percent% (default: 95)Flags servers whose JetStream memory usage is approaching the configured max_memory limit, where writes to memory-backed streams are rejected. Severity rises to critical near the limit.Reduce memory-backed stream usage, raise max_memory in the server config, or convert large memory-backed streams to file storage. Usage is measured against max_memory, the hard limit, which is a different number from js_reserved_memory (the sum of memory-backed stream MaxBytes reservations).
server-007Authentication Not RequiredinfoHealthauth_required = falseFlags servers that do not require client authentication. It misses a server with no_auth_user configured, which accepts clients without credentials as a default user while still reporting auth_required=true. That setting is not scraped, so this check cannot see it.Enable authentication in the server configuration. Options: NKey authentication, JWT-based operator mode with signed credentials, user/password, TLS certificate mapping, or auth callout for centralized policy enforcement. Avoid no_auth_user in production.
server-008Unexpected Server Restartwarning/criticalChangestart_time changed; version and config_digest unchanged (an unreported digest is unknown, so version alone decides); critical on every epoch while unplanned boot generations in the trailing hour exceed max_restarts (default: 2)Flags servers that restarted without a version upgrade or configuration change, and escalates when restarts repeat. Start times are compared across epochs by cluster and server name, since a restart mints a new server ID. A boot whose version or configuration digest differs from what the previous run last reported is treated as planned, as is a rollback. A server that reports no configuration digest is judged on version alone. The digest covers the config after includes and environment variables are resolved, so a value that changes every boot (such as a pod IP in client_advertise) makes every restart look planned and silences this check for that server.Check the NATS server logs and system logs (journalctl, dmesg) for the exit reason. Common causes are an OOM kill, hardware failure, a process crash or an external signal. If restarts repeat, also check file descriptor limits (ulimit), JetStream store corruption on startup, port conflicts with another process, TLS certificate expiry and account resolver timeouts. Verify disk space and ulimit settings before restarting.
server-010High Route RTTinfo/warningPerformanceRTT > multiplier (default: 2) x the other servers' route average and >= min_rtt (default: 10ms); warning above severe_multiplier (default: 3)Flags a route whose round-trip time to a cluster peer is well above what the rest of the cluster sees. Routes are compared within their cluster, so a cluster whose routes are all slow is not reported. The average excludes the reporting server's own routes, so a server with uniformly slow routes does not raise its own baseline.Compare this route's network path and server load with the cluster's other routes. Different availability zones, routing paths, congestion, packet inspection or an overloaded server can explain the difference. Route latency slows cross-server request/reply and JetStream replication.
server-011Connection Count Highwarning/criticalSaturationconnections > warn_percent% (default: 80) of max_connections; escalates to critical above crit_percent% (default: 99)Flags servers where active connections approach the configured maximum. Critical means the server has almost no connection slots left, and at its cap it refuses new clients. max_connections defaults to 65536, so the check rarely fires unless a lower limit is set.Increase max_connections in the server config or per-account limits. Distribute clients across more servers using DNS round-robin or a load balancer. Identify and close unnecessary connections.
server-012Stale ConnectionsinfoErrorsstale_connections delta > 0 (NATS Server 2.12.0+)Flags servers with new stale connection events since the previous epoch.The server sent PING and the client did not answer with PONG after the configured number of unanswered pings (defaults: ping_interval 2 minutes, ping_max 2). Common causes are a network partition or firewall dropping idle connections, a client process that crashed without closing its TCP connection, or a blocked client event loop that cannot process PINGs. Check firewall idle timeout settings and client health.
server-013Stalled ClientswarningPerformancestalled_clients delta > 0 (NATS Server 2.12.0+)Flags servers with new stalled client events since the previous epoch.The server is throttling a fast producer because a downstream client cannot keep up. Improve client message processing speed, add more client instances, or increase max_pending to buffer more. This is normal flow control and an early warning before slow consumer eviction.
server-014JetStream Subsystem Not Readywarning/criticalHealthhealthz JETSTREAM error in the not_ready family; a recovering meta layer escalates to critical once held > sustain (default: 15m), JetStream disabled is critical at onceFlags a server that cannot serve JetStream because JetStream is disabled in its configuration or its meta layer is still recovering. Disabled is critical from the first scrape. Recovering is a warning, and turns critical once it has been reported on every scrape for longer than the sustain window. Other JetStream health problems are reported by checks that name the Raft group.A booting server's meta layer recovers and clears on its own. If it is still recovering past the sustain window, it is stuck. Check its Raft state and whether the meta group's other peers can reach it. If JetStream is disabled here but enabled on its peers, compare this server's JetStream block with theirs.
server-015JetStream Asset Configuration RejectedcriticalConsistencyhealthz STREAM or CONSUMER error in the config familyFlags a stream or consumer whose configuration the cluster refused. The asset does not exist on this server and will not appear until the configuration changes, so unlike most health findings this one does not clear on its own.The cause gives the server's own objection, such as an ack wait that must be positive, subjects that overlap the JetStream API, or a limit the account cannot satisfy. nats stream info and nats consumer info report the asset as missing because it never came up. Fix the configuration and recreate the asset. Restarting the server does not help.
server-016Account Resolution FailurewarningConsistencyhealthz ACCOUNT error typeFlags servers that could not resolve a JetStream account. The finding names the account, which does not appear in the account inventory because it was never resolved.Read the cause. A validation failure means the resolver returned a JWT the server rejected, so check the account JWT's signature and issuer against the operator it claims. An expired account needs its JWT reissued. A lookup failure points at the resolver. Check connectivity to it and whether it holds this account.
server-018High Gateway RTTwarningPerformanceRTT > multiplier (default: 2) x the average RTT of other servers' links from the same cluster to the same remote cluster, and >= min_rtt (default: 25ms)Flags gateway connections whose round-trip time is far worse than other servers' links from the same cluster to the same remote cluster. A cluster pair whose links are all slow is not reported. A link is only compared when another server in the same cluster has a link to the same remote cluster, and servers without a cluster name are skipped.Compare this server's path to the remote cluster with its peers' paths. Check availability zones, NAT hops, network interfaces, packet inspection and load on both ends, since servers in the same cluster can differ in distance and routing.
server-019JetStream Storage vs Configured Limitwarning/criticalSaturationstorage > warn_percent% (default: 90) of max_store; critical above crit_percent% (default: 95)Flags servers where JetStream storage usage approaches the configured max_store limit, escalating to critical near the configured limit.New message writes can be rejected once usage reaches max_store. This compares stream storage with that configured ceiling, which is a different number from js_reserved_storage (the sum of stream reservations) and from free filesystem space. Increase max_store in the server config, add disk capacity, or reduce stream storage by purging data or adding retention limits.
server-053Meta Raft Group ClosedcriticalHealthmeta Raft group closed while the server still reportsFlags servers whose meta Raft group has shut down while the server is still reporting. Without it the server cannot take part in JetStream control-plane decisions, such as stream and consumer creation, deletion, placement and leader election. Only the meta group is checked, since stream and consumer groups close normally when their asset is deleted.Check the server logs for the shutdown reason. A meta group closes on a write error, a failed store recovery or an explicit shutdown. Confirm JetStream storage is readable and has free space, then restart the server to rejoin the meta group.
server-054JetStream Asset MissingcriticalConsistencyhealthz STREAM or CONSUMER error in the missing familyFlags a stream or consumer that was assigned to this server and that the server cannot find. Unlike a rejected configuration, the asset is supposed to be here. A placement change, a failed catch-up or a delete that did not propagate can cause this.Check whether the asset exists elsewhere in the cluster with nats stream info or nats consumer info. If it does, this server missed it: check the Raft group's peer list and whether a catch-up is stuck. If it exists nowhere, a delete propagated partially and the assignment on this server is a leftover.
server-055JetStream Store Write ErrorcriticalHealthhealthz error in the store familyFlags a stream whose message store refused a write or could not be recovered, so it cannot persist messages on this server. After a refused write, a replicated stream steps down and stays an observer here so its replicas elsewhere can take over. server-064 reports a failed write to a Raft group's own write-ahead log, naming the group.Check free space, inode exhaustion, permissions and filesystem health on the JetStream store directory. The cause includes the operating system's error. A replicated stream has already stepped down here, so confirm a replica on another server leads it before repairing this one.
server-056JetStream Asset Raft FaultcriticalHealthhealthz asset or JetStream error in the raft familyFlags a stream or consumer whose Raft group is in a state this server cannot serve from, such as a node skew, a stopped monitor or a group reporting itself unhealthy. It also catches any cause the other health checks do not recognize, so a condition introduced by a newer server still gets reported.These are server-side faults, not problems with the asset's configuration. Check the group's peers and leader with nats server report jetstream, and whether this server is partitioned from them. A restart usually clears a skew or a stopped monitor, but find out why it happened first. Report recurring skew upstream as a bug.
server-064Raft WAL Write ErrorcriticalHealthwal_error is set on a Raft groupFlags a Raft group whose write-ahead log failed a write it could not tolerate. nats-server then shuts down the group's node, so every asset in the group is unavailable on this server until the fault is fixed.The write failed on the JetStream store directory, so check free space, inode exhaustion, permissions and filesystem health on the server. An out-of-space or permission error also disables JetStream server-wide, so other groups may be affected too. The node does not recover on its own. Restart the server once storage is fixed.
server-065Server Not ReportingwarningHealthno VARZ observation while a peer reports a route to the server and another server reports VARZ at the same epochFlags a server with no VARZ observation at an epoch when another server reported VARZ and a peer reported a route to it. Checks that use STATSZ values (CPU, traffic, storage and other server measurements) can still fire for it. Only servers with cluster routes are covered, so a solo server, a single-member cluster or a node reachable only over gateways is never reported. A server the scraper's server filter excluded after it was scraped can also match while peers still route to it. It fires only on epochs ROUTEZ also ran in.Check the scraper's server filter, system account permissions and request errors, then request VARZ directly over the scraper's connection. Compare which other endpoints and STATSZ values still respond. If monitoring requests hang while STATSZ responds, take a goroutine profile with PROFILEZ and look for blocked monitoring handlers or stream operations, since a slow shared handler can delay VARZ. This finding does not identify the cause.

Cluster​

CodeCheck NameSeverityCategoryThresholdDescriptionRemediation
server-020Connection Count ChangewarningErrorsabs(connection delta) > churn_percent% (default: 50) of the previous count, on a baseline of at least min_conns (default: 100)Flags servers whose connected-client count rose or fell sharply between epochs. A restarted server starts a new baseline, so the connections it drops on restart are not flagged on that server.First rule out a planned drain or rolling restart. A server in lame duck mode sheds its clients on purpose, and this check cannot tell that apart from churn because lame duck state (the INFO frame and the $SYS.SERVER.*.LAMEDUCK event) is not scraped. Otherwise check, in order: misconfigured client reconnect backoff (the default is randomized jitter, not zero), authentication failures that disconnect clients right after connecting, slow consumer evictions causing reconnect loops, load balancer health checks opening short-lived connections, and network instability. Review client connection URLs and reconnect settings.
server-021Gateway DisconnectionwarningHealth(server, remote cluster) link present at prev epoch but missing at currentFlags servers that lost their connection to a remote cluster since the previous epoch. Links are tracked by server and remote cluster name, so a reconnect that gets a new gateway ID does not count as a loss. A server whose monitoring endpoints did not answer in the current epoch is not flagged.Gateways reconnect automatically with randomized jitter, so a brief network issue clears on its own. If the disconnection persists, check TLS certificate validity (including OCSP stapling if enabled, where stale OCSP responses are a common cause), confirm firewall rules between clusters allow the gateway port, and confirm gateway names are consistent across all clusters.
server-022Gateway Config MismatchwarningConsistencyGateway set differs from the set held by a strict majority of the cluster's servers with gateways (at least 3)Flags servers whose set of gateway connections differs from the cluster majority. The majority set must be held by more than half of the cluster's servers with gateways, and at least 3 must have gateways. A server with no gateway connections is not compared.Give every server in the cluster the same gateway configuration, since asymmetric gateways cause routing failures. With reject_unknown_cluster enabled, unlisted gateways are rejected. With it disabled (the default), gateways can be discovered through gossip. Check that every intended gateway is listed and that TLS configuration matches on both sides.
server-057Memory Usage OutlierwarningSaturationmemory > multiplierx (default: 1.5) cluster averageFlags servers whose memory usage exceeds the cluster average by more than the configured multiplier.Investigate what is consuming memory on the outlier server. Common causes include large Raft state, many subscriptions, or memory-backed streams. Consider rebalancing workload.
server-059High HA AssetswarningSaturationha_assets > warn_count (default: 1000)Flags servers with more highly-available JetStream assets than the warning count. Each replicated asset carries Raft memory, disk and network overhead, so a restart or failure of the server is costly and its load is hard to rebalance.Distribute streams and consumers across more servers. Consider reducing replica counts on low-priority streams or consolidating small streams.
server-060Cluster Name WhitespacewarningConsistencyWhitespace in cluster nameFlags servers whose configured cluster name contains whitespace. nats-server rejects a literal space at startup but does not check for tabs, newlines or carriage returns, which leave an error-prone name.Remove whitespace from the cluster name in the server configuration and restart the affected servers.
server-061Route Count LowwarningHealthRoutes < sum of negotiated pool plus inferred pinned routes per peer; one route when either peer disables poolingFlags a server with fewer routes than its cluster peers require. For each peer where both sides pool routes, NATS opens the larger pool size plus a dedicated route for each account pinned on both ends, with a floor of one for the system account. A pair with pooling disabled needs one route. Pinned accounts are counted from observed routes, so the expected count is a lower bound.Check cluster connectivity, route URLs, pool sizes and each peer's pinned-account configuration. A dedicated account observed on both ends should have a route between them. Pinned-account configuration is not scraped, so an account whose routes have all disappeared is not counted, and a clear result does not prove every dedicated route is present.

Account​

CodeCheck NameSeverityCategoryThresholdDescriptionRemediation
account-001Account Connection LimitwarningSaturationconnections summed across servers > warn_percent% (default: 90) of the account limitFlags accounts whose connections, summed across every server that reports them, exceed a configured share of the limit. The limit applies to the account as a whole, so an account spread over many servers is judged on its total.Raise max_conn in the account JWT, or reduce the account's connections. Moving clients to other servers doesn't help, because clients on every server count toward the same total. Each account has its own limit.
account-002Slow ConsumerswarningErrorsslow_consumers delta > 0Flags accounts with new slow consumer events since each server's previous observation, aggregated across servers. A counter reset or a server's first observation counts as no new events.Identify the affected consumers within this account and increase their processing throughput. Consider adding more consumer instances or reducing message rates.
account-003Inactive or Unresolved JWT ImportwarningConsistencyJWT import inactive, or its runtime attribution ambiguousFlags inactive JWT imports, and duplicate import declarations where the scrape can't tell which one activated the import. Each finding names the likely cause, such as a missing or expired activation token, a token signed by a rotated signing key, or a missing source export.Act on the reason shown for each import. For 'Runtime import attribution is ambiguous', remove or disambiguate the duplicate JWT import declarations. For 'Import subject does not satisfy export account-token position', put the importing account's identity in the configured subject token. For 'Missing activation token', request one from the exporting account's operator. For 'Expired activation token', renew it with the exporting account. For 'Activation token signed by rotated signing key', re-issue it with the current signing key. For 'Source account not found', check the source account in the import and confirm that account is loaded on the servers. For 'Source export not found', verify the export exists in the exporting account's JWT.
account-004Orphaned ExportinfoConsistencyNo matching importer in any accountFlags exports in non-system accounts with no matching importer in any account. An import matches when it names this account as its source, has the same type, and its subject is fully covered by the export subject, using NATS wildcard matching.Remove the unused export from the account JWT, or create the intended import in the consuming account.
account-005No Subscription InterestinfoConsistencyNo subscriptions on imported subjectFlags active stream imports into non-system accounts that no current subscription in the importing account matches, using NATS wildcard subject matching. A remapped import is matched on its local subject. Inactive imports are reported by account-003. Accounts without a complete current subscription snapshot are skipped, so partial or unknown subscription data never counts as no interest.Verify clients are subscribing to the correct subject. Remove the import if it is no longer needed.
account-006Account Subscription LimitwarningSaturationmost saturated open connection > warn_percent% (default: 90) of the per-connection limitFlags accounts with an open client or leafnode connection near the account's subscription limit. NATS applies this limit to each connection separately, so subscriptions on different connections are never added together. A lower user or server subscription limit can cap a connection before it reaches the account limit.Raise max_subs in the account JWT, or cut the subscriptions on the busiest connection. Spreading subscriptions over more connections helps, and consolidating clients onto fewer connections makes it worse. Removing unused subscriptions helps on any connection near the limit.

JetStream​

JetStream checks apply to streams, KV stores, and object stores. Each stream-scoped check result carries an entity_type value (stream, kvstore, or objectstore) indicating which kind of asset the finding relates to.

CodeCheck NameSeverityCategoryThresholdDescriptionRemediation
server-025JS API Pending HighwarningPerformanceInflight requests > max_pending (default: 1000)Flags servers where JetStream API inflight requests exceed the threshold.Reduce the rate of concurrent JetStream API calls. Check for clients making synchronous API calls in tight loops.
server-028JS API Error Rate HighwarningErrorserror_delta / total_delta > error_percent% (default: 1) with total_delta >= min_requests (default: 100)Flags servers where JetStream API errors exceed a percentage of total requests.Check server logs for the API error types. Common ones are permission denials (403), stream or consumer not found (404) and resource exhaustion (503). High error rates usually come from client misconfiguration, such as wrong stream names or missing permissions, more often than from the server.
server-062JS API Request Rate HighwarningPerformancePer-server API request rate > max_rps (default: 50) req/sFlags servers where the JetStream API request rate exceeds the threshold. The rate is measured between consecutive epochs, so a server missing from the previous epoch is not evaluated.Identify the source of API requests. Common causes: rapid stream/consumer creation, excessive info lookups, or tight-loop API calls. Reduce concurrent API call volume. The server queues requests internally and publishes an advisory when the queue saturates. Cache JetStream info responses client-side where possible.
stream-002Stream Replica LagwarningConsistencyReplica > lag_percent% (default: 10) behind leaderFlags stream replicas whose last sequence number trails the leader's by more than the lag percentage of the leader's last sequence. Replicas reporting sequence 0, such as during a snapshot sync, are skipped.Check the lagging replica's server for CPU, disk I/O or network contention. Replicas catch up on their own through Raft. If catchup stalls, force a full snapshot sync as a last resort by removing the peer with nats stream cluster peer-remove and adding it back.
stream-003High Subject CardinalitywarningSaturationnum_subjects > max_subjects (default: 1,000,000)Flags streams with more unique subjects than the configured maximum.Review the subject naming scheme. Consider partitioning high-cardinality data across multiple streams or using a flatter subject hierarchy.
stream-004Stream Message LimitwarningSaturationmessages > warn_percent% (default: 90) of limitFlags streams where message count is above the warning percentage of the limit.Set max_age to expire old messages, increase max_msgs, or set max_msgs_per_subject to spread the limit across subjects.
stream-005Stream Byte LimitwarningSaturationbytes > warn_percent% (default: 90) of limitFlags streams where byte usage is above the warning percentage of the limit.Review the retention and discard policies, then raise max_bytes or set max_age to expire messages you don't need. S2 compression reduces disk use but not the logical bytes counted against this limit.
stream-006Stream Consumer LimitwarningSaturationconsumers > warn_percent% (default: 90) of limitFlags streams where consumer count is above the warning percentage of the limit.Remove unused or inactive consumers, increase max_consumers, or consolidate consumers that read overlapping subject filters.
stream-007Stream Quorum LostcriticalHealthOnline replicas below a strict majority of num_replicas (R > 1)Flags replicated streams where enough replicas are offline to lose quorum. Offline replicas are counted from the stream leader's report, so a stream left with no leader is not flagged here, and server-044 reports its leaderless Raft group.Bring the offline replicas' servers back online. If a server is permanently lost, remove its peer with nats stream cluster peer-remove to lower the quorum requirement so the remaining peers can elect a leader.
stream-008Stream Subject/Message Count InconsistencywarningConsistencynum_subjects > msgs > 0 (invariant violation)Flags streams that report more unique subjects than total messages, which is an inconsistent stream state. Streams with no messages are skipped.Recheck the stream's state and the server logs, and contact support if the inconsistency persists. The finding doesn't show the cause or whether stored messages are damaged.
stream-009Stream Replica Message Count DivergenceinfoConsistencySpread among sampled, leader-confirmed current online replicas > divergence_percent (default: 5) and absolute spread >= min_divergence (default: 1000)Flags replicated streams whose replica message counts differ by at least the absolute threshold and by more than the proportional one. Only the leader and replicas it reports as current and online are compared.Compare replica counts and sequences with nats stream info --all, and check again once publishing and retention settle. Counts are sampled independently, so one spread doesn't mean corruption or lost data. If the difference persists, check peer health, Raft progress and server storage errors.
stream-010Mirror Last Seen StalenesswarningConsistencyinactive longer than stale (default: 5m), with zero lag and local source advancement during that timeFlags zero-lag mirrors that have been inactive longer than the stale threshold while their local source stream advanced during that time. Mirrors of external sources or of themselves, and ambiguous source reports, are skipped.Compare the current state of the mirror and source leaders. Check mirror errors and connectivity before choosing a recovery action.
stream-011Mirror Lag CriticalwarningConsistencymirror_lag >= operator-defined io.nats.monitor.lag-critical metadata threshold (a threshold of 0 disables)Flags mirror streams where mirror lag is at or above the operator-defined io.nats.monitor.lag-critical threshold. Requires io.nats.monitor.enabled set to true and no invalid monitor metadata (stream-034).Check network connectivity to the mirror source and resource contention on the mirror server.
stream-012Mirror Seen CriticalcriticalConsistencymirror_active >= operator-defined io.nats.monitor.seen-critical duration (threshold of 0 or less disables)Flags mirror streams where the time since the mirror was last active is at or above the operator-defined io.nats.monitor.seen-critical threshold. Requires io.nats.monitor.enabled set to true and no invalid monitor metadata (stream-034).Verify the source stream is active and network connectivity is healthy.
stream-013Min SourceswarningHealthnum_sources < operator-defined io.nats.monitor.min-sources metadata threshold (threshold of 0 or less disables)Flags streams where the source count is below the operator-defined io.nats.monitor.min-sources threshold. Requires io.nats.monitor.enabled set to true and no invalid monitor metadata (stream-034).Verify that all expected source streams exist and are configured correctly.
stream-014Max SourceswarningHealthnum_sources > operator-defined io.nats.monitor.max-sources metadata threshold (threshold of 0 or less disables)Flags streams where the source count exceeds the operator-defined io.nats.monitor.max-sources threshold. Requires io.nats.monitor.enabled set to true and no invalid monitor metadata (stream-034).Remove unexpected sources or update the threshold.
stream-015Peer ExpectwarningHealthdistinct peers in the leader's report != operator-defined io.nats.monitor.peer-expect metadata valueFlags streams whose peer count, from the stream leader's report, differs from the positive value set in io.nats.monitor.peer-expect. Zero and negative values are invalid and reported by stream-034. Requires io.nats.monitor.enabled set to true and no invalid monitor metadata.Check for offline replicas or verify the num_replicas configuration.
stream-016Peer Lag CriticalwarningConsistencyleader-reported peer lag > operator-defined io.nats.monitor.peer-lag-critical metadata threshold (a threshold of 0 disables)Flags stream replicas whose lag, as reported by the stream leader, exceeds the operator-defined io.nats.monitor.peer-lag-critical threshold. Requires io.nats.monitor.enabled set to true and no invalid monitor metadata (stream-034).Check the replica's server for resource contention.
stream-017Peer Seen CriticalwarningConsistencyleader-reported peer active time > operator-defined io.nats.monitor.peer-seen-critical duration (threshold of 0 or less disables)Flags stream replicas where the time since the replica was last active, as reported by the stream leader, exceeds the operator-defined io.nats.monitor.peer-seen-critical threshold. Requires io.nats.monitor.enabled set to true and no invalid monitor metadata (stream-034).Check whether the replica's server is offline or has network issues.
stream-018Message Count Thresholdwarning/criticalSaturationmsgs <= either positive metadata minimum; critical checked first; zero disables a boundFlags streams whose message count is at or below a positive io.nats.monitor.msgs-warn or msgs-critical minimum. Critical is checked first whatever order the two values are in, and zero disables a bound. Matches jsm.go. Requires io.nats.monitor.enabled set to true and no invalid monitor metadata (stream-034).Check publishers, retention settings and consumers that remove messages. If a small retained message count is expected, lower the minimum.
stream-019Subject Count Thresholdwarning/criticalSaturationinclusive metadata thresholds; ascending = ceiling, equal/descending = floor; missing = 0Flags streams whose subject count crosses the io.nats.monitor.subjects-warn or subjects-critical threshold. When warn is below critical, the thresholds are ceilings. Otherwise they are floors. A missing value counts as zero, and if both are zero or negative the check is disabled, matching jsm.go. Requires io.nats.monitor.enabled set to true and no invalid monitor metadata (stream-034).Set both subject thresholds. With only subjects-critical set, warn defaults to zero, so every count below critical raises a warning. If the stream has too many subjects, review the naming scheme and consider partitioning high-cardinality data. If it has too few, check whether the publishers for those subjects have stopped.
stream-034Invalid Stream Monitor MetadatawarningConsistencyInvalid recognized monitor metadataFlags streams with invalid values for recognized io.nats.monitor metadata keys. While any value is invalid, none of the stream's monitor thresholds are evaluated.Correct or remove the listed metadata values. Each value must match its key's type (boolean, integer or duration), and peer-expect must be positive. Values are validated even when monitoring is disabled.
system-003Consumer Count ChangewarningErrorsConsumer count delta > max_delta (default: 5000) per epochFlags when the total consumer count rises or falls by more than the threshold between consecutive epochs.Find what is creating and deleting consumers so quickly. Use durable consumers so clients reuse them instead of recreating them. If ephemeral consumers are deleted during brief pauses in activity, raise inactive_threshold (default 5s).

Meta Cluster​

CodeCheck NameSeverityCategoryThresholdDescriptionRemediation
server-030Meta Pending HighwarningPerformancepending > max_pending (default: 500)Flags when a server queues more JetStream CRUD and info requests than the configured threshold.Check JetStream API request volume and server CPU, disk I/O, and request processing latency. Reduce excessive polling or asset churn if requests arrive faster than the server can process them. The count includes routed CRUD and info requests, not unapplied Raft entries.
server-058Offline ReplicawarningHealthMeta peer offline by reporter consensusFlags meta cluster replicas that the reporters agree are offline. A peer is not flagged while another reporter, or the peer itself, still sees it online. An offline peer stays a group member because NATS never evicts one. system-009 reports a peer leaving the group.Bring the offline server back online. Check server logs and network connectivity. If the server is permanently lost, remove it with nats server cluster peer-remove, which requires the remaining members to have quorum.
server-063Meta Snapshot Slowwarning/criticalPerformanceSnapshot duration > warn_duration (default: 5s), critical > crit_duration (default: 30s)Flags a server whose last local meta Raft snapshot took longer than the warning or critical threshold. A slow snapshot delays meta log compaction.Reduce the number of JetStream assets (streams, consumers) to shrink snapshot size. Check disk I/O performance on meta cluster servers.
system-004Meta Leader FlappingwarningErrorsLeader changes > max_changes (default: 1) within window (default: 10m)Flags when the meta cluster leader changes more times than allowed within the lookback window. Changes are counted between consecutive scrapes, so a leader that changes and changes back between two scrapes is missed.Raft sends heartbeats every second and the election timeout is 4-9 seconds, so any disruption longer than 4 seconds starts a new election. Look for network instability between meta cluster peers (even brief packet loss can trigger elections), CPU saturation delaying heartbeats, or disk I/O stalls blocking Raft WAL writes. Run nats server report jetstream to see the current meta state.
system-006Meta Asset Replica Count HighwarningSaturationTotal replicas > max_assets (default: 5000)Flags when the total number of JetStream asset replicas exceeds the threshold.Remove unused streams and consumers. Consider reducing replica counts or consolidating small streams to reduce the total Raft group count.
system-007Meta Quorum Lostwarning/criticalHealthObserved online peers < floor(cluster_size / 2) + 1; warning if no leader is observed across two scrapes after earlier leadershipReports critical when peers reported offline leave the meta group below quorum. Reports warning when the group had a leader earlier but none appears in the two most recent scrapes. Missing JSZ responses can also cause the warning, and it doesn't mean every peer is offline.Check server health, storage and network connectivity, and bring enough of the original meta members back online to form quorum. JetStream operations that change metadata need the meta group, while existing stream groups keep their own quorum. If members are permanently lost, follow a supported disaster-recovery procedure with backups and support. Deleting Raft state is not routine peer removal.
system-008Even Cluster SizeinfoConsistencycluster_size is even and > 1Flags a meta cluster with an even number of peers. An even size tolerates no more failures than the odd size one below it.Use an odd meta cluster size when practical. Quorum is floor(N/2) + 1, so three and four members both tolerate one failure and five tolerate two. If an even cluster splits in half, neither side has quorum and the group loses availability. It still can't produce two leaders that commit independently.
system-009Meta Cluster Size DecreasedwarningHealthDistinct peer count below its peak over the last baseline (default: 30m) for longer than sustain (default: 5m)Flags when the meta cluster's distinct peer count drops below its recent peak and stays down. NATS never evicts an unresponsive peer, so an offline peer still counts here. server-058 reports offline peers. The drop must last longer than the sustain time, so a single missed leader scrape doesn't raise a false alarm.If the peer was removed on purpose (nats server cluster peer-remove), make sure the remaining cluster has an odd number of peers. Otherwise, investigate the lost peer right away. Check its server logs, network connectivity and disk health, since a smaller meta cluster tolerates fewer failures.
system-011Leader DisagreementwarningHealthmore than one server reporting meta leadership at the epochFlags when multiple servers report themselves as the meta cluster leader.Several servers can briefly report leadership while an election changes terms. Raft quorum rules prevent split-brain. If it persists, check network connectivity between cluster peers and look for an asymmetric partition, then restart the lower-term leader to force convergence.

Leafnode​

CodeCheck NameSeverityCategoryThresholdDescriptionRemediation
leafnode-001Leafnode Name WhitespaceinfoConsistencyWhitespace in leaf nameFlags leafnode connections whose remote server name contains whitespace. Whitespace makes the name hard to use in commands and to match across logs.Remove whitespace from the server name in the leafnode's configuration and restart it.
leafnode-002High Leaf RTTwarningPerformancertt > rtt (default: 100ms)Flags leafnode connections with round-trip time exceeding the threshold.Check network latency and congestion between the leaf server and the hub. Consider moving leaf servers closer to their hubs.
leafnode-003Leafnode Subscription Count Highwarning/criticalSaturationsubscriptions > warn_subs (default: 20,000) / crit_subs (default: 50,000)Flags leafnode connections carrying more subscriptions than the warning threshold. The hub processes those subscriptions when the leaf connects, and if that takes longer than the 2-second leafnode connect timeout, the connection drops and retries in a loop.Reduce the subscriptions propagated across the leafnode. Use explicit exports and imports instead of wildcard subscriptions, or consolidate subscriber applications.

Service​

CodeCheck NameSeverityCategoryThresholdDescriptionRemediation
service-001Service Version MismatchwarningConsistencyInstances differ in lang/versionFlags services where instances report different client versions or languages.Complete the rolling deployment so all instances run the same version. If intentional, verify compatibility between versions.
service-002Service DownwarningHealthInstances at the account's previous complete subscription snapshot and none at the next complete oneFlags services that were present in an account's previous complete subscription snapshot and missing from the next complete one. Partial or unknown snapshots are skipped.Restart the service instances. Check application logs and orchestration platform (Kubernetes, systemd) for failure reasons.

Connection​

CodeCheck NameSeverityCategoryThresholdDescriptionRemediation
connection-001High Client RTTwarningPerformancertt > rtt (default: 100ms)Flags open client connections whose round-trip time exceeds the RTT threshold.Move the client closer to the NATS server or use a leafnode to bridge the distance. Check for network congestion.
connection-002Client Pending PressurewarningPerformancepending_bytes > warn_percent% (default: 75) of the server's max_pendingFlags client connections whose queued outbound bytes exceed a percentage of the server's max_pending. The server closes a connection as a slow consumer once its queue reaches max_pending, and past three quarters of it the server stalls publishers sending to that client.Speed up the client's message processing or reduce the publish rate to this subscriber. Raising max_pending buys time but holds a larger backlog in server memory. Pending bytes are sampled once per scrape, so a backlog that fills between scrapes can hit the limit without this check seeing it. server-013 (stalled clients) and server-004 (slow consumers) keep cumulative counts of those events and miss none.
connection-003Connection StoppedinfoErrorsstop_time set with a reason other than 'Client Closed' or 'Server Shutdown'Flags connections closed for an abnormal reason. Graceful client closes ('Client Closed') and planned server shutdowns ('Server Shutdown') are excluded.Check the stop reason in the detail column. 'Slow Consumer (Pending Bytes)' and 'Slow Consumer (Write Deadline)' mean the client couldn't keep up with the message rate. 'Authentication Failure' means invalid or expired credentials. 'Maximum Connections Exceeded' means a server or account limit was reached. 'Stale Connection' means the client missed ping/pong keepalives.

Consumer​

CodeCheck NameSeverityCategoryThresholdDescriptionRemediation
consumer-001Consumer Replica OfflinewarningHealthis_offline = trueFlags consumer replicas that are reported as offline.Bring the offline server back online. Check server logs for the reason the replica went offline.
consumer-002Consumer Replica LagwarningConsistencyonline replica lag > max_lag (default: 1000) operationsFlags online consumer replicas lagging behind the leader by more than the maximum allowed number of operations. Offline replicas are reported by consumer-001.Check for resource contention on the lagging server. If persistent, consider removing and re-adding the consumer replica.
consumer-003Consumer Quorum LostcriticalHealthOnline replicas < quorum needed (R > 1)Flags replicated consumers with enough replicas offline to lose quorum. The replica count comes from the reported peer set, so consumers that inherit their replica count from the stream (num_replicas 0) are included. When the consumer leader is offline no replica rows are reported, so that quorum loss is not flagged.Restore offline replicas by bringing their servers back online. Without quorum the consumer cannot make progress.
consumer-004Consumer Delivered Below Stream First SequenceinfoConsistencydelivered_consumer_seq > 0 and 0 < delivered_stream_seq < stream first_seq - 1Reports consumers with delivery history whose reported stream position is more than one sequence below the stream's current first sequence.Inspect the stream's origin, filters, pending messages and application activity. Retention, expiry, message deletion and some filtered purges can open a gap, while a full purge normally moves the consumer to the adjacent boundary. Consumer filters and holes in a filtered mirror can also skip positions the consumer never needed to receive, even when the consumer has no filter itself. The size of the gap doesn't count missed messages or show that the consumer is stuck.
consumer-005Consumer Sequence Ahead of Stream SequenceinfoConsistencydelivered_consumer_seq > 0 and delivered_stream_seq > stream last_seq > 0Flags consumers with delivery history whose reported stream position is ahead of the parent stream's observed last sequence.Compare fresh stream and consumer state before considering recovery. The two are sampled separately, even within one server's JSZ response, so normal publishing can move the consumer past an older stream snapshot, and this can recur across scrapes. The finding doesn't indicate lost data or corruption.
consumer-006Outstanding Ack CriticalcriticalHealthnum_ack_pending >= operator-defined io.nats.monitor.outstanding-ack-critical metadata threshold (monitoring enabled, threshold of 0 or less disables)Flags consumers with io.nats.monitor.enabled set to true where num_ack_pending on the leader is at or above the operator-defined io.nats.monitor.outstanding-ack-critical threshold. Invalid monitor metadata skips the consumer (see consumer-020).Increase consumer throughput, add consumer instances, or raise the threshold in the consumer metadata.
consumer-007Waiting CriticalcriticalHealthnum_waiting >= operator-defined io.nats.monitor.waiting-critical metadata threshold (monitoring enabled, threshold of 0 or less disables)Flags consumers with io.nats.monitor.enabled set to true where num_waiting on the leader is at or above the operator-defined io.nats.monitor.waiting-critical threshold. Invalid monitor metadata skips the consumer (see consumer-020).Inspect outstanding fetches and message demand, and avoid redundant or overly long-lived pull requests. If this many waiting requests is intentional, raise the threshold.
consumer-008Unprocessed CriticalcriticalHealthnum_pending >= operator-defined io.nats.monitor.unprocessed-critical metadata threshold (monitoring enabled, threshold of 0 or less disables)Flags consumers with io.nats.monitor.enabled set to true where num_pending on the leader is at or above the operator-defined io.nats.monitor.unprocessed-critical threshold. Invalid monitor metadata skips the consumer (see consumer-020).Add consumer processing capacity or look for a stalled consumer.
consumer-009Last Delivery CriticalcriticalHealthno recorded timestamp or time since last delivery >= operator-defined io.nats.monitor.last-delivery-critical metadata threshold (monitoring enabled, threshold of 0 or less disables)Flags consumers with io.nats.monitor.enabled set to true whose last delivery, as reported by the leader, is at least as old as the io.nats.monitor.last-delivery-critical threshold, or that report no delivery timestamp. The threshold is a duration such as 30s or 5m. Zero or less disables the check. Invalid monitor metadata skips the consumer (see consumer-020).Check whether processing is stalled or the stream is idle. A missing delivery timestamp can also follow a server restart or leader change, so check consumer sequences and application activity before concluding nothing was delivered.
consumer-010Last Ack CriticalcriticalHealthno recorded timestamp or time since last ack >= operator-defined io.nats.monitor.last-ack-critical metadata threshold (monitoring enabled, threshold of 0 or less disables)Flags consumers with io.nats.monitor.enabled set to true whose last acknowledgement, as reported by the leader, is at least as old as the io.nats.monitor.last-ack-critical threshold, or that report no acknowledgement timestamp. The threshold is a duration such as 30s or 5m. Zero or less disables the check. Invalid monitor metadata skips the consumer (see consumer-020).Check whether processing is stalled or the stream is idle. A missing acknowledgement timestamp can also follow a server restart or leader change, so check consumer sequences and application activity before concluding nothing was acknowledged.
consumer-011Redelivery CriticalcriticalErrorsnum_redelivered >= operator-defined io.nats.monitor.redelivery-critical metadata threshold (monitoring enabled, threshold of 0 or less disables)Flags consumers with io.nats.monitor.enabled set to true where num_redelivered on the leader is at or above the operator-defined io.nats.monitor.redelivery-critical threshold. Invalid monitor metadata skips the consumer (see consumer-020).Investigate processing failures and acknowledgement timeouts, and tune retry timing. The count can include messages that exhausted max_deliver, so arrange recovery for those.
consumer-012Pinned Consumer Policy MismatchwarningConsistencyio.nats.monitor.enabled and io.nats.monitor.pinned metadata true but priority_policy != pinned_client (NATS Server 2.11.0+)Flags consumers with io.nats.monitor.enabled and io.nats.monitor.pinned set to true that are not using the pinned_client priority policy, so the declared client pinning is not in effect. Invalid monitor metadata skips the consumer (see consumer-020).Recreate the consumer with priority_policy=pinned_client (e.g. nats consumer add --pinned-groups), or edit its configuration with nats consumer edit --config <file>.
consumer-020Invalid Consumer Monitor MetadatawarningConsistencyInvalid recognized monitor metadataFlags consumers with invalid values for recognized io.nats.monitor metadata keys. Any invalid key stops every monitor threshold check on that consumer, so this reports a configuration error rather than a threshold breach.Correct or remove the listed metadata values so each boolean, integer or duration value matches its key's type. Values are validated even when monitoring is disabled.

User​

CodeCheck NameSeverityCategoryThresholdDescriptionRemediation
user-001Bearer Token UserinfoErrorsbearer = true with active connectionsFlags bearer token users with active connections.Migrate the user to NKey-based authentication where possible. Bearer tokens skip nonce signature verification during CONNECT, relying on JWT validity alone. They are appropriate for WebSocket and HTTP contexts where NKey signing is impractical, but should not be used for long-lived server-to-server connections.
user-002Excessive User ConnectionsinfoErrorsconnections > max_connections (default: 100)Flags users with more active connections than the configured maximum.Investigate why a single user has so many connections. Consider using connection pooling or separate user credentials per service instance.

Change​

CodeCheck NameSeverityCategoryThresholdDescriptionRemediation
server-023Config Reload DetectedinfoChangeconfig_load_time differs from the server's previous recorded valueDetects servers whose configuration was reloaded by comparing config_load_time with the server's own previous value. A restart is not a reload and is covered by server-008.Verify the configuration change was intentional. Review server logs to confirm the reload was successful and no errors occurred.
server-024JetStream Domain ChangedwarningChangejs_domain differs from the value last reported under the same server nameFlags a server whose JetStream domain differs from the value it last reported. Changing the domain requires a restart, which gives the server a new pk, so the comparison is matched by server name.Confirm the domain change was intended and the new domain is correct. A different JetStream domain changes how streams and consumers are addressed across clusters, so check that dependent clients and mirrors were updated.
stream-001Stream Configuration ChangedinfoChangenum_replicas, retention_policy, max_msgs, max_bytes, max_age or max_consumers differs from the stream's previous configFlags streams whose num_replicas, retention_policy, max_msgs, max_bytes, max_age or max_consumers changed from the stream's previous reported configuration. Changes to other settings are not flagged.Verify the stream configuration change was intentional and monitor for downstream effects on consumers.

Optimization Checks​

Account​

CodeCheck NameSeverityCategoryThresholdDescriptionRemediation
account-007Account Storage Quota Approaching LimitwarningSaturationFile storage reservations > warn_percent% (default: 90) of the applicable JWT limit set; most saturated set per accountFlags accounts whose file-stream JetStream reservations (max_bytes) are near the disk quota in any limit set. Untiered quotas count max_bytes once per replica, and an Rn tier counts max_bytes once for each stream in that tier. A stream without a leader still holds its reservation. Actual storage use is covered by a separate check.Reduce unused max_bytes reservations or raise the account's disk quota for that limit set. Until then, new reservations can be rejected even while existing streams still have room to write.
account-008Excessive Account Claims SizeinfoConsistencyclaims_size > max_size (default: 64 KiB)Flags accounts whose decoded claims JSON exceeds the configured byte threshold. The measured value is the stored JSON representation, not the signed JWT's wire size.Review large imports, exports, signing key lists, revocations and metadata in the account claims, and remove entries that are no longer needed. Large claims add parsing work each time the account JWT is fetched or changed. Servers cache resolved accounts, so the cost isn't paid on every connection.

Placement​

CodeCheck NameSeverityCategoryThresholdDescriptionRemediation
account-011Cross-Cluster Stream AccessinfoPerformance>= min_conn_count (default: 5) clients and >= min_conn_percent% (default: 10) of the account's connections in a cluster with no local stream leadersFlags accounts with clients in a cluster that leads none of the account's streams, when the account does lead streams in another cluster. Both floors keep a few stray clients from turning into placement advice.First check whether these clients use the account's streams at all, since clients that only use core NATS need no local stream leader. A stream's replicas all live in one cluster, so where clients reach it across a gateway by accident, consider moving the stream, adding a local mirror, or moving the clients closer to it.
account-012High Gateway Traffic RatioinfoPerformanceGateway traffic > gateway_percent% (default: 30) of account traffic within the selected time range, with >= min_traffic (default: 1 MiB) movedReports the share of an account's bytes that crossed gateways in the selected window. Each server measures from its own previous scrape, which can fall before the window starts. A counter reset adds no traffic for that interval.Compare the gateway share with where publishers, subscribers and streams are meant to run. An account spread across clusters can still keep its traffic local, so client placement does not predict the ratio. Look for clients reaching remote clusters by accident, and consider moving a stream or running a responder closer to its clients. If this share is expected for your deployment, disable account-012.
account-014Gateway Interest ModeinfoPerformanceinterest_mode = 'Optimistic'Flags accounts still using optimistic interest mode on a gateway to a remote cluster. Reports from every local server are combined into one finding per account and remote cluster.In optimistic mode a gateway forwards messages before it knows the remote side has interest, then stops sending subjects it learns have none. NATS 2.9+ normally switches accounts to interest-only mode when gateways connect or accounts load, so check both ends of the gateway and whether this is a brief transition. In the legacy optimistic path, the switch happens once enough distinct no-interest subjects build up. The subscription count shown is not that threshold.
stream-033Stream Leader Not Co-locatedinfoPerformancestream leader in a different cluster than the account's dominant client connection cluster, with at least min_conns (default: 5) total connections and the dominant cluster holding at least min_dominance_percent% (default: 60) of themFlags streams whose leader is in a different cluster from most of the account's client connections. Clients in other clusters reach the leader through gateways. Consumer Raft groups follow their stream's placement, so the finding is reported on the stream, where placement can be changed.Use preferred placement tags to co-locate the stream leader with the majority of its clients. To force a leader election, use nats stream cluster step-down, which may relocate the leader to a better-positioned replica.

Cost​

CodeCheck NameSeverityCategoryThresholdDescriptionRemediation
server-038Wasted JetStream Memory ReservationinfoConsistencyMemory usage < min_utilization_percent% (default: 20) of reservedFlags servers where JetStream memory usage is below the minimum utilization of reserved capacity.Reduce the JetStream memory reservation to match actual usage, or migrate memory-backed streams to this server to improve utilization.
server-039Wasted JetStream Storage ReservationinfoConsistencyStorage usage < min_utilization_percent% (default: 20) of reservedFlags servers where JetStream storage usage is below the minimum utilization of reserved capacity.Reduce the JetStream storage reservation to match actual usage, or migrate file-backed streams to this server to improve utilization.
stream-021Over-Replicated Inactive StreaminfoConsistencyR3+ stream with no new messages across time rangeFlags R3+ streams that received no new messages across the selected time range. The stream must be observed at or before the range's start and at its end. Sealed streams are skipped.Extra replicas still use storage and Raft resources. Reduce the replica count to R1 for inactive streams, or delete the stream if it is no longer needed.
stream-022Memory Storage Large StreaminfoSaturationMemory-backed stream > max_memory (default: 100 MiB)Flags memory-backed streams using more memory than the configured maximum.Convert the stream to file-backed storage if low-latency access is not required. Memory-backed streams consume server RAM directly.
stream-023Uncompressed Large StreaminfoConsistencyFile stream > max_uncompressed (default: 1 GiB) with compression disabledFlags file-backed streams larger than the uncompressed size threshold with no compression enabled.Enable S2 compression on the stream configuration to reduce disk usage and I/O costs.

Balance​

CodeCheck NameSeverityCategoryThresholdDescriptionRemediation
server-031Uneven Leader DistributioninfoSaturationLeaders > 1.5x cluster average (min 3 servers)Flags servers hosting disproportionately many stream and consumer leaders. Only JetStream servers in a named cluster are counted, and a server that answered only STATSZ is left out of the comparison entirely.Use nats stream cluster step-down and nats consumer cluster step-down to redistribute leaders across the cluster. Target servers with the highest leader counts first.
server-032Connection HotspotinfoSaturationConnections > 2x cluster average (min 100 absolute)Flags servers with more than double the cluster average connections.Review client connection configuration. Use DNS round-robin or load balancer to distribute connections more evenly across cluster servers.
server-033Subscription HotspotinfoSaturationSubscriptions > 2x cluster average (min 100 absolute)Flags servers with more than double the cluster average subscriptions.Redistribute client connections to balance subscription load. Check for clients with excessive subscriptions.
server-034Stream Replica Count ImbalanceinfoSaturationReplicas > 1.5x cluster average (min 3 servers, min 10 replicas)Flags servers hosting disproportionately many stream replicas, which concentrates storage I/O and memory on a few servers. Only JetStream servers in a named cluster are counted, and a server that answered only STATSZ is left out of the comparison entirely.Use placement tags to distribute new streams more evenly. Consider removing and re-adding replicas to rebalance.
server-035JetStream Storage SkewinfoSaturationStorage > 2x cluster average (min 1 GiB absolute)Flags servers whose JetStream storage exceeds double the cluster average.Migrate large streams to other cluster servers or add storage capacity to balance disk usage.
server-036Account Connection ConcentrationinfoSaturation> 70% of the account's client connections in a cluster on one server (min 3 servers, min 10 connections)Flags servers hosting more than 70% of an account's open client connections within their cluster. An account connected through fewer than 3 of the cluster's servers is not evaluated.Configure client connection URLs to include multiple servers. Use a load balancer to spread connections across the cluster.
server-037JetStream Storage Saturation with SkewwarningSaturationstorage > saturation_percent% (default: 90) of max_store AND cluster skew (max - min utilization) > skew_pp (default: 30) percentage pointsFlags servers near their configured max_store JetStream storage limit in a cluster whose storage is also unevenly spread across servers.New writes fail on this server once it reaches max_store. Move streams from it to less-used peers, or raise max_store or add disk capacity on this server. Use placement tags to guide future stream placement.
stream-020Stream-Consumer Leader Co-locationinfoSaturationReplicated consumer leaders on stream leader's server > consumer_leader_percent% (default: 50) (min 3 replicated consumer leaders)Flags streams where the stream leader's server also hosts a disproportionate share of replicated consumer leaders. R1 consumers are excluded because their leader cannot be moved off its only replica's server. Streams with fewer than 3 replicated consumer leaders are skipped.Run nats consumer cluster step-down on consumers led from the stream leader's server to spread their leaders across the cluster. This moves acknowledgement and delivery work, and the I/O and CPU load that comes with it, off a single node.

Idle Resources​

CodeCheck NameSeverityCategoryThresholdDescriptionRemediation
account-009Inactive AccountinfoHealthZero connections and zero throughput for threshold (default: 24h), excludes systemFlags non-system accounts that are currently reported but have had no client connections, leaf connections or byte counter increases for longer than the inactivity threshold. A byte counter that decreases also counts as activity, since the account may have been removed and re-registered. An account must have been observed for at least the threshold before it can be flagged.Confirm the account is no longer needed, then remove or disable it.
connection-004Idle Client ConnectionsinfoConsistencyIdle > idle (default: 5m) with zero lifetime messagesFlags client connections idle for longer than the idle threshold that have never sent or received a message.Review the client's purpose, subscriptions and expected traffic pattern. A subscriber waiting for rare messages, an idle request/reply client or a health connection can be intentionally quiet. Close the connection only after establishing that it is unused or leaked.
consumer-013Inactive ConsumerinfoConsistencydelivered_stream_seq unchanged between earliest and latest samples in range (min 2 samples, consumer reporting at current epoch)Flags consumers whose last delivered stream sequence is unchanged between the earliest and latest observations in the time range. Requires at least two samples in the range, the latest at the current epoch.Inspect pending messages and application activity. Redelivery attempts can continue while the stream position remains unchanged. Remove the consumer only after confirming it is no longer needed.
consumer-014Drained ConsumerinfoConsistencynum_pending = 0 AND num_ack_pending = 0 at current epoch, stream last_seq unchanged between earliest and latest samples in range (min 2 samples)Flags consumers with nothing pending delivery or acknowledgement at the current epoch, on a stream whose last sequence didn't change across at least two samples in the window.Confirm the consumer is no longer needed before removing it. Both counters can be zero while messages that exhausted max_deliver stay unprocessed in the stream, so check application ownership, retained messages and delivery exhaustion advisories.
server-040Underutilized ServerinfoHealthMax per-core CPU < max_cpu_percent% (default: 5) AND max connections < max_connections (default: 10) across range (min 5 samples, server reporting at current epoch)Flags servers that remained nearly idle across the selected time range.Consider decommissioning the server or migrating workload to it from busier servers.
stream-024Inactive StreaminfoHealthlast_seq unchanged across time range (excludes sealed)Flags unsealed streams that received no new messages across the selected time range. The stream must be observed at or before the range's start and at its end.Delete the stream if it is no longer needed, or seal it to prevent accidental writes. If temporarily inactive, no action is needed.

System Improvement​

CodeCheck NameSeverityCategoryThresholdDescriptionRemediation
account-013Unlimited JetStream AccountinfoConsistencyany limit set with js_mem_storage = -1 OR js_disk_storage = -1, for a class a server offers (excludes system)Flags non-system JetStream accounts whose JWT leaves memory or disk storage unlimited in any limit set, for a storage class at least one server offers. Each tier of a tiered JWT is checked separately, and a class limited to zero counts as disabled. Accounts without a JWT are not checked.Set explicit JetStream memory and disk storage limits in the account JWT, in every tier for a tiered JWT, so one account cannot reserve capacity until the cluster is exhausted.
consumer-016High Consumer RedeliverywarningErrorscurrent num_redelivered > redelivery_percent% (default: 10) of delivery attempts within the check window, with at least 100 attemptsFlags consumers whose current count of messages tracked for redelivery is high compared with delivery attempts in the window. Tracked messages can predate the window or have exhausted max_deliver, so the ratio is not the share of attempts that were redeliveries. Requires at least 100 delivery attempts in the window.Investigate acknowledgement timeouts and processing failures, choose retry timing for the workload, and arrange recovery for messages that exhausted max_deliver.
consumer-017Ack Pending BuildupwarningErrorsnum_ack_pending > warn_percent% (default: 90) of max_ack_pendingFlags consumers approaching their maximum ack pending limit. At the limit, the server stops delivering until acks catch up. Consumers with unlimited max_ack_pending are skipped.Scale out consumer instances to process messages faster, increase max_ack_pending (default 1,000), or investigate why messages are not being acknowledged. Ack pending can also be limited at the stream level via consumer limits and at the account level.
consumer-018Unbound Push ConsumerinfoErrorsPush consumer with no bound subscriberFlags push consumers with no subscriber currently bound.If delivery is expected, start the subscribing application. The server holds push deliveries until a subscriber binds, so messages stay pending without using up delivery attempts. Check pending messages and application ownership before removing an unused consumer or replacing it with a pull consumer.
consumer-019Consumer ACK Floor Divergenceinfo/warningErrorsgap > elevated_multiplierx (default: 2) max_ack_pending or elevated_abs (default: 100,000); escalates to warning above severe_multiplierx (default: 5) or severe_abs (default: 1,000,000)Flags consumers whose consumer-sequence gap between delivered position and ACK floor is a large multiple of max_ack_pending, or exceeds an absolute threshold. The gap includes redeliveries and messages already acknowledged above the floor, so it is not the outstanding message count.Compare num_ack_pending and num_redelivered, then investigate slow messages, out-of-order processing and retry patterns. Interest and workqueue streams can remove acknowledged messages above the floor, so a large gap doesn't mean storage is retained. Acknowledge each message only after its processing is complete.
leafnode-004Leaf Compression DisabledinfoConsistencycompression = 'off'Flags leaf connections reporting compression off after negotiation. Compression reduces bandwidth between leaf and hub.Enable S2 compression in the leafnode configuration (compression: s2_auto adapts compression to RTT). Check both sides, since the negotiated mode doesn't show which one turned compression off.
server-041Route Pending PressureinfoPerformancepending_size > pending (default: 1 MiB)Flags route connections with more pending data than the pending threshold.Investigate network bandwidth between cluster peers and load on the remote peer. Reduce message rates on high-volume intra-cluster subjects or upgrade network capacity between peers.
server-042Gateway Pending PressureinfoPerformancepending_size > pending (default: 1 MiB)Flags gateway connections with more pending data than the pending threshold.Investigate network bandwidth between clusters and load on the remote cluster. Reduce inter-cluster message rates by improving stream/consumer placement, or upgrade inter-cluster network capacity.
server-043Raft Apply LagwarningPerformancecommitted - applied > max_lag (default: 100) AND catching_up = falseFlags Raft groups where applied entries trail committed entries by more than the maximum allowed lag. Committed entries take effect only once applied. Members that are catching up are excluded, and server-048 reports sustained catch-up.Check disk I/O and CPU on the affected server, which is falling behind in applying committed Raft entries.
server-044Leaderless Raft GroupcriticalHealthleader = '' AND ever_had_leader = trueFlags a Raft group that has had a leader before and now reports none. A leaderless group cannot accept writes. This happens during an election or a sustained loss of quorum.Check whether later scrapes show a completed election. If there is still no leader, confirm that a majority of the group's peers are online and reachable, and check server logs for storage or network failures.
server-045Raft IPQ BackpressurewarningPerformanceGREATEST(ipq_prop_len, ipq_entry_len, ipq_resp_len, ipq_apply_len) > max_ipq (default: 1000)Flags a Raft group whose internal queue lengths exceed the threshold, a sign of a processing backlog.Raft's internal queues (proposals, append entries, apply, responses) are backing up. The apply queue matters most, since a long one means JetStream cannot consume committed entries fast enough. Check server CPU, disk I/O and network latency.
server-046Subscription Fanout AnomalyinfoConsistencymax_fanout > multiplier (default: 10) x avg_fanout AND avg_fanout > 1Flags servers where max fanout is disproportionately higher than average fanout.Investigate subjects with high subscriber counts. A large max-to-average fanout ratio indicates one or more subjects with excessive subscribers, which can create hot spots.
server-047Subscription ChurninfoErrorschurn delta > max_churn (default: 10,000) between adjacent server observationsFlags servers with excessive subscription insert and remove operations since their own previous observation.If one client is responsible, it is likely subscribing and unsubscribing in a loop. Find it by connection name or IP and fix the client code. If many clients are responsible, it is likely a reconnection storm, with clients re-subscribing all at once. Look for a network event or server restart just before it.
server-048Raft Sustained Catching Upinfo/warningHealthcatching_up = true; escalates to warning once held > sustain (default: 30m)Flags a Raft group member catching up to its leader. Catch-up usually finishes between scrapes, so a single sighting is info. A member still catching up on every scrape for longer than the sustain window is a warning, since it cannot serve as a failover target and may not keep pace with the group.Check disk I/O, network bandwidth and CPU on the catching-up server. A short catch-up is usually a burst being replayed. A sustained one means the server needs more resources or a re-sync.
server-050Subscription Count GrowthinfoErrorsGrowth over min_epochs (default: 10) observations with >= min_growth_percent% (default: 20) increase, at least min_mono_percent% (default: 80) of adjacent intervals non-decreasing, and connections changed < 5%Flags servers whose subscription count grows steadily while the connection count ends within 5% of where it started, a sign of a subscription leak. Most reliable on a single server, because on clustered or leaf node servers the count also includes remote interest.Find the client creating subscriptions without unsubscribing by comparing per-connection subscription counts, then fix it to unsubscribe when done. On clustered or leaf node servers the subscription count includes interest propagated over routes and leaf connections, while connections count only local clients, so the growth may be cluster-wide interest instead of a local leak. Confirm with per-connection counts before calling it a leak.
server-051Raft WAL Size Excessivewarning/criticalSaturationWAL > warn_size (default: 10 GiB) or warn_percent% (default: 50) of js_max_store; critical above crit_size (default: 50 GiB) or crit_percent% (default: 80)Flags Raft groups with an excessively large write-ahead log, risking disk exhaustion and cascading OOM failures. Triggers on absolute size or relative to js_max_store. The relative thresholds apply only when js_max_store is set.Check Raft catch-up, snapshot creation and log compaction on the affected server, along with free disk space and I/O latency. Raft commits depend on replication quorum, so unacknowledged consumer messages do not hold the log back. Contact support if the WAL keeps growing or snapshots fail.
server-052Raft Group Peer Count MismatchwarningConsistencyObserved peers > configured num_replicasFlags Raft groups where the observed peer count exceeds the expected replica count from stream or consumer configuration. A consumer with num_replicas 0 inherits its stream's count. Each group is reported once, against its leader when the leader reports it.This usually follows a peer-remove then peer-add where the old peer was not fully removed, or a replica count decrease that did not fully propagate. Remove the extra peer with nats stream cluster peer-remove, or update num_replicas to the desired count.
stream-025Streams Without LimitsinfoConsistencymax_msgs = -1 AND max_bytes = -1 AND max_age = 0 AND max_msgs_per_subject = -1 (excludes sealed)Flags unsealed streams with no message, byte, age or per-subject limit.Configure at least one retention limit (max_msgs, max_bytes, max_age, or max_msgs_per_subject) to prevent unbounded disk growth.
stream-026Direct Gets DisabledinfoPerformanceallow_direct = false AND num_replicas > 1Flags replicated streams with allow_direct disabled, so individual message-get requests must be served by the stream leader. R1 streams are skipped, since their single server serves every read either way.Enable allow_direct so clients that use the direct-get API can read from replicas and spread read load. Before enabling it, check whether replica freshness meets the clients' read requirements.
stream-027High Interior Deletes on StreaminfoSaturationInterior deletes > max_deleted (default: 100,000,000) or holes exceed 90% of retained sequence positionsFlags streams with many interior sequence holes or a high share of deleted positions. Deleted ranges are stored compactly, so the count does not measure memory use.Check measured memory use and recovery time, and review retention and update patterns. If expiring retained messages is acceptable, setting max_age can reduce old sequence holes.
stream-028Large Deduplication Windowinfo/warningSaturationwindow > elevated_window (default: 1h) or severe_window (default: 6h) AND last-sequence advancement/sec >= min_rate_per_sec (default: 10)Flags streams with a long deduplication window and active publishing, estimated from sequence advancement. The window can use a lot of memory when publishers set message IDs. Needs a previous observation in the selected range.The server keeps a deduplication entry in memory for each distinct Nats-Msg-Id published within the window. The publish rate is estimated from sequence advancement, which can't tell whether publishers set IDs and can overstate the rate after sequence jumps. Confirm publishers set Nats-Msg-Id, then size the window to their retry interval.
stream-029KV Buckets Without max_ageinfoSaturationmax_age = 0 AND deleted >= min_deleted (default: 100,000)Flags KV buckets with no max_age configured that have accumulated a large number of interior deletes.Key overwrites and history limits create these holes without any key being deleted, so the count is not the number of KV delete markers. Check memory use and recovery time before changing retention. Setting max_age also expires live key values and history, so use it only if that TTL suits the application.
stream-030R1 Streams in Multi-Node ClustersinfoHealthnum_replicas = 1 in multi-node clusterFlags R1 (single-replica) streams, which have no redundancy, in clusters with more than one JetStream node. Mirrors are excluded.If the hosting node goes down, the stream is offline until that node recovers. Consider R3, which needs three eligible nodes with enough capacity (placement tags and resource limits can rule nodes out). R1 is fine for ephemeral, cacheable or easily reproducible data.
stream-031WorkQueue Discard New with Aggressive Consumer SettingswarningConsistencydiscard_policy=new AND ack_policy != flow_control AND (ack_wait < min_ack_wait (default: 30s) OR 1 <= max_deliver < min_max_deliver (default: 10))Flags WorkQueue streams with discard_policy=new that have a consumer using timed acknowledgements with a short ack_wait or low max_deliver. Fast retries can exhaust delivery attempts and leave work retained until the stream fills.Set ack_wait longer than processing time, allow enough deliveries for retries, and plan how to recover messages that exhaust max_deliver. Those messages stay in the stream until acknowledged or removed, and once a limit is reached the stream rejects new publishes, so monitor rejected publishes and make sure publishers handle the error. Switching to discard-old can evict unprocessed work and does not fix retries.
stream-032Sustained Consumer Growth on StreamwarningErrorsAt least min_growth_epochs (default: 5) observations, net growth >= min_total_growth (default: 500), and at least min_mono_percent% (default: 80) of adjacent intervals non-decreasingFlags streams where consumer count has been growing steadily, which can indicate a consumer leak.Find what creates the consumers. Set inactive_threshold on ephemeral consumers so unused ones expire (it may be set too high), or use durable consumers and delete them when they're no longer needed.