Skip to main content

Checks

Insights runs over 140 automated checks against every system it indexes, on every epoch. Operational checks evaluate the health and operational state of your NATS deployment at that epoch; optimization checks evaluate a trailing window ending at it (six hours by default) to surface waste, imbalance, and slow trends. Each check produces findings with a severity level and remediation guidance, giving operators actionable information rather than raw metrics.

For the full catalogue of checks (codes, severities, thresholds, and remediation), see the Checks Reference.

How Checks Work​

When an epoch commits, Insights evaluates every check against that system's data, using that system's thresholds, and stores the results in hx.check_findings in the same transaction as the epoch. A check examines specific entities (servers, streams, consumers, accounts, connections, leaf nodes, users, services, or the system as a whole) and produces a finding when it detects a condition that warrants attention. A row in hx.check_findings is an entity that tripped a check: there are no rows for entities that passed.

Each finding includes:

  • Severity: how urgent the condition is
  • Affected entity: the specific server, stream, consumer, or other entity involved
  • Description: what was detected
  • Remediation: guidance on how to investigate and resolve the issue

Findings appear on entity detail pages and in the entity tables of the web UI, and the System page grades each category by the share of entities that pass its checks. insights checks findings <code> evaluates one check on demand over a requested time range.

Check Codes​

Every check is identified by a lowercase, hyphenated code: the entity the finding is actionable at, then a number. For example, server-003 (High CPU Usage) or stream-002 (Stream Replica Lag). The prefixes are server, stream, consumer, account, connection, leafnode, user, service, and system.

Codes are stable identifiers, not a dense sequence: a retired code is never reused, so gaps are expected. The same spelling is used everywhere a check is named: check-thresholds and disabled-checks in configuration, notification subscriptions and ignore patterns, the finding feed subject, the Alertmanager alertname, and hx.check_findings.code. Run insights checks list for the current set and insights checks info <code> for one check's parameters.

Finding Lifecycle​

A finding is identified by its check code and entity. After each epoch, a findings tracker for each system compares that epoch's findings with the previous epoch's and reports each change as a transition:

StatusMeaning
firingThe finding is present now and was absent in the previous epoch
changedThe finding is still present, at a different severity; the transition carries the severity it left
resolvedThe finding was present in the previous epoch and is absent now

Severity is not part of a finding's identity, so a check that escalates reports changed rather than a resolve followed by a new fire. Every transition carries starts_at, the first epoch of the finding's current unbroken run, which is derived from the stored history and so survives a restart; a resolve also carries ends_at.

Transitions are published on $INS.sys.<id>.feed.finding.<status>.<severity>.<code>, so a subscriber filters on the subject rather than on the payload:

nats sub '$INS.sys.prod.feed.finding.firing.critical.>'
nats sub '$INS.sys.prod.feed.finding.*.*.stream-002'

The same transitions drive webhook and Alertmanager delivery. See the Notifications guide for endpoint configuration.

Severity Levels​

SeverityMeaning
CriticalImmediate attention required. The system is experiencing or is about to experience an outage, data loss, or significant degradation.
WarningAction should be taken soon. The condition may lead to problems if left unaddressed, or it indicates a configuration that does not follow best practices.
InfoInformational finding. No immediate action is required, but the condition is worth noting for optimization or awareness.

Most checks have a fixed severity. A few compute it from the measurement, such as server-063 (Meta Snapshot Slow), which is a warning or critical depending on how long the snapshot took.

Check Categories​

Checks are organized into six categories, each addressing a distinct operational concern. Every check belongs to exactly one category.

Health & Availability​

Is the system up and reachable?

These checks monitor the fundamental operational state of the NATS deployment. They detect offline stream and consumer replicas, lost stream, consumer, and meta quorum, leaderless Raft groups, route and gateway disconnections, a JetStream subsystem that is not ready or cannot write to its store, servers that stopped answering monitoring requests, and services that are down. They also flag underutilized servers and inactive accounts and streams that may indicate decommissioned infrastructure.

Performance & Latency​

Is the system fast enough?

These checks evaluate latency and throughput across the system. They detect high CPU usage, slow consumers, elevated round-trip times on routes, gateways, leaf nodes, and client connections, JetStream API and meta pressure, slow meta snapshots, Raft apply lag, pending message buildup, and suboptimal placement of stream leaders relative to their consumers' clients.

Error & Failure Patterns​

What is failing, and how?

These checks identify recurring error conditions and failure modes. They detect connection churn spikes, consumer churn, leader flapping, slow consumer evictions, high redelivery rates, ack pending saturation, JetStream API errors, and subscription churn. They also flag security concerns like bearer token usage and excessive per-user connection counts.

Resource Saturation​

Are resources running out?

These checks track resource consumption against limits. They detect JetStream memory and storage pressure, connection counts approaching maximums, memory usage outliers, high HA asset counts that degrade Raft performance, approaching stream limits (messages, bytes, consumers), account-level limit pressure, and uneven distribution of leaders, replicas, connections, storage, and subscriptions across cluster members.

Data & State Consistency​

Is the data correct and coherent?

These checks validate that the system is configured correctly and consistently. They detect stream, consumer, mirror, and peer replica lag, JetStream assets the cluster rejected or that are missing, version mismatches across servers and services, naming issues (whitespace in cluster or leaf node names), gateway configuration mismatches, even-numbered meta cluster sizes (which weaken quorum), inactive imports, orphaned exports, imports without subscription interest, over-replicated inactive streams, wasted JetStream reservations, streams without retention limits, disabled compression, invalid monitoring metadata, and unlimited JetStream accounts.

Change & Deployment​

What changed recently?

These checks surface recent changes to the deployment. They detect server configuration reloads, JetStream domain changes, stream configuration modifications (replicas, retention, limits), and unexpected server restarts. Change detection is essential for correlating incidents with deployment events.

Configuring Thresholds​

Many checks have configurable thresholds that control when a finding is produced. You can override these thresholds in your configuration file without modifying any code:

check-thresholds:
  server-003:
    cpu_percent: 90.0
  stream-003:
    max_subjects: 1000000
  connection-001:
    rtt: 500ms

Each check that supports threshold configuration documents its parameters and defaults. Thresholds are specified by check code and parameter name. Values are typed by the parameter: numbers, Go durations (500ms, 15m), or IEC byte sizes (64 KiB, 1 MiB). Optimization checks also accept lookback, the length of the window they evaluate.

On a node with a systems: list, an entry can carry its own check-thresholds; its overrides for a code replace the top-level overrides for that code on that system.

A check code or parameter name that does not exist fails startup. Run insights config check to validate a configuration before restarting with it. See Configuration → check-thresholds.

Turning a Check Off​

Some checks describe a condition that is structural in a given deployment rather than a problem to fix, where no threshold makes them useful. List those check codes under disabled-checks:

disabled-checks:
  - account-012
systems:
  - id: edge-west
    disabled-checks:
      - stream-025

A systems: entry's list is added to the top-level list for that system; it cannot turn back on a check the top level disables.

A disabled check keeps running and keeps its history. What changes is how it is treated: every surface that lists it marks it disabled, the category and overall health grades leave it out, and nothing it finds is delivered: no finding feed transition, no webhook, no alert. Turning a check off is a statement that its findings are not actionable here, not that they stopped being true, so they stay visible and stop counting. See Configuration → disabled-checks.

Next Steps​