Checks

Insights runs more than 140 checks against your NATS deployment. When a check finds a problem, it reports a finding with a severity and steps to fix it.

The Checks Reference lists every check with its code, severity, thresholds, and remediation.

There are two kinds of checks:

• Operational checks look at the state of the system at one epoch.
• Optimization checks look at a window of recent epochs, six hours by default, to find waste and imbalance.

How Checks Work

Each time an epoch is stored, Insights runs every check against that system's data, using that system's thresholds. A check looks at one kind of entity (servers, streams, consumers, accounts, connections, leaf nodes, users, services, or the whole system) and reports a finding for each entity with a problem. Entities without a problem produce nothing. Findings are stored in the hx.check_findings table.

Each finding has:

• Severity. How urgent it is.
• Entity. The server, stream, consumer, or other entity involved.
• Description. What the check found.
• Remediation. How to investigate and fix it.

Findings appear on entity pages and in entity tables in the web UI. The System page grades each category by the share of entities that pass its checks. To run one check over a time range you choose, use insights checks findings <code>.

Check Codes

Each check has a code made of the entity type and a number, such as server-003 (High CPU Usage) or stream-002 (Stream Replica Lag). The entity types are server, stream, consumer, account, connection, leafnode, user, service, and system.

Codes don't change, and a retired code is never reused, so the numbering has gaps. Use the same code everywhere you name a check: threshold overrides, disabled checks, notification subscriptions, the finding feed subject, the Alertmanager alertname, and the code column of hx.check_findings. insights checks list lists every check, and insights checks info <code> shows one check's parameters.

Finding Lifecycle

A finding is identified by its check code and entity. After each epoch, Insights compares the new findings with the previous epoch's and reports each change:

A change in severity is reported as changed, not as a resolve followed by a new finding. Every change includes starts_at, the first epoch of the finding's current run. It is computed from stored history, so it survives a restart. A resolve also includes ends_at.

Changes are published on $INS.sys.<id>.feed.finding.<status>.<severity>.<code>, so you can subscribe to just the ones you want:

nats sub '$INS.sys.prod.feed.finding.firing.critical.>'
nats sub '$INS.sys.prod.feed.finding.*.*.stream-002'

The same changes drive webhook and Alertmanager notifications. See Notifications.

Severity Levels

Most checks always report the same severity. A few set it from what they measured. For example, server-063 (Meta Snapshot Slow) is a warning or critical depending on how long the snapshot took.

Check Categories

Every check belongs to one of six categories.

Health & Availability

Is the system up and reachable?

Offline stream and consumer replicas, lost quorum, leaderless Raft groups, dropped routes and gateways, JetStream that isn't ready or can't write to storage, servers that stopped answering monitoring requests, and services that are down. Also underused servers and inactive accounts and streams that may be left over.

Performance & Latency

Is the system fast enough?

High CPU, slow consumers, high round-trip times on routes, gateways, leaf nodes, and client connections, JetStream API and meta pressure, slow meta snapshots, Raft apply lag, pending messages building up, and stream leaders placed far from their consumers.

Error & Failure Patterns

What is failing, and how?

Connection and consumer churn, leader flapping, slow consumer evictions, high redelivery, full ack pending, JetStream API errors, and subscription churn. Also security concerns such as bearer tokens and too many connections per user.

Resource Saturation

Are resources running out?

JetStream memory and storage near their limits, connection counts near their maximum, unusual memory use, too many replicated assets for Raft to handle well, streams and accounts near their limits, and leaders, replicas, connections, storage, or subscriptions spread unevenly across a cluster.

Data & State Consistency

Is the configuration correct and consistent?

Stream, consumer, mirror, and peer replica lag, JetStream assets the cluster rejected or lost, mixed server and service versions, whitespace in cluster or leaf node names, gateway configuration mismatches, even-sized meta clusters, unused imports and exports, imports without subscribers, over-replicated idle streams, unused JetStream reservations, streams without limits, disabled compression, invalid metadata, and accounts with unlimited JetStream.

Change & Deployment

What changed recently?

Config reloads, JetStream domain changes, stream configuration changes, and unexpected restarts. Use these to connect an incident to a change.

Configuring Thresholds

Many checks have thresholds that decide when they report a finding. Override them in the config file by check code and parameter name:

check-thresholds:
  server-003:
    cpu_percent: 90.0
  stream-003:
    max_subjects: 1000000
  connection-001:
    rtt: 500ms

Values can be numbers, durations (500ms, 15m), or byte sizes in binary units (64 KiB, 1 MiB). Optimization checks also accept lookback, the length of the window they look at. In a systems: list, an entry can set its own check-thresholds, which replace the top-level ones for that check on that system.

An unknown check code or parameter name stops Insights from starting. Run insights config check to validate a config file before you restart. See Configuration.

Turning a Check Off

Some checks report something that is normal for your deployment, and no threshold makes them useful. List those under disabled-checks:

disabled-checks:
  - account-012
systems:
  - id: edge-west
    disabled-checks:
      - stream-025

A systems: entry adds to the top-level list for that system. It can't turn back on a check that the top level turns off.

A disabled check still runs and keeps its history. It is marked disabled wherever it is listed, it doesn't count toward grades, and its findings aren't sent anywhere: no feed message, no webhook, and no alert.

Next Steps

• Quick Start. See checks in action with the simulator.
• Architecture. Where checks run.
• Notifications. Send findings to webhooks and Alertmanager.