NATS Server Monitoring

Insights automatically collects and indexes data from NATS server monitoring endpoints, providing historical analysis and real-time visibility into your infrastructure.

Monitoring Endpoints

NATS servers expose their monitoring data both as HTTP endpoints and as requests on the system account. Insights uses the system account requests, so it needs no HTTP access to the servers. Each epoch starts by discovering servers with $SYS.REQ.SERVER.PING, then requests every endpoint below. A request addresses all servers at once ($SYS.REQ.SERVER.PING.<ENDPOINT>) or one server by id ($SYS.REQ.SERVER.<id>.<ENDPOINT>).

Endpoint	System request	HTTP equivalent	Purpose
varz	$SYS.REQ.SERVER.PING.VARZ	/varz	Server-wide variables and statistics
connz	$SYS.REQ.SERVER.PING.CONNZ	/connz	Client connection details
routez	$SYS.REQ.SERVER.PING.ROUTEZ	/routez	Cluster route connections
gatewayz	$SYS.REQ.SERVER.PING.GATEWAYZ	/gatewayz	Super-cluster gateway information
leafz	$SYS.REQ.SERVER.PING.LEAFZ	/leafz	Leaf node connections
subsz	$SYS.REQ.SERVER.PING.SUBSZ	/subsz	Subscription statistics
accountz	$SYS.REQ.SERVER.PING.ACCOUNTZ	/accountz	Account information
accstatz	$SYS.REQ.ACCOUNT.PING.STATZ	/accstatz	Per-account statistics summary
jsz	$SYS.REQ.SERVER.PING.JSZ	/jsz	JetStream information
healthz	$SYS.REQ.SERVER.PING.HEALTHZ	/healthz	Health check endpoint
ipqueuesz	$SYS.REQ.SERVER.PING.IPQUEUESZ	/ipqueuesz	Internal IP queue status
raftz	$SYS.REQ.SERVER.PING.RAFTZ	/raftz	Raft group status

The discovery reply is the server's statsz. A server whose varz reply is missed but which still answers discovery gets a stats row built from that reply, marked statsz_fallback in hx.server_stats, and readers treat it as not having reported. Each epoch also records how many servers it expected and how many it reached on hx.epochs.

What Insights Collects

For each epoch (scrape interval), Insights captures a snapshot from every monitored server across all endpoints. This data is indexed into DuckDB, enabling:

• Time-series analysis — Track metrics like CPU, memory, connections, and throughput over time
• Cross-server correlation — Compare metrics across servers in a cluster
• Per-account breakdown — View resource usage by account in multi-tenant deployments
• JetStream health — Monitor streams, consumers, replication lag, and API errors
• Cluster topology — Track routes, gateways, and leaf node connectivity

Key Metrics

Server Health

Metric	Source	Description
CPU Usage	varz	CPU utilization (stacked by core count)
Memory	varz	Resident memory allocation
Connections	varz	Current client connections vs configured maximum
Slow Consumers	varz	Clients unable to keep up with message delivery
Stale Connections	varz	Connections detected as stale

JetStream

Metric	Source	Description
Memory/Storage Usage	jsz	Current usage vs reserved limits
API Error Rate	jsz	JetStream API errors relative to total calls
Stream Count	jsz	Number of streams and consumer count
HA Assets	jsz	Replicated assets count per server
Replication Status	raftz	Raft group health and catching-up state

Cluster

Metric	Source	Description
Route RTT	routez	Round-trip time between cluster members
Leaf RTT	leafz	Latency to leaf node connections
Route Count	varz	Connected routes vs expected topology
Gateway Status	gatewayz	Super-cluster gateway connectivity

Checks

Insights runs over 140 automated checks against the collected data on every epoch — operational checks against that epoch's state, optimization checks across a trailing window — identifying health issues, performance problems, and configuration concerns. When a finding changes state, the transition is published on $INS.sys.<id>.feed.finding.> and can be delivered to webhook or Alertmanager endpoints (see Configuration → notifications). See Checks for details.

Further Reading

• NATS Server Monitoring documentation
• Configuration Reference for scraper settings
• Deployment Guide for topology options