Notifications
Insights can push an alert to an external endpoint whenever a check changes state. Each epoch, every system's findings tracker diffs the current check findings against the previous epoch and delivers the transitions (a check firing, resolving, or changing severity) to any endpoint subscribed to that check code. This turns the checks into an active alerting source rather than something you have to watch in the UI.
The same transitions are published on NATS whether or not any endpoint is configured; see Finding feed on NATS.
Endpoint types
Endpoints are declared under two keys by the payload format they receive:
webhook: receives the full Alertmanager-compatible payload envelope (status, alerts, common labels/annotations). Point this at a generic receiver, a relay, or your own HTTP service.alertmanager: receives a bare alert array compatible with the Alertmanager v2POST /api/v2/alertsAPI. Point this straight at an Alertmanager instance.
Both types share the same configuration fields.
Configuration
Notifications are configured in the YAML config file only; there is no equivalent flag or environment variable. Add a notifications section:
notifications:
webhook:
- name: "PagerDuty Relay" # unique across all endpoints
url: "https://example.com/webhook"
enabled: true # default true
auth-type: "bearer" # "", "basic", "bearer", or "hmac"
auth-token: "…" # token (bearer), password (basic), or secret (hmac)
subscriptions: ["server-*", stream-002] # code patterns this endpoint receives
ignore: [server-046] # patterns dropped after a subscription match
alertmanager:
- name: "local-am"
url: "http://localhost:9093/api/v2/alerts"
subscriptions: ["*"]
| Field | Default | Notes |
|---|---|---|
name | (required) | Human-readable; must be unique across every endpoint |
url | (required) | Must be http or https |
enabled | true | Set false to keep a definition without delivering |
auth-type | (none) | basic, bearer, or hmac |
auth-user | (empty) | Username for basic auth |
auth-token | (empty) | Password (basic), token (bearer), or secret (hmac) |
subscriptions | (empty) | Check-code patterns to deliver; see the Checks Reference |
ignore | (empty) | Patterns dropped after a subscription match; carves exceptions out of a wildcard |
systems | (empty) | Systems to deliver for; must be systems this node owns. Empty delivers for every system on the node |
Patterns. subscriptions and ignore each take a list of check-code patterns: an exact code (server-001), a trailing-wildcard prefix (server-*), or * for every code. Codes are lowercase and hyphenated, and * is only allowed as the last character. ignore is applied after subscriptions. An empty subscriptions list delivers nothing: subscribing is explicit. A pattern that matches no known check code fails startup, so a typo can't silently deliver nothing; an uppercase, underscored pattern (SERVER_001, SERVER_*) fails with its current spelling in the error. An exact code of a retired check is accepted with a warning and delivers nothing. The endpoint's detail page in the UI expands each pattern into the checks it resolves to.
Scoping to systems. On a node serving several systems, systems restricts an endpoint to a subset; everything else still delivers everywhere. Note it is the inverse of subscriptions: an empty subscriptions delivers nothing (subscribing to a check is explicit), but an empty systems delivers for every system the node owns.
Authentication. basic sends an Authorization: Basic header from auth-user/auth-token; bearer sends Authorization: Bearer <auth-token>; hmac signs the request body with HMAC-SHA256 keyed by auth-token and sends X-Hub-Signature-256: sha256=<hex digest>.
Delivery timing is fixed. Each delivery uses a 10s request timeout and makes up to 3 attempts with exponential backoff; a client error (4xx other than 429) ends that attempt set at once. A delivery whose attempts all fail is then retried as a batch, as described below.
Delivery never blocks ingest. Each endpoint has its own delivery worker, so transitions are posted off the scrape path and one endpoint that accepts the connection and then hangs can never stall indexing or delay any other endpoint. While a receiver is slow or down, that endpoint's pending transitions are coalesced per finding (the latest state of each finding is kept), so its backlog is bounded by the number of findings, not by how long the receiver is unavailable, and no finding's current state is lost. A receiver that recovers sees each finding's net state; intermediate flapping (a fire and resolve while it was behind) collapses to the final state rather than replaying every edge, and a fire that resolved before it was ever delivered is dropped rather than sent as a lone resolve.
A failed delivery is retried for that endpoint alone, on an exponential backoff (starting at 2s, capped at once a minute), until it succeeds. A down receiver never causes duplicate deliveries to healthy endpoints. This applies to every failure, a 4xx response included: a receiver that rejects the payload keeps receiving the same batch once a minute until it accepts it, and transitions that arrive meanwhile wait behind that batch. The endpoint's delivery log shows the rejection. Retry re-sends the coalesced batch, so delivery to a given endpoint is at-least-once: a receiver may occasionally see a transition again if the connection dropped after it had already processed the request.
Validation is strict. Every endpoint is checked at startup, disabled ones included. A missing name or url, a URL that is not http/https or has no host, a duplicate name, an unknown auth-type, a subscription or ignore pattern that matches no check, or a systems entry naming a system this node does not own fails startup with an error naming the entry rather than being silently dropped, so you can't believe alerting is on while nothing is delivered. insights config check runs the same validation without starting the node.
Subscribing to a disabled check delivers nothing. A check listed in disabled-checks keeps running, but its findings never enter the findings tracker, so nothing is delivered or published on the finding feed for it: not a fire, not a resolve, and nothing on the standing re-send. Turning a check off does not send a resolve for anything it had firing. The subscription is not an error; the Notifications page marks it as disabled so the silence is not mistaken for health.
Payload links. Alerts include a link back to the relevant entity in the UI. Set --web.external-url to your instance's public base URL so those links resolve for recipients. The link is not scoped to a system: on a node serving one system it opens the entity, and on a node serving several it opens the system list.
Payload labels and annotations
Each alert carries a set of Alertmanager-style labels (for routing and grouping) and annotations (human-readable context). Labels use unprefixed, generic names so they compose with your existing Alertmanager relabeling.
| Label | Always present | Value |
|---|---|---|
alertname | yes | The check code, e.g. server-001 |
severity | yes | info, warning, or critical |
entity_type | yes | The kind of entity the finding is about, e.g. server, stream |
source | yes | Always insights |
system | yes | The id of the system the finding came from, the same id shown in the UI. Route or group on this to fan alerts out per system. |
scope | when the check is known | The check's scope |
category | when the check is known | The check's category |
entity_key | when non-empty | A secondary NATS name that disambiguates the finding: service name, remote server name, or Raft group name; empty for most entity types |
entity_pk | when non-zero | This instance's internal id for the entity. It keeps findings on different entities distinct. Without it, two entities that share a scope and name (e.g. the same check firing on two clusters) would produce identical label sets and collapse into one alert. Instance-scoped and not stable across restarts, so don't use it as a durable external key. |
The system label is also promoted to the payload's commonLabels and groupLabels, since one delivery only ever spans a single system.
| Annotation | Present | Value |
|---|---|---|
entity_name | when non-empty | The resolved human-readable entity name |
summary | when the check is known | The check's display name |
description | when the check is known | What the check evaluates |
remediation | when non-empty | Suggested remediation |
Delivery cadence
Transitions (a check firing, resolving, or changing severity) are queued for delivery the moment an epoch commits. A severity change is delivered as a firing alert at the new severity; because severity is a label, Alertmanager endpoints also receive a resolve for the former severity in the same delivery, so the old alert clears instead of waiting out its endsAt.
Alertmanager endpoints also receive the full active set re-sent on a fixed wall-clock cadence (every minute, and once immediately at startup), each alert carrying a rolling endsAt four minutes ahead. Alertmanager holds an alert while it keeps arriving and auto-resolves it once Insights stops re-sending. Because the re-send runs on a clock rather than on scrape epochs, delivery does not stop when scrapes do. Webhook endpoints are edge-triggered and receive no re-send.
An alert's startsAt is the first epoch of the finding's current unbroken run, read from the check history, so a standing finding keeps its real start across restarts.
The Notifications page in the UI
Each system has a Notifications page in the web UI. It lists the endpoints that deliver for that system, and each endpoint's detail page shows its configuration, the checks its patterns resolve to (marking any that are disabled), and a delivery log recording the system each delivery described. The detail page's Test button delivers a synthetic firing alert to the endpoint and reports the result inline, so you can confirm connectivity and auth end to end without waiting for a real check to change state. Endpoints themselves are managed exclusively through the config file (see above); the page is read-only apart from sending a test.
Verifying a delivery locally
Run a local Alertmanager as a receiver, start Insights with the config above, and trigger a test delivery:
docker run --rm -p 9093:9093 prom/alertmanager # a local receiver
insights serve -c config.yaml # start Insights
# → Notifications UI → "local-am" → Test
# a firing alert appears in Alertmanager at http://localhost:9093
Finding feed on NATS
Every transition is also published as a core NATS message, per system, on:
$INS.sys.<id>.feed.finding.<status>.<severity>.<code>
status is firing, resolved, or changed (a severity move while the finding stays up), severity is info, warning, or critical, and code is the check code. The tokens are in the subject so a subscriber filters server-side:
nats sub '$INS.sys.prod.feed.finding.firing.critical.>' # everything newly critical
nats sub '$INS.sys.prod.feed.finding.*.*.stream-002' # one check, every transition
The JSON payload names the check and entity (code, entity_type, entity_pk, entity_key, entity_name), the status and severity, prev_severity on a changed, starts_at, and ends_at on a resolve. The feed is fire-and-forget: there is no reply, no queue group, and nothing is stored, so a subscriber that is not connected misses what was published meanwhile. It is published regardless of the notifications section, and disabled checks are left out of it. A subscriber needs only subscribe permission on $INS.sys.<id>.feed.finding.>. See the API reference for the subject space.
Delivery log and NATS permissions
Endpoint configuration is held in memory, read from the config file at startup. Only the delivery log is stored in JetStream: a node-scoped stream named webhook-deliveries_<node_id> on subjects $INS.node.<node_id>.webhook.delivery.<endpoint_id>, keeping the last 200 deliveries per endpoint for up to 7 days. It is created on the sink connection when at least one endpoint is configured on a node that holds a catalog or serves the web UI. Without JetStream, delivery continues and only the history is lost; the endpoint page says the log is unavailable. Routine Alertmanager re-sends are logged only when they fail.
The stream's required grants are documented with the rest of the sink connection's permissions in the Deployment guide. This matters only against an external sink with a least-privilege user; the default embedded sink needs no grants.