Notifications

Insights can send an alert whenever a check finding changes. After each epoch, Insights compares each system's findings with the previous epoch and sends the changes to every endpoint subscribed to that check code. A change is a finding that started firing, resolved, or changed severity. Nothing is sent while a finding stays the same.

The same changes are also published on NATS, whether or not you configure an endpoint. See Finding Feed on NATS.

Endpoint Types

There are two endpoint types. They take the same settings and differ only in what they send:

webhook receives the full payload (status, alerts, group labels, and so on), in the format of an Alertmanager webhook_config receiver.
alertmanager receives a list of alerts, in the format of the Alertmanager v2 POST /api/v2/alerts API.

Configuration

Endpoints can only be set in the YAML config file. There are no flags or environment variables for them. Each endpoint accepts:

Field	Description	Default
`name`	A name, unique across all endpoints	none
`url`	The `http` or `https` URL to POST to	none
`enabled`	Whether the endpoint is active	`true`
`auth-type`	`basic`, `bearer`, `hmac`, or empty for none	none
`auth-user`	Username for basic auth	none
`auth-token`	Password (basic), token (bearer), or signing secret (hmac)	none
`subscriptions`	Check code patterns this endpoint receives	none
`ignore`	Check code patterns to leave out of the subscriptions	none
`systems`	Systems this endpoint sends alerts for. They must be systems this node owns. Empty means all	none

# config.yaml
notifications:
  webhook:
    - name: 'PagerDuty Relay'
      url: 'https://example.com/webhook'
      enabled: true
      auth-type: 'bearer'
      auth-token: '…'
      subscriptions:
        - 'server-*'
        - stream-002
      ignore:
        - server-046
  alertmanager:
    - name: 'local-am'
      url: 'http://localhost:9093/api/v2/alerts'
      subscriptions:
        - '*'

Patterns. subscriptions and ignore take check code patterns: an exact code (server-001), a prefix ending in * (server-*), or * for every check. * can only be the last character. ignore is applied after subscriptions, so you can use it to make exceptions to a wildcard. An empty subscriptions list sends nothing. The code of a retired check is accepted with a warning and sends nothing. The endpoint's page in the web UI lists the checks each pattern matches. See the Checks Reference for every code.

Systems. On a node with several systems, systems limits an endpoint to some of them. Unlike subscriptions, an empty systems list means every system the node owns.

Authentication. basic sends an Authorization: Basic header built from auth-user and auth-token. bearer sends Authorization: Bearer <auth-token>. hmac signs the request body with HMAC-SHA256 using auth-token as the key and sends the signature in X-Hub-Signature-256: sha256=<hex digest>.

Validation. Insights checks every endpoint at startup, including disabled ones. It refuses to start, and names the endpoint, if any of these is wrong: a missing name or URL, a URL that isn't http or https or has no host, a duplicate name, an unknown auth-type, a pattern that matches no check, or a system this node doesn't own. An old-style code such as SERVER_001 fails with its current name in the error. insights config check runs the same checks without starting the node.

Disabled checks. A check listed in disabled-checks still runs, but none of its changes are sent or published. Disabling a check doesn't send a resolve for findings it had firing. The endpoint's page marks subscriptions to disabled checks.

Set --web.external-url to the public URL of Insights so the links in alerts work. On a node with one system, a link opens the entity. On a node with several, it opens the list of systems.

See Configuration for every key.

Delivery

Sending alerts never slows down data collection. Each endpoint has its own sender, so a slow or unreachable endpoint doesn't delay the others.

Each attempt has a 10 second timeout and is tried up to 3 times with increasing waits. A 4xx response other than 429 stops the attempt right away.
Failed deliveries are retried for that endpoint only, starting after 2 seconds and settling at once a minute, until they succeed. This includes 4xx responses: an endpoint that rejects a delivery gets the same delivery every minute until it accepts it, and newer changes wait behind it.
The backlog stays small. While an endpoint is behind, Insights keeps only the latest state of each finding. A finding that fires and resolves before it is delivered is dropped instead of sent as a lone resolve.
Delivery is at least once. A retry resends the whole batch, so an endpoint can receive a change twice if the connection dropped after it processed the request.

Timing

Changes are sent as soon as an epoch is stored. A severity change is sent as a firing alert at the new severity. Alertmanager endpoints also receive a resolve for the old severity in the same delivery, because severity is a label.

Alertmanager endpoints also receive all active alerts every minute, and once at startup. Each alert's endsAt is set four minutes ahead, so Alertmanager keeps the alert open while it keeps arriving and resolves it after Insights stops sending it. This resend runs on a timer, so it continues if scraping stops. Webhook endpoints receive only changes.

An alert's startsAt is the first epoch of the finding's current run, so a long-running finding keeps its real start time across restarts.

Labels and Annotations

Each alert has Alertmanager labels for routing and grouping, and annotations with details.

Label	Present	Value
`alertname`	always	The check code, such as `server-001`
`severity`	always	`info`, `warning`, or `critical`
`entity_type`	always	The kind of entity, such as `server` or `stream`
`source`	always	`insights`
`system`	always	The id of the system the finding came from. Also set in the payload's `commonLabels` and `groupLabels`
`scope`	when the check is known	The check's scope
`category`	when the check is known	The check's category
`entity_key`	when not empty	A second NATS name that tells findings apart, such as a service or Raft group name
`entity_pk`	when not zero	Insights' internal id for the entity. Not stable across restarts or nodes

Annotation	Present	Value
`entity_name`	when not empty	The entity's name
`summary`	when the check is known	The check's name
`description`	when the check is known	What the check looks for
`remediation`	when not empty	How to fix it

Web UI

Each system has a Notifications page that lists the endpoints sending alerts for it. An endpoint's page shows its settings, the checks its patterns match, and a delivery log. The Test button sends a test alert to that endpoint and shows the result. You can only add or change endpoints in the config file.

Try It Locally

To see a delivery without waiting for a finding, run a local Alertmanager, start Insights with the config above, and click Test on the endpoint's page:

docker run --rm -p 9093:9093 prom/alertmanager   # a local receiver
./insights serve -c config.yaml                  # start Insights
# Notifications page > "local-am" > Test

The alert appears in Alertmanager at http://localhost:9093.

Finding Feed on NATS

Every change is also published as a NATS message, for each system, on:

$INS.sys.<id>.feed.finding.<status>.<severity>.<code>

status is firing, resolved, or changed (the severity changed while the finding stayed active). severity is info, warning, or critical, and code is the check code. Because they are part of the subject, you can subscribe to only what you need:

nats sub '$INS.sys.prod.feed.finding.firing.critical.>'   # everything newly critical
nats sub '$INS.sys.prod.feed.finding.*.*.stream-002'       # every change to one check

The JSON message includes the check and entity (code, entity_type, entity_pk, entity_key, entity_name), status, severity, prev_severity for a changed, starts_at, and ends_at for a resolve. Messages aren't stored, so a subscriber that isn't connected misses them. The feed is published whether or not notifications is configured, and it leaves out disabled checks. A subscriber needs only permission to subscribe to $INS.sys.<id>.feed.finding.>. See the API reference for the other subjects.

Delivery Log

Endpoint settings come from the config file and are kept in memory. The delivery log is stored in a JetStream stream named webhook-deliveries_<node_id>, on the subjects $INS.node.<node_id>.webhook.delivery.<endpoint_id>. It keeps the last 200 deliveries per endpoint, for up to 7 days, and is created when at least one endpoint is configured. Without JetStream, alerts are still sent but no log is kept. If Insights uses an external NATS server with restricted permissions, its user needs JetStream API access to create and read this stream. The embedded NATS server needs no extra permissions.