# Notifications

Insights can push an alert to an external endpoint whenever a check changes state. Each epoch, every system's findings tracker diffs the current check findings against the previous epoch and delivers the transitions (a check firing, resolving, or changing severity) to any endpoint subscribed to that check code. This turns the checks into an active alerting source rather than something you have to watch in the UI.

The same transitions are published on NATS whether or not any endpoint is configured; see [Finding feed on NATS](#finding-feed-on-nats).

## Endpoint types

Endpoints are declared under two keys by the payload format they receive:

* **`webhook`**: receives the full Alertmanager-compatible payload envelope (status, alerts, common labels/annotations). Point this at a generic receiver, a relay, or your own HTTP service.
* **`alertmanager`**: receives a bare alert array compatible with the [Alertmanager v2 `POST /api/v2/alerts`](https://prometheus.io/docs/alerting/latest/clients/) API. Point this straight at an Alertmanager instance.

Both types share the same configuration fields.

## Configuration

Notifications are configured in the YAML config file only; there is no equivalent flag or environment variable. Add a `notifications` section:

```
notifications:

  webhook:

    - name: "PagerDuty Relay"        # unique across all endpoints

      url: "https://example.com/webhook"

      enabled: true                  # default true

      auth-type: "bearer"            # "", "basic", "bearer", or "hmac"

      auth-token: "…"                # token (bearer), password (basic), or secret (hmac)

      subscriptions: ["server-*", stream-002]   # code patterns this endpoint receives

      ignore: [server-046]                       # patterns dropped after a subscription match

  alertmanager:

    - name: "local-am"

      url: "http://localhost:9093/api/v2/alerts"

      subscriptions: ["*"]
```

| Field           | Default    | Notes                                                                                                   |
| --------------- | ---------- | ------------------------------------------------------------------------------------------------------- |
| `name`          | (required) | Human-readable; must be unique across every endpoint                                                    |
| `url`           | (required) | Must be `http` or `https`                                                                               |
| `enabled`       | `true`     | Set `false` to keep a definition without delivering                                                     |
| `auth-type`     | (none)     | `basic`, `bearer`, or `hmac`                                                                            |
| `auth-user`     | (empty)    | Username for `basic` auth                                                                               |
| `auth-token`    | (empty)    | Password (`basic`), token (`bearer`), or secret (`hmac`)                                                |
| `subscriptions` | (empty)    | Check-code patterns to deliver; see the Checks Reference                                                |
| `ignore`        | (empty)    | Patterns dropped after a subscription match; carves exceptions out of a wildcard                        |
| `systems`       | (empty)    | Systems to deliver for; must be systems this node owns. Empty delivers for **every** system on the node |

**Patterns.** `subscriptions` and `ignore` each take a list of check-code patterns: an exact code (`server-001`), a trailing-wildcard prefix (`server-*`), or `*` for every code. Codes are lowercase and hyphenated, and `*` is only allowed as the last character. `ignore` is applied after `subscriptions`. An empty `subscriptions` list delivers nothing: subscribing is explicit. A pattern that matches no known check code fails startup, so a typo can't silently deliver nothing; an uppercase, underscored pattern (`SERVER_001`, `SERVER_*`) fails with its current spelling in the error. An exact code of a retired check is accepted with a warning and delivers nothing. The endpoint's detail page in the UI expands each pattern into the checks it resolves to.

**Scoping to systems.** On a node serving several systems, `systems` restricts an endpoint to a subset; everything else still delivers everywhere. Note it is the inverse of `subscriptions`: an empty `subscriptions` delivers nothing (subscribing to a check is explicit), but an empty `systems` delivers for every system the node owns.

**Authentication.** `basic` sends an `Authorization: Basic` header from `auth-user`/`auth-token`; `bearer` sends `Authorization: Bearer <auth-token>`; `hmac` signs the request body with HMAC-SHA256 keyed by `auth-token` and sends `X-Hub-Signature-256: sha256=<hex digest>`.

**Delivery timing is fixed.** Each delivery uses a 10s request timeout and makes up to 3 attempts with exponential backoff; a client error (4xx other than 429) ends that attempt set at once. Redirects are not followed, so a 3xx response also ends it: set `url` to the address that accepts the POST, not one that redirects to it. A delivery whose attempts all fail is then retried as a batch, as described below.

**Delivery never blocks ingest.** Each endpoint has its own delivery worker, so transitions are posted off the scrape path and one endpoint that accepts the connection and then hangs can never stall indexing or delay any other endpoint. While a receiver is slow or down, that endpoint's pending transitions are coalesced per finding (the latest state of each finding is kept), so its backlog is bounded by the number of findings, not by how long the receiver is unavailable, and no finding's current state is lost. A receiver that recovers sees each finding's net state; intermediate flapping (a fire and resolve while it was behind) collapses to the final state rather than replaying every edge, and a fire that resolved before it was ever delivered is dropped rather than sent as a lone resolve.

A failed delivery is retried for that endpoint alone, on an exponential backoff (starting at 2s, capped at once a minute), until it succeeds. A down receiver never causes duplicate deliveries to healthy endpoints. This applies to every failure, a 4xx response included: a receiver that rejects the payload keeps receiving the same batch once a minute until it accepts it, and transitions that arrive meanwhile wait behind that batch. The endpoint's delivery log shows the rejection. Retry re-sends the coalesced batch, so delivery to a given endpoint is at-least-once: a receiver may occasionally see a transition again if the connection dropped after it had already processed the request.

**Validation is strict.** Every endpoint is checked at startup, disabled ones included. A missing `name` or `url`, a URL that is not `http`/`https` or has no host, a duplicate `name`, an unknown `auth-type`, a subscription or ignore pattern that matches no check, or a `systems` entry naming a system this node does not own fails startup with an error naming the entry rather than being silently dropped, so you can't believe alerting is on while nothing is delivered. `insights config check` runs the same validation without starting the node.

**Subscribing to a disabled check delivers nothing.** A check listed in `disabled-checks` keeps running, but its findings never enter the findings tracker, so nothing is delivered or published on the finding feed for it: not a fire, not a resolve, and nothing on the standing re-send. Turning a check off does not send a resolve for anything it had firing. The subscription is not an error; the Notifications page marks it as disabled so the silence is not mistaken for health.

**Payload links.** Alerts include a link back to the relevant entity in the UI. Set `--web.external-url` to your instance's public base URL so those links resolve for recipients. The link is not scoped to a system: on a node serving one system it opens the entity, and on a node serving several it opens the system list.

## Payload labels and annotations

Each alert carries a set of Alertmanager-style labels (for routing and grouping) and annotations (human-readable context). Labels use unprefixed, generic names so they compose with your existing Alertmanager relabeling.

| Label         | Always present          | Value                                                                                                                                                                                                                                                                                                                                                      |
| ------------- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `alertname`   | yes                     | The check code, e.g. `server-001`                                                                                                                                                                                                                                                                                                                          |
| `severity`    | yes                     | `info`, `warning`, or `critical`                                                                                                                                                                                                                                                                                                                           |
| `entity_type` | yes                     | The kind of entity the finding is about, e.g. `server`, `stream`                                                                                                                                                                                                                                                                                           |
| `source`      | yes                     | Always `insights`                                                                                                                                                                                                                                                                                                                                          |
| `system`      | yes                     | The id of the system the finding came from, the same id shown in the UI. Route or group on this to fan alerts out per system.                                                                                                                                                                                                                              |
| `scope`       | when the check is known | The check's scope                                                                                                                                                                                                                                                                                                                                          |
| `category`    | when the check is known | The check's category                                                                                                                                                                                                                                                                                                                                       |
| `entity_key`  | when non-empty          | A secondary NATS name that disambiguates the finding: service name, remote server name, or Raft group name; empty for most entity types                                                                                                                                                                                                                    |
| `entity_pk`   | when non-zero           | This instance's internal id for the entity. It keeps findings on different entities distinct. Without it, two entities that share a scope and name (e.g. the same check firing on two clusters) would produce identical label sets and collapse into one alert. Instance-scoped and not stable across restarts, so don't use it as a durable external key. |

The `system` label is also promoted to the payload's `commonLabels` and `groupLabels`, since one delivery only ever spans a single system.

| Annotation    | Present                 | Value                                   |
| ------------- | ----------------------- | --------------------------------------- |
| `entity_name` | when non-empty          | The resolved human-readable entity name |
| `summary`     | when the check is known | The check's display name                |
| `description` | when the check is known | What the check evaluates                |
| `remediation` | when non-empty          | Suggested remediation                   |

## Delivery cadence

Transitions (a check firing, resolving, or changing severity) are queued for delivery the moment an epoch commits. A severity change is delivered as a firing alert at the new severity; because `severity` is a label, Alertmanager endpoints also receive a resolve for the former severity in the same delivery, so the old alert clears instead of waiting out its `endsAt`.

Alertmanager endpoints also receive the full active set re-sent on a fixed wall-clock cadence (every minute, and once immediately at startup), each alert carrying a rolling `endsAt` four minutes ahead. Alertmanager holds an alert while it keeps arriving and auto-resolves it once Insights stops re-sending. Because the re-send runs on a clock rather than on scrape epochs, delivery does not stop when scrapes do. Webhook endpoints are edge-triggered and receive no re-send.

An alert's `startsAt` is the first epoch of the finding's current unbroken run, read from the check history, so a standing finding keeps its real start across restarts.

## The notifications page in the UI

Each system has a Notifications page in the web UI. It lists the endpoints that deliver for that system, and each endpoint's detail page shows its configuration, the checks its patterns resolve to (marking any that are disabled), and a delivery log recording the system each delivery described. The detail page's **Test** button delivers a synthetic firing alert to the endpoint and reports the result inline, so you can confirm connectivity and auth end to end without waiting for a real check to change state. Endpoints themselves are managed exclusively through the config file (see above); the page is read-only apart from sending a test.

## Verifying a delivery locally

Run a local Alertmanager as a receiver, start Insights with the config above, and trigger a test delivery:

```
docker run --rm -p 9093:9093 prom/alertmanager   # a local receiver

insights serve -c config.yaml                     # start Insights

# → Notifications UI → "local-am" → Test

#   a firing alert appears in Alertmanager at http://localhost:9093
```

## Finding feed on NATS

Every transition is also published as a core NATS message, per system, on:

```
$INS.sys.<id>.feed.finding.<status>.<severity>.<code>
```

`status` is `firing`, `resolved`, or `changed` (a severity move while the finding stays up), `severity` is `info`, `warning`, or `critical`, and `code` is the check code. The tokens are in the subject so a subscriber filters server-side:

```
nats sub '$INS.sys.prod.feed.finding.firing.critical.>'   # everything newly critical

nats sub '$INS.sys.prod.feed.finding.*.*.stream-002'       # one check, every transition
```

The JSON payload names the check and entity (`code`, `entity_type`, `entity_pk`, `entity_key`, `entity_name`), the `status` and `severity`, `prev_severity` on a `changed`, `starts_at`, and `ends_at` on a resolve. The feed is fire-and-forget: there is no reply, no queue group, and nothing is stored, so a subscriber that is not connected misses what was published meanwhile. It is published regardless of the `notifications` section, and disabled checks are left out of it. A subscriber needs only subscribe permission on `$INS.sys.<id>.feed.finding.>`. See the API reference for the subject space.

## Delivery log and NATS permissions

Endpoint configuration is held in memory, read from the config file at startup. Only the delivery log is stored in JetStream: a node-scoped stream named `webhook-deliveries_<node_id>` on subjects `$INS.node.<node_id>.webhook.delivery.<endpoint_id>`, keeping the last 200 deliveries per endpoint for up to 7 days. It is created on the sink connection when at least one endpoint is configured on a node that holds a catalog or serves the web UI. Without JetStream, delivery continues and only the history is lost; the endpoint page says the log is unavailable. Routine Alertmanager re-sends are logged only when they fail.

The stream's required grants are documented with the rest of the sink connection's permissions in the [Deployment guide](/insights/1.2/guides/deployment.md#sink-connection-permissions). This matters only against an external sink with a least-privilege user; the default embedded sink needs no grants.
