Alert intake from other tools

Send alerts from tools you already run into Watchtower. They become alerts with source = external:<kind>, are deduplicated per project, source and key, and go through the same routing tree, muting, inhibition and notification path as rule alerts.

Credential

Create an ingestion key with scope ingest:alerts. The project comes from the key, never from the request. A key without the scope gets 403; a missing or unknown key gets 401. Each alert counts against the key's 6,000 a minute (all or nothing per request); bodies are limited to 1 MiB and a request to 1,000 alerts. At most 2 intake requests per key and 2 in the whole edge write at once; more get 429. Dedup keys over 512 bytes are replaced by their SHA-256. Labels starting with wt. or __ are dropped, and a sender's rule_id, owner_team and env are kept as external_<name>.

Routes (on wt-edge)

• Alertmanager and Grafana: one alert per entry in alerts; status firing or resolved; the fingerprint is the dedup key; the severity label sets the severity.

• PagerDuty: event_action trigger, acknowledge or resolve on one dedup_key is one alert with three events. Answers 202 {status, message, dedup_key}.

• Opsgenie: alias (else message) is the dedup key; priority P1 critical, P2 high, P3 warning, P4 and P5 info.

• CloudWatch: ALARM triggers, OK resolves. A SubscriptionConfirmation is confirmed only when SubscribeURL is https and its host matches ^sns\.[a-z0-9-]+\.amazonaws.com$; it is fetched without redirects, within 5 s. Every message's SNS signature is verified first (WT_SNS_VERIFY=off turns that off).

• Gatus: use a custom alerting provider with this body template: {"endpoint_name":"[ENDPOINT_NAME]","endpoint_group":"[ENDPOINT_GROUP]","endpoint_url":"[ENDPOINT_URL]","alert_description":"[ALERT_DESCRIPTION]","status":"[ALERT_TRIGGERED_OR_RESOLVED]"}. Leave [RESULT_ERRORS] out: Gatus does not JSON-escape placeholders, so quotes in it would break the body. Set send-on-resolved: true.

Generic webhook

Create a source with POST /v1/projects/{prj}/alert-sources ({"kind": "generic", "name", "mapping"}; the response's url is where to post). The mapping: A value starting with $ is a path ($, .key, ['key'], [index]; at most 16 segments), anything else is a literal:

{"title": "$.check_name", "dedup_key": "$.check_id", "severity": "$.sev",
 "status": "$.state", "resolved_values": ["ok"], "labels": {"team": "payments"}}

Or pick a preset: {"preset": "checkly"}, also betterstack, uptimerobot (JSON body), datadog and sentry. If cred_id is set only that key may post to the source.

Email

POST /v1/projects/{prj}/alert-sources with {"kind": "email", "name": "nagios"} returns an address like alerts+<40 hex>@<domain>, shown once (only its SHA-256 is stored; delete and recreate the source to rotate it). Mail sent there becomes one alert per subject:

• the subject without Re:/Fwd: and leading [...] tags is the title and dedup key, so a repeated subject updates the same alert;

• a subject tagged [RESOLVED], [OK], Recovered:, Up: (any case) resolves it;

• extract parse rules ({"extract": {"host": "Host: (\\S+)", "severity": "Severity: (\\w+)"}}, up to 10, first capture group) fill $.fields.<name>; fields.severity sets the severity. A custom mapping over the document {subject, subject_key, status, from, to, text, fields} may replace the default ({"preset": "email"}).

Operator config: WT_INBOUND_EMAIL_DOMAIN on the API (the address domain; email sources are refused without it) and WT_INBOUND_EMAIL_TOKEN (16+ characters) on the edge. Point the domain's MX at an inbound mail provider that posts parsed messages as JSON (to/To, from/From, subject/Subject, text/TextBody; Postmark and similar shapes work) to POST /v1/intake/email with Authorization: Bearer <WT_INBOUND_EMAIL_TOKEN>. Without the token the route is 404. Each address has its own rate limit.

Authentication: a post creates an alert only with the operator's bearer secret and a valid address token (160 bits, stored hashed). {"senders": ["@example.com", "ops@other.org"]} on a source also limits who may post (403 otherwise); this trusts the provider's from, so have it reject mail failing SPF/DKIM/DMARC.

Mailgun, SendGrid and SES have built-in routes that check the provider's own signature. A failed signature creates no alert. Each route is 404 until its setting is present:

For SES, create a receipt rule with an SNS action using UTF-8 encoding and subscribe the route to the topic; the subscription confirms itself once its signature checks out. On these routes senders is enforced on the provider's DKIM/DMARC verdict where it reports one, not on the from header alone. Plain-text and simple MIME bodies are read; HTML-only mail keeps an empty body and non-UTF-8 charsets are not converted.

Anything else needs an adapter (a small proxy or function) that verifies the provider and re-posts here:

Buttons on a Mattermost channel

With WT_EDGE_PUBLIC_URL set, a firing alert posted to a tenant Mattermost receiver has Acknowledge, Resolve and Silence 1 h buttons.

Mattermost does not sign button posts, so Watchtower cannot trust which user it says clicked. The buttons authorize with a token Watchtower signed: scoped to that one alert or label set and that action, valid for an hour, usable once. Anyone who can read the channel can use them. Do not post to a channel whose members should not be able to act on the alert.

The action is recorded as Mattermost channel <name>, unverified user <name>, in the alert's history and the audit log. Both names are what Mattermost sent; neither is a Watchtower account. If the project requires a resolution note, Resolve answers with a link to the console instead of resolving.