Notification channel catalog (P3-N5, ALT-033)

Embed
Preview of the card
<a href="https://forumulate.com/p/watchtower/docs/channels"><img src="https://forumulate.com/api/public/orgs/watchtower/spaces/docs/pages/5ef2b311-120d-4d22-a6e4-d77f8c0fd676/social-card.svg?v=3e7ae33eec8b" alt="Notification channel catalog (P3-N5, ALT-033)" width="600" height="315"></a>

Every receiver has a kind. The table below is the destinations catalog of the notifier and must equal channels.All() (TestChannelCatalogMatchesDoc). Secrets are write-only: sealed by the API (wt/receiver-secret/v1), opened by the notifier per attempt, never logged, exported or put in an error or metric label. Inaccessible receivers return 404.

All outbound calls go through the notifier's SSRF-safe dialer (private, loopback and link-local targets are refused, also after redirects and DNS rebinding), with a 10 s timeout and a 64 KiB response cap. 2xx is success; 5xx, 408 and 429 (honouring Retry-After) are retried with backoff; other 4xx are permanent. Delivery is at-least-once: every request carries a key derived from the outbox idempotency key where the provider supports one (Matrix transaction id, PagerDuty dedup_key, Opsgenie alias).

Kind

Secrets (JSON)

Notes

slack_webhook

url

Incoming webhook, mrkdwn section block.

teams_webhook

url

Adaptive Card 1.4 via Workflows or incoming webhook.

discord

url

One embed, colour by severity, mentions disabled.

mattermost

url, channel?

Incoming webhook.

google_chat

url

Space webhook.

telegram

bot_token, chat_id

Bot API sendMessage.

rocketchat

url

Incoming webhook.

zulip

site, email, api_key, stream, topic?

Stream message, basic auth.

webex

bot_token, room_id

Messages API, markdown.

matrix

homeserver, access_token, room_id

PUT .../send/m.room.message/{txn}; txn = outbox key.

email

host, port?, username?, password?, from, to?

SMTP, STARTTLS required (465 = implicit TLS). Multipart text + HTML. Digests carry List-Unsubscribe.

sms

providers (1 or 2), to?

Twilio- or Vonage-compatible, failover.

voice

providers (1 or 2), to?

TTS call; press 1 to acknowledge a page.

pagerduty

routing_key

Events API v2.

opsgenie

api_key, region?

Alerts API.

splunk_oncall

api_key, routing_key

REST alert endpoint.

ntfy

topic, server?, token?

JSON publish.

pushover

app_token, user_key

Critical = priority 1, resolved = silent.

gotify

server, app_token

Token in a header, not the URL.

apprise

server, key

Bridge: POST {server}/notify/{key}; reaches every Apprise service.

signal

server, number, recipient, token?

Self-hosted signal-cli-rest-api POST /v2/send.

whatsapp

phone_number_id, access_token, to, template?, language?

Meta Cloud API; set a template for pages outside the 24 h window.

zenduty

integration_key

Events API; entity_id = group key, resolved flush resolves.

ilert

integration_key

Events API ALERT / RESOLVE keyed by group.

squadcast

url

Incident-source webhook, trigger / resolve keyed by group.

dingding

url, msg_type?

DingTalk robot, markdown (default) or text.

wecom

url, msg_type?, mentioned_list?

WeCom group robot.

wechat

corp_id, secret, agent_id, to_user?

WeCom app API: gettoken, then message/send.

line

channel_token, to

LINE Messaging API push.

threema

gateway_id, secret, recipient_id

Threema Gateway basic mode.

sensugo

url, api_key, namespace?, entity?, check?, handler?

Sensu Go event; resolved = status 0.

alertmanager

url, basic_user?, basic_password?

Forwards alerts to an external Prometheus Alertmanager (/api/v2/alerts).

incidentio

url, token

incident.io alert source; deduplicated by group.

kafka_rest

url, topic, api_version?, cluster_id?

Kafka REST Proxy v2 or v3, one record per notification keyed by group; basic_user?, basic_password?.

sns

access_key, secret_key, region, one of topic_arn / target_arn / phone_number

Amazon SNS Publish, SigV4-signed; FIFO topics get group and dedup ids.

mqtt

broker_url, topic, qos?, retain?, username?, password?

MQTT 3.1.1 publish (QoS 0/1, mqtts for TLS) through the SSRF guard.

Registered by other tasks: webhook and escalation_policy (notifier core, O2), slack_app and teams_app (C1), jira (C3), web_push (O6), gatus_push (H2).

Self-hosted targets (ntfy, Gotify, Apprise, Matrix, Zulip, Mattermost) must be reachable on a public address: the SSRF guard refuses private ones unless the operator runs the notifier in dev mode.

Paging tools

  • PagerDuty: event_action: trigger with dedup_key = alert group key (the page id for pages). A flush with nothing firing sends resolve for the same key. Severity: critical, high -> error, warning, info.

  • Opsgenie: alias = group key, so a repeat firing deduplicates into the open alert; a resolved flush calls close?identifierType=alias. Priority P1 (critical) to P5.

  • Splunk On-Call: entity_id = group key; RECOVERY on resolve.

Tenant receivers and system receivers

Tenant sms, voice and email receivers bring their own provider accounts or SMTP server and a fixed destination (secrets.to); the API refuses to save them without (422), and the notifier never falls back to the operator's accounts for them. The per-project system receivers contact:sms, contact:voice and contact:email are the opposite: they use only the operator's accounts (environment below), send only to verified contact addresses, and ignore every field a user could have set. The name prefix contact: is reserved: the API refuses to create or rename a receiver to it (422) and to edit or delete one (409).

Email

STARTTLS is required. Plain SMTP is allowed only on a tenant receiver whose public config has both insecure_skip_tls: true and insecure_skip_tls_by, which the API sets from the caller's identity only when the caller is an OrgAdmin of the organization (anyone else gets 403; a client-supplied _by is discarded). The notifier ignores insecure_skip_tls without _by, and always on the system receiver. The system receiver takes WT_SMTP_HOST, WT_SMTP_PORT, WT_SMTP_USER, WT_SMTP_PASSWORD, WT_SMTP_FROM. Alert emails use the default N4 templates (subject, text and HTML) when rendering succeeds.

SMS and voice (R25)

providers is a list of one or two accounts: {"kind": "twilio", "account_sid", "auth_token", "from"} or {"kind": "vonage", "api_key", "api_secret", "from", "application_id", "private_key"} (the last two are the Voice API's). The first is the primary. The second is tried within the same attempt only when the first provably did not accept the request: a connect, DNS or TLS failure, a 5xx or 429 before acceptance, or a Vonage status the docs mark as retryable (throttled, internal error, quota). A permanent error (bad number, bad credentials) does not fail over. An uncertain outcome (the request was written and then timed out or the connection broke, or a 2xx could not be read) is neither retried nor failed over, since either could message the person twice: it is recorded as sent with error class uncertain in the delivery history. The system receivers take their accounts from WT_TWILIO_ACCOUNT_SID, WT_TWILIO_AUTH_TOKEN, WT_TWILIO_FROM and WT_VONAGE_API_KEY, WT_VONAGE_API_SECRET, WT_VONAGE_FROM, WT_VONAGE_APPLICATION_ID, WT_VONAGE_PRIVATE_KEY, plus WT_EDGE_PUBLIC_URL for acknowledgements.

Contact addresses are sealed by the API (wt/contact-address/v1, aad user | method | "address") and opened by the notifier (go/notifier/contacts); verification codes use wt/contact-code/v1 with purpose "code". Errors, logs and metrics never contain an address or code.

Inbound acknowledgements

Only the operator's accounts (the system receivers) can acknowledge: the edge verifies only the operator's provider secrets, so tenant accounts get no callback URL and no reply instruction.

wt-edge exposes POST /v1/provider/{kind}/inbound (kind = twilio or vonage). The provider signature is verified before anything is interpreted: Twilio X-Twilio-Signature (base64 HMAC-SHA1 over the public URL plus the form parameters sorted by name, keyed by WT_TWILIO_AUTH_TOKEN; the URL is WT_EDGE_PUBLIC_URL + path + query), Vonage a signed JWT (HS256, Authorization: Bearer, key WT_VONAGE_SIGNATURE_SECRET, payload_hash = SHA-256 of the body, iat within 5 minutes). With no secret configured every request is refused with 401. The edge also needs WT_SECRET_KEY (the same value as the notifier) to check callbacks and codes.

  • Keypad press (voice, both providers): the call's gather URL is {edge}/v1/provider/{kind}/inbound?page=&project=&user=&exp=&sig=, where sig is hex(HMAC-SHA256(key("wt/provider-callback/v1"), "page|project|user|exp")), made by the notifier and checked by the edge together with exp (10 minutes after the call was placed). The identifiers are therefore authenticated whichever provider calls back, and a captured callback cannot be pointed at another page or user. A used sig is remembered in memory until exp (shortcut: per edge replica, lost on restart; the expiry bounds the window). Digit 1 acknowledges; anything else is ignored.

  • SMS reply (Twilio only): a page SMS ends with Reply ACK <code>. The code is 8 random characters from 23456789ABCDEFGHJKMNPQRSTUVWXYZ (no ambiguous ones), made when the SMS is sent and stored in page_ack_codes only as HMAC-SHA256(key("wt/ack-code-mac/v1"), code) with a 24 h expiry. The edge accepts a code once, and only if the reply's From equals the verified contact address the page was sent to, compared as digits (E.164 without the +). A reply that does not match is ignored (200, so the provider does not retry) and does not use up the code. If the acknowledgement cannot be delivered to the API the code is released again.

  • Vonage SMS replies are signed with a sig parameter, not a JWT, and are not supported: the edge answers 501. Use the keypad, or Twilio for SMS acknowledgements.

The edge then calls the API's internal listener POST /internal/pages/{id}:ack {project_id, user_id} with X-WT-Internal-Token (WT_API_INTERNAL_URL, WT_INTERNAL_TOKEN); the API checks the user's own role in the project.

Templates (N4)

Alert flushes are rendered with the default N4 templates before the channel builds its payload: chat-like channels get a title (first line) and a Markdown body as text, email a subject, text and HTML, SMS and voice their short forms. Channels put these into their JSON with encoding/json, so label values cannot inject structure. Pages, verification codes and tests use each channel's own short text. Receivers do not reference saved templates yet.

Updated