Status monitors

A status space watches things and shows readers whether they are up. This page covers the monitors themselves: what each kind checks, where it is checked from, the fields, and in particular expectStatus — which exists because the default rule calls a healthy endpoint down. How the page those monitors feed looks and behaves (its looks, sub-pages, incidents, subscribers and embeds) is in Status pages.

Kinds of monitor

A monitor is a component on the page. Its kind says where its state comes from, and it is fixed at creation.

A heartbeat is up while its last ping is no older than one and a half times its interval, unknown until the first ping arrives, and down after that.

Services only your network can reach

http, protocol and browser monitors run from our probes on the public internet. An internal address such as http://lc-api:8080/health on a Docker Compose network, a VPN or a private subnet cannot be reached from them, and no setting changes that. Two ways to watch one without writing a pinger:

• A Watchtower private location. Run Watchtower's probe inside that network with a wtp_ probe token, point the monitor at private:<location>, turn its badge on (POST /v1/projects/{prj}/monitors/{mon}/badge, which answers the wtb_ token once) and add a watchtower component here. The probe reads the internal URL, and this page mirrors the result.

• A heartbeat from a job that already runs inside: forumulate heartbeat ping <monitor> after it checks the service. See Pinging a heartbeat.

Pinging a heartbeat

Send the monitor's secret in a header, to an address that carries none:

curl -fsS -X POST "$PING_ENDPOINT" -H "Authorization: Bearer $PING_TOKEN"
# a scheduler that cannot set Authorization can use:  -H "X-Forumulate-Ping-Key: $PING_TOKEN"

PING_ENDPOINT is the monitor's pingEndpoint (https://forumulate.com/api/status/ping/{monitorId}) and PING_TOKEN its pingToken. The answer is always 200 {"ok": boolean}: true when the ping was recorded, false for a missing or wrong secret or an unknown monitor (one answer for all three, so neither can be probed for).

Two older forms keep working for URLs already deployed:

• POST /status/ping/{monitorId}/{token} — the secret in the path (the monitor's pingUrl). Same body and answer. Anything that records URLs records the secret, so move to the header form when you next touch the job.

• GET /status/ping/{monitorId}/{token} — only for a monitor with allowGetPing: true. A GET URL ends up in browser history, proxy logs and link previews, so a monitor created since this field exists refuses a GET that carries its right token with 405 (Allow: POST, reason: method_not_allowed) and records nothing. A wrong token or an unknown monitor gets the same {"ok": false} a POST gets, and never a word about allowGetPing. Monitors created before keep GET on until you turn it off, and on 2027-03-01 the GET form ends for every monitor, allowGetPing or not: from then on a GET with the right token answers 405 and records nothing. Until then every GET ping is answered with Deprecation: true and Sunset: Mon, 01 Mar 2027 00:00:00 GMT. forumulate plan warns about each heartbeat that still takes GET (the full text is in the CLI guide):

  = unchanged  status_monitor  status/Scheduled jobs
        warning: accepts GET pings (allowGetPing: true): a GET carries the ping secret in its URL, …

  Declare allowGetPing: false on the monitor in forumulate.yaml (or PATCH it, or use the console's "Accept GET pings" switch) once the job pings by POST, and the warning goes.

Keep the ping secret out of your tracing. An HTTP client instrumented with OpenTelemetry records the request URL as url.full (older semantic conventions: http.url), and Sentry, Datadog and most crash reporters keep outgoing requests as breadcrumbs. With the path form, every one of them stores a credential that can report your job healthy while it is dead. The header form leaves nothing in the URL; if you must use the path form, redact /status/ping/*/* and /status/restart/*/* in your tracer's URL scrubber.

Reporting a run that did not go cleanly

A ping with no body is a clean run — exactly what every ping was before this existed. A job that ran but had part of it fail can say so, instead of going silent (which reads as an outage once 1.5× the interval has passed):

curl -X POST "$PING_ENDPOINT" -H "Authorization: Bearer $PING_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"ok": false, "detail": "settle_crypto_invoices failed"}'
# or, from a cron line with no body:
curl -X POST "$PING_ENDPOINT?status=degraded&detail=queue%20lagging" \
  -H "Authorization: Bearer $PING_TOKEN"

• degraded and partial_outage still count as the job being up for the uptime percentage — it ran. They colour the component (and the headline turns degraded), and never make it look better than its history does.

• The next clean ping clears the report. Silence past the window is still an outage, whatever the last report said.

• The ping never refuses a body it cannot read: an unparseable body or an unknown status is treated as a clean run (or falls back to ok), because a job whose ping starts failing gets its alerting switched off.

• Like a missed heartbeat, a reported state does not open an incident on its own.

From the body to the page

What one ping says, what the monitor stores as lastPingState, and what the component then reads. lastPingState is null after a clean run: null is the healthy value, not "unknown".

status wins over ok when both are sent. metricThresholds can only make a row worse, never better: a metric above degradedAbove turns a clean or degraded ping into degraded, above downAbove into major_outage (a failed check), and a ping that already said something worse keeps what it said. The detail the job sent (or, when the numbers decided, the number and the limit) is stored as lastPingDetail, shown in the console only. A declared state (declaredState) overrides all of this until it is cleared.

Health from numbers

A job often knows a number rather than a verdict: how far behind it is, how long the queue is. Send the number and let the monitor decide:

curl -X POST "$PING_ENDPOINT" -H "Authorization: Bearer $PING_TOKEN" \
  -H 'content-type: application/json' -d '{"metrics": {"lagSeconds": 400}}'

The monitor declares what is too much, per metric name:

status_monitors:
  - space: status
    name: Replication
    kind: heartbeat
    intervalSeconds: 300
    metricThresholds:
      degradedAbove: { lagSeconds: 300 }
      downAbove: { lagSeconds: 3600 }

• A value above degradedAbove reads degraded; above downAbove reads down (a failed check, as status: down is). Equal to the limit is fine.

• The state is the worse of what the numbers say and what the ping said: a job that reports status: down stays down whatever its numbers are, and a number never makes a ping read better than it said.

• When the numbers decided the state, the console's detail names the number (lagSeconds 400 is above 300 (degradedAbove).); a detail the job sent wins.

• Names are letters, digits, _, . and - (up to 64). A value that is not a finite number is dropped; the ping is never refused for it.

• The status page draws each metric (up to three per row, thresholded ones first) as a small sparkline of the newest 24 pings that carried it, with the thresholds as dashed lines. It is server-rendered SVG: no script, no request to anyone else.

• metricThresholds is set on create, PATCH, or the manifest; null clears it. GET …/status/monitors returns it on every monitor.

Pinging from your job

Ping at the end of a run, not the start: a job that hangs or crashes then goes quiet, which is the signal. The command line needs no secret pasted anywhere: it reads the monitor's secret from the API with your token (which needs space.update on the status space) and sends it as a Bearer header:

forumulate heartbeat ping Reminders --org acme --space status     # a clean pass
forumulate heartbeat ping Reminders --fail --message "3 of 40 emails bounced"

The monitor is named by its name or id; --space is optional (every STATUS space is searched). It posts {"ok": true}, or {"ok": false, "detail": …} with --fail. A cron line with only curl:

# every 5 minutes: ping on success, report a failed pass otherwise
*/5 * * * * /opt/jobs/reminders.sh \
  && curl -fsS -X POST "$PING_ENDPOINT" -H "Authorization: Bearer $PING_TOKEN" \
  || curl -fsS -X POST "$PING_ENDPOINT" -H "Authorization: Bearer $PING_TOKEN" \
       -H 'content-type: application/json' -d '{"ok": false, "detail": "reminders.sh exited non-zero"}'

From a long-running Node process (18+), startHeartbeat from @forumulate/sdk is the whole loop. It never throws, gives every ping its own timeout, and when your check throws it posts {"ok": false, "detail": …} so the component goes amber with the reason instead of silently ageing into an outage:

import { startHeartbeat } from '@forumulate/sdk';

const stop = startHeartbeat({
  url: process.env.PING_ENDPOINT, // the monitor's pingEndpoint (no secret in it)
  token: process.env.PING_TOKEN, // its pingToken, from your secret store: sent as a Bearer
  everyMs: 60_000, // at most the monitor's intervalSeconds
  check: async () => {
    await db.query('select 1'); // throw = amber, with this message as the detail
    return { ok: true };
  },
});
// on shutdown: stop();

check is optional (omit it for a plain "I am alive" ping), timeoutMs bounds each ping (default 10 s), and onError hears about a ping that could not be delivered (the default is silence: the monitor itself reports the gap). The timer is unref'd, so the helper never keeps a process alive on its own. For a one-shot job (a cron run), ping once at the end instead, with forumulate heartbeat ping or the curl line above.

One monitor per job type gives you per-job reporting: a monitor is the unit that is up, degraded or down.

Who can read the secret. A heartbeat's pingToken and pingUrl (the path form, which embeds the token) are a write credential for that monitor, so reading them takes space.update on its status space — everywhere they appear. With only space.read (a viewer, or a token with the spaces:read scope), the monitor list answers pingUrl: null, pingToken: null, pingUrlRequiresScope: "spaces:manage", pingUrlRequiresCapability: "space.update" and still gives the secret-free pingEndpoint; a list that does include a secret is sent Cache-Control: no-store. The places that give it:

• forumulate heartbeat ping-url Reminders --org acme prints the URL (--json: {monitor, id, pingUrl, pingEndpoint, pingToken}).

• forumulate apply --secrets-file secrets.json writes every heartbeat's URL to the file under monitors/<space>/<name> ({"pingUrl": "…"}) on EVERY apply, unchanged monitors included, merged over what the file already holds. Stdout shows it redacted (pingUrl: <redacted: …>); the step in the manifest answer carries pingUrl for every heartbeat, in that answer only.

• forumulate apply --env-out .env.ping writes each heartbeat's endpoint and token as dotenv lines, mode 0600, keeping any other lines already in the file: FORUMULATE_PING_ENDPOINT_<NAME> and FORUMULATE_PING_TOKEN_<NAME> (Nightly export → NIGHTLY_EXPORT). The bare FORUMULATE_PING_ENDPOINT and FORUMULATE_PING_TOKEN are written only while the manifest declares exactly one heartbeat. The apply that finds a second one removes them from the file, so code that reads the bare names fails loudly instead of pinging the wrong monitor. Read the suffixed names once you have more than one. A deploy step that runs apply and then hands that file to the app's environment gets a new heartbeat to a running worker with nothing copied by hand, and a rotation reaches it on the next deploy.

• GET /orgs/{orgId}/spaces/{spaceId}/status/monitors/{monitorId}/ping-url reads just that one: {id, name, pingEndpoint, pingToken, pingUrl, pingSecretVersion, previousValidUntil, allowGetPing} (404 for a monitor that is not a heartbeat), never cached.

• GET /orgs/{orgId}/spaces/{spaceId}/status/monitors lists every monitor of a status space (the space by id; {orgId} takes the slug), each heartbeat with its pingEndpoint, lastPingAt and lastPingState, and its secret under the rule above.

Names: a token scope is area:verb (spaces:read, spaces:manage; forumulate tokens scopes lists them all), and a role capability is area.verb (space.read, space.update). Each scope grants a set of capabilities: spaces:manage includes space.update. The older one-word scopes read, write and admin still work and are broader.

Checking the read-only answer yourself. Mint a token with only the spaces:read scope and list the monitors with it:

forumulate tokens create "monitor read check" --scopes spaces:read --expiresInDays 1   # prints the token once
FORUMULATE_TOKEN=<that token> forumulate status components list --org acme --space status --json
#   → each heartbeat: "pingUrl": null, "pingToken": null,
#     "pingUrlRequiresScope": "spaces:manage", "pingUrlRequiresCapability": "space.update"
forumulate tokens revoke "monitor read check" --yes

The same token answers 403 to the single-monitor …/ping-url read and to …/ping-url/rotate. The console's Account → API tokens page mints one the same way.

A heartbeat's url is null in every read: it has no address. (Neither has a manual component.) manifest export writes kind: heartbeat and no url.

Rotating the ping secret

A secret that leaked (a log, a screenshot, a departed contractor's crontab) is replaced without deleting the monitor or its history:

forumulate heartbeat rotate Reminders --org acme --overlap 86400
# or
POST /orgs/{orgId}/spaces/{spaceId}/status/monitors/{monitorId}/ping-url/rotate
{ "overlapSeconds": 3600 }

The answer is the new {pingUrl, pingEndpoint, pingToken, pingSecretVersion, previousValidUntil}. The monitor id, its history and pingEndpoint stay the same. The previous secret keeps answering {"ok": true} — pings and restart reports, every form — until previousValidUntil, overlapSeconds from now by the database's clock (default 3600, at most 604800; 0 retires it at once), then {"ok": false}. Only the immediately previous secret is kept: rotating again inside the overlap retires the first one.

Each rotation sends the status.monitor.ping_secret_rotated webhook ({componentId, spaceId, name, pingSecretVersion, previousValidUntil, actorId}, never the secret) and lands in the audit log as status.rotatePingSecret. In the console it is the monitor row's "Rotate secret" button. In a manifest, declare pingSecretVersion one higher than the monitor's: plan shows pingSecretVersion: 1 -> 2, apply rotates and returns the new pingUrl in its answer (and --secrets-file), and a version below the monitor's is a plan error — a secret never moves back.

Reporting a restart

A container that restarts in a few seconds is invisible to a 60-second poll, so the service has to say so itself. Call POST /status/restart/{monitorId} from your boot or deploy script with the monitor's ping secret in a header. It needs no session, because a process that has just started has none; the secret is the same pingToken the heartbeat pings with (rotated together, accepted during the same overlap).

curl -X POST "https://forumulate.com/api/status/restart/$MONITOR_ID" \
  -H "Authorization: Bearer $PING_TOKEN" -H "Forumulate-Version: 2027-03-01" -H 'content-type: application/json' \
  -d '{"reason": "deploy"}'

The path form, POST /status/restart/{monitorId}/{token} (the pingUrl with /status/ping/ replaced by /status/restart/), keeps working. A deploy tool that can send a webhook needs no ping secret at all: give it an inbound address with source: coolify or deploy (see Deploys from Coolify or any pipeline).

• reason is optional, up to 120 characters, and defaults to restarted.

• The restart is recorded as a failed check at the database's clock, labelled restart: <reason>, so it shows as a dip in the uptime history beside the checks.

• It does not raise an alert or open an incident: a deploy is not an outage.

• A restart within 30 seconds of a failed check already recorded on that monitor is folded into it, so a crash loop does not turn the whole bar into dips.

• It always answers 200, so a deploy script cannot fail on it: {"ok": true} for a known monitor and a valid token (a folded restart included), and {"ok": false} for a bad token or an unknown monitor, the same answer for both so that neither can be probed for.

When a heartbeat goes stale

GET /orgs/{orgId}/integration-health keeps its status.heartbeat_age check, and it reports ok: false with stale: true for any heartbeat silent for three whole intervals: never pinged though it has existed that long, or pinged and then stopped. The detail names each monitor; the fix names the way out that needs no hand-copied value — forumulate heartbeat ping-url <monitor> --json (or GET …/monitors/{id}/ping-url) into an environment variable such as HEARTBEAT_<MONITOR>_PING_TOKEN, or forumulate heartbeat ping <monitor> at the end of the job. A never-pinged monitor younger than three intervals is still "not set up yet", and a late one (past 1.5 intervals, under three) is plain ok: false, as before.

kind cannot be changed after creation. PATCH accepts the field and ignores it. To change a kind, delete the monitor and create it again. In forumulate.yaml that is done for you: a declared kind that differs from the stored one plans with the warning kind heartbeat -> http cannot change in place: apply deletes this monitor and makes it again as http…, and apply does exactly that. The monitor's uptime history (and a heartbeat's ping URL) start over.

Protocol probes

Not everything worth watching answers HTTP: a database port, a mail server, a DNS record, a certificate about to lapse, a domain about to expire. Each of these kinds takes one object, probe, checked against the kind on create and on PATCH; a missing piece is a 422 that names it.

POST /orgs/{orgId}/spaces/{spaceId}/status/monitors
{
  "name": "Primary database",
  "kind": "tcp",
  "probe": { "host": "db.yourcompany.com", "port": 5432 }
}

• Every host must resolve to a public address, exactly as an http URL must; a private or loopback target is refused.

• The monitor's url is filled in for you as the probe's address (tcp://db.yourcompany.com:5432, dns://1.1.1.1/yourcompany.com?MX, domain:yourcompany.com), which is what the console shows.

• A PATCH with probe replaces the stored probe whole, judged against the stored kind.

• keyword, expectStatus and the other HTTP rules are stored and ignored on a probe, as they are on a heartbeat.

Browser checks: what you need to set up

browser monitors run on a browser runner that Forumulate operates. Until it is configured on Forumulate's side, creating a browser monitor answers 422 with "Browser checks need a browser runner, and this platform has none configured yet", and the console does not offer the kind. Nothing on your side needs setting. GET …/status/regions answers browserChecks: true once the runner is there.

Creating one

POST /orgs/{orgId}/spaces/{spaceId}/status/monitors
{
  "name": "API",
  "url": "https://api.yourcompany.com/health",
  "intervalSeconds": 60
}

Request headers

Watch an endpoint that wants a token:

POST /orgs/{orgId}/spaces/{spaceId}/status/monitors
{ "name": "Query API", "url": "https://api.example.com/v1/health",
  "headers": [{ "name": "Authorization", "value": "Bearer wth_…" }] }

• At most 5, sent on every check. Values are write-only, stored encrypted; GET …/monitors/{id} answers "headersSet": ["Authorization"] and never a value. On PATCH, sending a name with a blank or masked value keeps what is stored; headers: null removes them all.

• Host, Cookie, Proxy-* and the hop-by-hop headers (Connection, Keep-Alive, TE, Trailer, Transfer-Encoding, Upgrade) are a 422 naming the header: the probe's HTTP client owns them.

• In a manifest the same headers: list takes ${ENV} values. plan never prints one: it shows headers: (secret) unchanged, (secret) changed or (secret) set, decided by a hash of the declaration.

• requestHeaders (an object, {"Authorization": "…"}) is the older spelling of the same field and still works.

JSON status mapping

An http check is up or down on its status code. For an endpoint that answers 200 for "degraded" too, say which JSON field carries the state:

"json": { "path": "status",
          "map": { "up": "operational", "degraded": "degraded_performance",
                   "down": "major_outage" },
          "default": "major_outage" }

• path is dotted (data.status, items[0].ok); map has 1–10 entries whose values are operational, degraded_performance, partial_outage or major_outage.

• It is read after the status check: a failing status is down whatever the body says. A body that is not JSON, a missing path, or a value not in map is default.

• degraded_performance and partial_outage colour the component while it still counts as up (like a heartbeat's reported state); major_outage fails the check, so failureThreshold, incident automation and uptime treat it as an outage. A declaredState still overrides all of it.

• With automatic incidents on, entering degraded_performance opens a minor incident, "<name> is degraded", and partial_outage a major one, "<name> has a partial outage"; going back to operational (or on to an outage, which opens "<name> is not responding") resolves it. A return within an hour re-opens the same incident, and none opens during maintenance that affects the component.

• It costs one extra request per interval, to the same URL with the same headers.

Manifest monitor keys

monitors[] (per space) and status_monitors[] (org-wide, plus space) take the same keys as the API: an unknown key is refused by name. method and expectStatus are accepted and converge by PATCH, never by recreating.

plan never prints a change whose two sides are equal: a field that reads back the same as declared (absent and null count as the same) is not a change, so two applies in a row plan nothing for the monitor.

A ${ENV} that is not set in the environment makes the CLI stop with the variable's name. One that is set but empty becomes an empty string, so a header declared with it is a 422 (a header needs a value) rather than one sent blank.

Pausing a monitor

paused: true is for a component you are working on, not removing. The row and its history stay on the page, no new checks are taken, no alert is sent, and the component reads unknown (or its declaredState) until you set it back to false. alertsEnabled: false is the other thing: it only mutes the alerts while the checks carry on.

The order on the page

Components are drawn in the order you give them:

PUT /orgs/{orgId}/spaces/{spaceId}/status/monitors/order
{ "ids": ["<api-id>", "<web-id>"] }

Components you leave out keep their order after the ones you list, so moving two components can send just those two. An id from another page is a 422.

Update with PATCH /orgs/{orgId}/spaces/{spaceId}/status/monitors/{monitorId}. Send only the fields you are changing; sending a field with the value it already has is a 200 that changes nothing. There is no PUT on a monitor. (PUT …/status/settings is a different thing — the space's status settings.)

expectStatus

What "up" means without it

A monitor is up when both of these hold:

the request connected
and the status code is under 400

Both, and in that order. A request that never connected reports a status of 0, which satisfies "under 400" — so the status condition on its own calls a server that cannot be resolved healthy, which is the one thing a status page exists to notice.

What "up" means with it

the request connected
and the status code is exactly the number you set

The under-400 rule is gone. A monitor with expectStatus: 401 is down on a 200.

Why you would want that

An unauthenticated probe of an API that requires a token answers 401 when it is perfectly healthy. Some answer 405 to a GET they only accept as POST. The under-400 rule called both permanently down, which made the monitor useless and the board a liar.

PATCH /orgs/{orgId}/spaces/{spaceId}/status/monitors/{monitorId}
  { "expectStatus": 401 }

Clearing it

PATCH …/status/monitors/{monitorId}
  { "expectStatus": null }

null is a real value here, and it is the one that restores the default under-400 rule.

Omitting the field does not clear it. An absent key means "leave this alone", so a PATCH that does not mention expectStatus keeps whatever is stored. The two are different on purpose, and they are the thing to get right when a client builds the patch body from a form.

An empty string is a validation error, not a clear. Send null.

What it accepts

A single integer between 100 and 599, or null. Nothing else:

Out of range is refused rather than clamped, because clamping 50 up to 100 would silently decide that only a 100 Continue is healthy.

It is http-only

expectStatus on a heartbeat monitor is accepted, stored, and silently ignored — not refused. A heartbeat has no response to read a status from; its verdict is purely the age of the last ping. keyword and certExpiryDays behave the same way for the same reason.

If you set it on a heartbeat and the board does not change, that is why.

Combining conditions

keyword and certExpiryDays are additional conditions, ANDed on top. So a monitor with expectStatus: 200 and keyword: "ok" is up only when it connected, returned exactly 200, and the body contained ok.

Probing from several regions

A failure seen from one place is often that place: a routing problem between one data centre and your host, not your host. So a component can be checked from several regions, and only read as down once enough of them agree.

GET /orgs/{orgId}/spaces/{spaceId}/status/regions
# { "regions": [ { "key": "primary", "label": "Primary", "live": true, "primary": true },
#                { "key": "eu", "label": "Europe", "live": true, "primary": false } ],
#   "browserChecks": false }

PATCH /orgs/{orgId}/spaces/{spaceId}/status/monitors/{monitorId}
{ "regions": ["primary", "eu"], "confirmRegions": 2 }

• regions lists the region keys it is probed from. Empty (the default) is the primary checker alone. A list that leaves out the primary's key is checked only from the regions it names.

• An unknown key is a 422 that names it.

• The rule: the component is down when at least confirmRegions of the regions that have reported each see it failing failureThreshold times in a row. confirmRegions is capped at the number of regions that have reported, so a component probed from one live region is judged by it alone.

• Alerts are sent once, not once per region.

• Every check is stored with the region it ran from.

What readers see: on a page with two or more regions, each component row carries a small dot per region, and the component's own page has a "Checked from" table with each region's latest result (the regions part of theme.status.subpages.component). On a page with one region, neither is drawn.

In the studio: Checks and alerts → Checks, open a check, then Probe from and Regions that must agree it is down.

What you need to set up

Probe regions are run by Forumulate, not by you: each region is a checker host that Forumulate operates and declares on its side (STATUS_PROBE_REGIONS). Until at least one region besides the primary is declared and answering:

• GET …/status/regions lists only the primary (or lists a declared region with live: false while it is not answering yet),

• the console does not show the region pickers at all,

• readers see no region dots and no "Checked from" table,

• and every component is checked from the primary checker alone, exactly as before regions existed.

There is nothing to configure on your side beyond picking regions once they are offered.

Vendor components

A page can show the services you depend on next to your own. A vendor component reads the vendor's public status and puts it in your page's words:

Watchtower monitors

A watchtower component mirrors one monitor in Watchtower (or any self-hosted Watchtower). It is read every intervalSeconds from the monitor's public status badge, so no Watchtower API key is stored here: turn the badge on in Watchtower (POST /v1/projects/{prj}/monitors/{mon}/badge) and paste its wtb_ token.

{
  "name": "Checkout",
  "kind": "watchtower",
  "intervalSeconds": 60,
  "watchtower": {
    "project": "prj_01jabcdefghjkmnpqrstvwxyz0",
    "monitor": "mon_01jabcdefghjkmnpqrstvwxyz1",
    "badge": "wtb_…",
    "baseUrl": "https://watchtowerops.net"
  }
}

baseUrl defaults to Watchtower's public host; a self-hosted install sets its own. The same block goes in a manifest's status_monitors (or a space's monitors), and plan --check compares the block, never the derived url. For incidents as well as state, add a watchtower inbound address too (below): an alert naming the monitor opens on this component.

A vendor's maintenance windows (from the kinds that publish them) can be shown on your page with the partners band. See Status pages.

Opening incidents from outside

Your alerting tool already knows when something breaks. An inbound address lets it open (and resolve) an incident on the page:

POST /orgs/{orgId}/spaces/{spaceId}/status/inbound
{ "source": "grafana", "monitorId": "<component id>", "label": "Grafana prod" }

The answer carries the url to paste into the tool as a webhook (or, for email, the address to mail), plus endpoint and token for the forms below.

Signed vendor webhooks. Watchtower and Hermetica sign every webhook they send (X-Watchtower-Signature / Hermetica-Signature, both t=<unix>,v1=<hex HMAC-SHA256(secret, "t.body")>). Create the address with "signingSecret": "<the endpoint's secret>" and every call must carry a valid signature with a timestamp within five minutes, or it answers 401 with the reason (missing header, or wrong secret/changed body/stale timestamp) and changes nothing. Without a secret the token in the URL is the only credential, as for every other source, and unsigned calls are accepted. The secret is stored encrypted and never answered; the address says signed: true. To change it, create a new address and delete the old one.

generic takes a mapping of dotted paths into the JSON your tool sends:

monitorId names the component the incident affects; unset, it names none.

Keeping the token out of URLs

The url carries the hook's token in its path, which every proxy and log that keeps URLs keeps too. A sender that can set a header or sign its body should use the hook's endpoint (/public/status-hooks/{hookId}, no secret) instead:

On the endpoint, a missing or wrong credential is 401. POST …/status/inbound/{hookId}/rotate with { "overlapSeconds": 3600 } issues a new token (and a new url and email address); the endpoint and the hook's id stay. The previous token keeps working in every form until previousValidUntil (the database's clock, default an hour, at most a week). It sends status.inbound.rotated, never with either token.

Deploys from Coolify or any pipeline

A deploy restarts the service, and a 60-second poll cannot see a restart. An inbound address with source: coolify or source: deploy turns each deploy notice into a restart window on its component — the same dip in the uptime history the service's own restart report writes — without putting the monitor's ping secret into the deploy pipeline:

POST /orgs/{orgId}/spaces/{spaceId}/status/inbound
{ "source": "coolify", "monitorId": "<component id>", "restartWindowSeconds": 120 }

• monitorId is required: the window is recorded on that component.

• Any POST counts as "a deploy happened". The body only names it: Coolify's notification (event, application_name, message), or for deploy any JSON with service / app / name and event / status. A body that cannot be read still records the window.

• restartWindowSeconds (default 120, 10–3600) is how long one deploy is: a "started" and a "finished" notice inside it are one restart, not two dips.

• It opens no incident and raises no alert: a deploy is not an outage.

• In Coolify, add a Webhook notification with the hook's url (Coolify sends a URL only). A pipeline that can set headers uses the endpoint with Authorization: Bearer <token>.

Metrics

A page can chart numbers beside the components: response time from our own checks, and custom metrics you create (PUT …/status/metrics/{key}). A custom metric's points come from one of two places.

You push them, one at a time or up to 500 at once:

POST /orgs/{orgId}/spaces/{spaceId}/status/metrics/api.p95/points
{ "value": 212 }

A point more than 5 minutes ahead of the server's clock or more than 30 days behind it is refused and counted in rejected.

We pull them from a provider you already use, every everySeconds (default 300, 60–86400):

PUT /orgs/{orgId}/spaces/{spaceId}/status/metrics/api.p95/source
{
  "source": { "provider": "datadog", "query": "avg:trace.http.request.duration{service:api}", "site": "datadoghq.eu" },
  "secret": { "apiKey": "…", "appKey": "…" }
}

The secret is stored encrypted and never returned: the answer says only which names are set. "source": null stops pulling and keeps the points. In the studio: Checks and alerts → Metrics and embeds, then Pull from.

What you need to set up

A pulled metric needs your own account with the provider and the keys in the table above, entered in the studio or with the call above. Until they are, the metric is charted from whatever you push, and an empty metric draws an empty chart.

What a metric says about itself (PUT …/status/metrics/{key}): name, unit, description, decimals (0–4), aggregate (avg, sum, max, min, last), visible, sortKey, component (the component whose row chart draws it when it has no measured response time), and the chart's axisMin, axisMax and a dashed threshold line.

What a reader sees

GET /public/orgs/{orgId}/spaces/{spaceId}/status

uptime is null, not 0, when nothing has been recorded yet. A percentage is published only once there is enough to stand on: at least 20 checks spanning at least 6 hours (window.minSamples / window.minSpanHours; both halves — 25 checks taken in the first hour still read "Not enough data yet"). Until then window.sufficient is false and the page says so, with the bar. history is 90 day-buckets, oldest first, each true, false or null; a day is true only when it had zero failures.

A reader never sees expectStatus. It is on the authenticated monitor routes and on GET …/status/board, and nowhere public. Only visible: true monitors appear at all.

Other public endpoints:

A status space that is not PUBLIC answers all of these with a flat 404 unless the request carries a valid rotating share token.

SMS alerts for readers

What you need to set up

Readers can subscribe to a status page by email, Slack, Discord, Microsoft Teams, Google Chat, Telegram or webhook (see Status pages). SMS is offered as well once your organization connects its own Twilio account. There is no shared sending number: texts are sent from yours and billed by Twilio to you. Until the account is connected, the page offers only the other channels. All three routes need org.manage_integrations.

PUT /orgs/{orgId}/sms
{
  "provider": "twilio",
  "accountSid": "ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "authToken": "your-twilio-auth-token",
  "fromNumber": "+14155550123"
}

• accountSid must start with AC and be 34 characters.

• authToken is 16 to 128 characters, write-only and stored encrypted. Omit it on an update to keep the stored one.

• fromNumber is E.164, with the country code.

• Unknown keys are refused.

GET /orgs/{orgId}/sms answers { configured, provider, accountSid, fromNumber, updatedAt }, never the token. DELETE /orgs/{orgId}/sms disconnects it and the page stops offering SMS.

A reader's number is verified before anything is sent: the page texts a six-digit code and delivers nothing until it is typed back, so nobody can enrol a number that is not theirs. A number that replies STOP is removed from the subscription instead of being retried.

Monitoring Forumulate itself

The Forumulate API has three unauthenticated health endpoints. Point an http monitor (yours or any other tool's) at the one that answers your question.

• GET /health/live answers {"status": "ok"} as soon as the process accepts connections and never touches the database or Redis. It tells you the process is up, not that it can serve you.

• GET /health/ready checks that Postgres and Redis both answer and that the process is not saturated (event-loop lag under one second at p99). It answers {"status": "ok", "db": true, "queue": true} with 200, or "degraded" with 503 when any of those fails. This is the one with a status-code contract.

• GET /health checks only the database and always answers 200, with {"status": "ok" | "degraded", "db": boolean}. Read the status field; the code stays 200 so a brief blip is not reported as an outage.

In the API

Every operation below is also reachable from the other surfaces without a dedicated command. forumulate api <Operation> calls it from the CLI, the JavaScript SDK has a typed method for each one (the last column names it; the Python and Go SDKs are generated from the same spec), and an agent reaches the organization-scoped ones through the MCP agent server — its api_* family tools, or call_api. Where a CLI command or a hand-made MCP tool exists too, the last column names it. Operation names link to the API reference.

Org Sms

Public Status

Public Status Access

Public Status Alerts

Public Status Embeds

Public Status Incidents

Public Statuspage

Public Status Search

Public Status Siblings

Status

Status Authoring

Status Events Ingest

Status Heartbeat

Status Integrations

Status Metrics

Status Slack

Status Subscribers

Status Templates