Health endpoints for status pages

Embed
Preview of the card
<a href="https://forumulate.com/p/watchtower/docs/health-endpoints"><img src="https://forumulate.com/api/public/orgs/watchtower/spaces/docs/pages/b42c5fd9-5b5f-45b4-8907-7771295f7f60/social-card.svg?v=3e7ae33eec8b" alt="Health endpoints for status pages" width="600" height="315"></a>

Watchtower builds no status page. External tools (Gatus, Forumulate, any HTTP checker) poll a token-scoped endpoint and draw their own page.

GET /v1/health/{project}                    whole project
GET /v1/health/{project}/{env}              one environment
GET /v1/health/{project}/{env}/{service}    one service
Authorization: Bearer wth_...

{project} is the project slug, {env} the environment name. A token names exactly one target and the URL must name the same one.

Response

Content type application/health+json. HTTP 200 for up and degraded, 503 for down and unknown, so a poller that only checks the status line works.

{"status": "degraded", "reasons": ["error rate 3.4%"], "checked_at": "2026-10-29T10:00:00Z", "version": "1.4.2"}

  • reasons lists at most 20 short strings. They name only this target's own alert rules, monitors and metrics, never another service's.

  • version is the last seen service.version (empty for project and environment tokens that saw none).

  • The status is computed at most once per 10 s per token, so checked_at can be up to 10 s old. Polling faster than that is pointless; each token is also limited to 2 requests per second. The cache is per token (the policy is per token), so two tokens of one target compute separately. If the telemetry store is unreachable the answer is unknown ("telemetry unavailable"), also cached.

A missing, malformed, unknown or revoked token, and a token used for another target, all answer the same 404 application/problem+json. The endpoint never reveals which tokens or targets exist.

How the status is computed

The worst of these wins (up < degraded < unknown < down):

Fact

Result

open alert with severity at or above down_severity (default high)

down

open alert with a lower severity

degraded

linked monitor failing in every location (maintenance rounds ignored)

down

error rate (last 5 min, at least 10 requests) at or above error_rate_down (0.10)

down

error rate at or above error_rate_degraded (0.02)

degraded

p95 latency above p95_ms_degraded (service tokens default to the smallest threshold of a whole-service latency SLO with objective of at least 0.95, of this environment or of all; other tokens: not checked)

degraded

no telemetry for stale_after_s (600) or never seen

stale_as (unknown)

a maintenance window lists the token as a health target

overrides all of the above with maintenance_as (degraded or up); the other reasons stay listed

An alert belongs to an environment through its env label, else its rule's env; a rule without one counts for every environment. Service tokens take alerts whose service label names the service, or (no label) whose rule's log filter lists it, plus monitors of that service. Alerts tied to no service (for example PromQL rules) count for project and environment tokens only. Environment tokens consider the environment's alerts and monitors; project tokens the whole project.

Tokens

Managed under /v1/projects/{prj}/health-tokens (needs the credentials permission): POST creates one with an optional env_id, service (needs env_id) and policy; GET lists them; DELETE .../{token} revokes. The secret is in the create response only. Watchtower stores its SHA-256 and compares in constant time. The token never appears in logs or audit events. A project has at most 100 live tokens (revoked ones do not count); DELETE .../{token}/revoked removes the record of a revoked token (needs the project-management permission).

Gatus

endpoints:
  - name: checkout
    url: "https://api.example.com/v1/health/acme/production/checkout"
    headers: { Authorization: "Bearer ${WT_HEALTH_TOKEN}" }
    conditions: ["[STATUS] == 200", "[BODY].status == up"]

Use [STATUS] == 200 alone to treat degraded as healthy.

Updated