Uptime check types

Each monitor has a kind, a config object and optional assertions (the Gatus-compatible condition language, for example [CONNECTED] == true). A monitor with no assertions gets the default of its kind. Unknown config fields are rejected. The table below is the driver catalog; a test (TestDriverCatalogMatchesDoc) keeps it equal to the registered drivers.

Passwords are never config values: password_ref names a secret of the project vault (P3-U5).

Where a check may connect

Every driver connects through the dial-time address guard (go/internal/netguard). A public probe location refuses loopback, link-local, RFC 1918, CGNAT, ULA and metadata addresses, after name resolution, so a rebinding name cannot get through. Private probes (private:<name>) may reach private space; metadata, multicast and unspecified addresses stay blocked for them too. The error text never names the blocked address.

• dns never uses the probe host's own resolver. Without resolver it asks 1.1.1.1. resolver must be an IP address (with an optional port), and the connection to it is guarded like any other, so a public probe cannot query an internal resolver, and answers are returned only for the name the monitor asks about.

• icmp and network_path cannot dial. They resolve the host once, check the address with the same guard and then use that address. Hops in private space are shown as private on public probes.

• kafka brokers advertise their own addresses; those connections are guarded too.

• domain talks only to the IANA bootstrap file (https://data.iana.org/rdap/dns.json, cached 24 h) and the https RDAP service it lists for the TLD, without following redirects, reading at most 1 MiB, within 10 s. A successful lookup is cached for one hour per domain, so a short check interval does not hammer the registry.

Operating notes

• icmp uses unprivileged datagram sockets. Where the OS refuses them the result error is icmp_unprivileged_unavailable (on Linux, include the probe's group in net.ipv4.ping_group_range).

• network_path needs a raw ICMP socket (CAP_NET_RAW or root), because unprivileged sockets do not deliver Time Exceeded messages. Without it the error is network_path_needs_raw_icmp. At most 30 hops and 3 probes per hop.

• network_path stops probing a hop after its first lost probe (one 1 s wait per silent hop) and always fits inside the monitor timeout. If time runs out first, the result is up with the hops traced so far (reached is false in the result); it fails only when the hop limit is reached without an answer from the destination.

• tcp and udp with send but no expect succeed right after the send; without expect the udp check proves almost nothing, since UDP has no handshake.

• http client_cert_secret is {{secret.NAME}}, a vault secret holding the PEM certificate and key, presented for mTLS. cache_bypass sends Cache-Control: no-cache, Pragma: no-cache and a random _wt=<nonce> query parameter.

• mysql with user logs in (password from the vault). login_query runs one SELECT or SHOW statement (5 s; writes, locks, SLEEP and file reads are refused) and exposes the row count as [MYSQL_ROWS].

• snmp types its values: int and uint are numbers, string, oid, and timeticks (a duration). Read one with [SNMP:1.3.6.1.2.1.1.3.0]. Version 3 needs user, auth_protocol (sha or sha256) and auth_password_ref; priv_protocol: aes adds privacy. Public probes refuse private targets like every driver.

• Typed condition values: [DNS_RCODE], [DNS_ANSWER], [SNMP:<oid>], [SSH_BANNER], [TCP_CONNECTED], [MYSQL_ROWS], [GRPC_STATUS], [CERTIFICATE_EXPIRATION], [BODY]/[BODY].path, [RESPONSE_TIME], with ==, !=, <, <=, >, >=, pat() and any().

• A monitor may set severity, escalation_policy_id and declare_incident; they are saved on its managed alert rule (severity, and labels the escalation and incident paths read).

• tls reports the chain, the SANs, the expiry and the protocol version in the result, and reports an unverifiable certificate with its details before failing.

• dns with dnssec: true sets the AD bit in the query and fails when the resolver does not return an authenticated answer. It trusts the resolver's validation; it does not validate signatures itself.

• websocket and dns use small built-in clients, not extra libraries.

• Not built: SNMP, Kafka SASL, [DOMAIN_EXPIRATION] on http and tls monitors.

Multi-step, browser and suite checks

multistep. config.steps is a list of at most 20 {name, request, extract, assertions}. request is an http config. extract maps a variable to an assertion path ({"token": "[BODY].access_token"}); later steps write {{steps.<name>.<var>}} in any string of their request. {{secret.<NAME>}} takes a vault secret (P3-U5; until the vault exists the run fails as config_error). The API accepts a credential header (Authorization, Cookie, anything named like a key or token) only as {{secret.NAME}}. Secret values are replaced by *** in every result. A step that errors or fails its assertions (default [STATUS] < 400) stops the run.

browser. config.steps is at most 50 of goto, click, fill, press, select, wait_for (selector or ms), assert_text, assert_url (match: contains, equals, regex), assert_visible, screenshot. A selector is CSS, xpath=..., text=... or role=button[name="Sign in"]. Elements are waited for up to 5 s. The run records LCP, CLS and INP (largest interaction), console errors and failed requests (query strings are removed). The timeout is at most 120 s and the whole run, browser start included, fits in it. On failure a screenshot is stored at checks/{project}/{monitor}/{run}.png. Chromium is found through WT_CHROME; WT_CHROME_NO_SANDBOX=1 is for containers running as root. All browser traffic goes through a local proxy that applies the address guard, so a public probe reaches no private address; a response is cut at 8 MiB. wtctl checks import-playwright <file.spec.ts> turns each test() block of a Playwright file into a monitor (an object for one, an array for several) and rejects, with every file:line listed, anything it cannot translate, including test.use, test.skip, hooks and export lines.

Running browser checks. A probe runs at most WT_BROWSER_CONCURRENCY browser checks at once (default 2); a check waits for a slot inside its own timeout and fails as browser_busy if none frees up. Chromium runs with --js-flags=--max-old-space-size=512, and on Linux the probe stops a browser whose memory (with its children) passes 1 GiB (browser_memory_limit); set a container memory limit as well. Run the probe container with --init (or tini) so Chromium's child processes are reaped. WebRTC and QUIC are disabled so no UDP leaves the browser around the guard proxy. A fill value of {{secret.NAME}} is blanked in screenshots. Browser, multi-step and suite results carry a JSON details column (vitals, console errors at most 50, failed requests at most 50, steps, suite sub-results; 64 KiB at most). Failure screenshots are stored when WT_CHECK_BLOBS is set (s3://bucket with AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, AWS_ENDPOINT, or file:///directory); without it they are dropped and counted in wt_uptime_screenshots_dropped_total on the admin /metrics.

suite. config.checks lists up to 20 monitor ids of the same project, run in order (mode: sequential). The first failure stops the run and the rest are reported as skipped. A suite cannot contain a suite. A monitor of another project is reported as not found.

Sign-in: TOTP, passkeys and Kerberos

TOTP. {{totp.NAME}} in any string of an http, multistep or browser config is the current one-time code of the vault secret NAME, which holds the TOTP seed: base32 (spaces and case ignored; 6 digits, 30 s, SHA-1) or an otpauth://totp/... URI whose digits, period and algorithm (SHA1, SHA256, SHA512) are honoured. The seed stays in the vault; the code is computed when the step runs (a browser fill computes it at the moment it types), and the field is blanked in screenshots. Referencing it needs the same vault permission as {{secret.NAME}}.

Passkeys. A browser config takes passkeys (at most 5): rp_id, credential_id (base64url, as registered), private_key_ref (vault secret with the ECDSA P-256 private key, PKCS#8 as PEM or base64), optional user_handle (base64url), sign_count and discoverable. Before the first step the tab gets a virtual CTAP2 authenticator (internal transport, user presence and verification answered) holding these credentials, so navigator.credentials.get() sign-ins complete headless. Register the credential once with the relying party and store its key in the vault.

Kerberos (SPNEGO). http auth {"type": "negotiate", "username", "realm", "kdc", "password_ref", "spn"} logs in at the KDC (host[:port], default 88, over TCP through the probe's address guard, so a public probe cannot reach a private KDC), gets a ticket for spn (default HTTP/<url host>) and sends Authorization: Negotiate <token>. AES enctypes only (aes128/aes256-cts-hmac-sha1-96); RC4 and DES are refused; no keytabs or cross-realm referrals.

Visual regression, waterfalls, recorder, setup/teardown and scripts

expect_visual. Browser step {"action":"expect_visual","name":"home","threshold":0.02,"mask":["#clock"]}. The first passing run stores the screenshot as the baseline checks/{project}/{monitor}/baseline/{name}.png; later runs compare per pixel (a pixel differs when a channel is off by more than 8/255; areas of the mask selectors are ignored) and fail when the share of differing pixels is above threshold (0 to 1, default 0.02). A failed step stores {step}-current.png and {step}-diff.png next to the run's screenshots. It needs the blob store (WT_CHECK_BLOBS). Screenshots are kept 30 days: set a 30-day expiry rule on the checks/ prefix of the bucket; baselines live under baseline/ and must be excluded from it. POST /v1/projects/{p}/monitors/{id}/baselines/reset deletes all baselines of the monitor (the next run records new ones); POST .../baselines/accept?run=<run id> with {"name":"home","step":2} makes the {step}-current.png of that failed run the baseline home. Both need ManageMonitors, are audited (monitor.baselines_reset, monitor.baseline_accepted) and answer 404 for a monitor of another project. The API reads the same bucket as the probe, so set WT_ARTIFACT_STORE on the API to the location WT_CHECK_BLOBS names on the probe; without it the routes answer 503. In the UI a failed visual step offers "Accept as baseline" and "Reset all baselines".

Browser emulation and more steps. config.emulation (all fields optional) shapes the browser: device (desktop-1080p, iphone-13, pixel-7, ipad-pro-11: viewport, pixel ratio, touch and user agent), viewport (width, height, scale, mobile), user_agent, locale (also Accept-Language), timezone, geolocation (lat, lon, accuracy; permission granted), cookies (name, value, url, at most 20), headers (at most 20, sent with every request including third-party ones), network (offline, slow-3g, 3g, 4g) or a custom throttle (latency_ms, down_kbps, up_kbps), and cpu_slowdown (1 to 20). Cookie and header values may be {{secret.NAME}}. Steps added: hover, drag (selector to target, sent as mouse events, not native HTML5 drag data), scroll (dx, dy, or a selector to bring into view), upload (selector of a file input, file_name, file_b64 up to 1 MiB), switch_frame (selector of a same-origin iframe, or value: main; later steps act inside it, upload excepted), switch_tab (value: new follows the tab the previous step opened, main goes back) and evaluate (script: a JavaScript expression that must be truthy, or whose string form equals value; a Promise is awaited). A failed run stores evidence next to its screenshots as checks/{project}/{monitor}/{run}/har.json (HAR 1.2 of every step's requests, without headers or bodies) and trace.json (steps, vitals, errors); capture lists what to keep (har, trace, video; default har and trace). video adds frames.zip, the last 40 screencast frames as JPEG, not an encoded video. Evidence needs WT_CHECK_BLOBS and follows the same 30-day expiry as screenshots; the keys are in details.artifacts.

Waterfall. Every browser step records its requests (at most 200 per step, trimmed so the details stay under 64 KiB): url, method, status, start_ms, dns_ms, connect_ms, tls_ms, ttfb_ms, download_ms, bytes. The five phases add up to the request time. They are stored in the result details (steps[].waterfall).

Recorder (not in the public wtctl since DX-21: the uptime engine is server-side; the recorder returns with a server route). wtctl checks record <url> --out check.yaml opens a visible Chromium, records navigation, clicks, input and Enter/Tab/Escape, and writes the steps on Ctrl+C. Selectors prefer #id, [data-testid], [aria-label] and fall back to a minimal CSS path. Password fields are written as {{secret.PASSWORD}}, never as typed. Set WT_CHROME for a custom browser.

Setup and teardown (multistep). setup steps run before steps and fail the check when they fail. teardown steps always run last (also after a failure, with their own 10 s budget); a failing teardown is reported in teardown_error and the step records but does not change the verdict. The 20-step limit covers all three lists.

Scripts (multistep). A step may list script entries, evaluated after its assertions over {response, vars, steps} (response.status, response.body parsed as JSON when possible, response.headers). "expr" must be true; "name = expr" stores the result as {{steps.<step>.<name>}}. The language has literals, . and [] access, arithmetic, comparisons, && || ! and len lower upper trim contains startsWith endsWith matches int float string. No I/O, 1 ms budget, 1 KiB per script.

OpenAPI import (not in the public wtctl since DX-21, same reason). wtctl checks import-openapi spec.yaml --base-url https://api.example.com --auth-secret TOKEN prints a multistep monitor with one step per operation (path order, at most 20; the rest are listed on stderr), example request bodies and a [STATUS] == <2xx> assertion. --auth-secret adds Authorization: Bearer {{secret.TOKEN}}. | trace_test | Trigger an HTTP request, then assert on the resulting trace (see trace tests) | trigger, variables, wait_for_trace_s, assertions | [CONNECTED] == true |

Assertions for HTTP and the new drivers

• Phase timings: [DNS_TIME], [CONNECT_TIME], [TLS_TIME], [TTFB] (milliseconds), plus [BODY_SIZE], [REDIRECT_COUNT], [FINAL_URL] and [HTTP_VERSION].

• Body checks: [BODY] == schema({...}) (JSON Schema subset: type, required, properties, items, enum, const, bounds, pattern, allOf/anyOf/oneOf/not; no $ref), re([BODY], 'regex', group) (RE2), css([BODY], 'div > a.x'), xpath([BODY], "//a[@id='x']/@href") (path subset: /, //, names, *, [n], [@a], [@a='v'], text(), @a), [BODY_HASH:sha256] (also sha512, sha1, md5) and [BODY_HMAC] (HMAC-SHA256 with the secret named by hmac_key_ref).

• http_version is 1.1 or 2 (https only); HTTP/3 is not supported because the probe has no QUIC stack. proxy_url is accepted on private probes only. dns_server is an IP resolver; the dial to it is guarded.

• postgres with query exposes [PG_ROWS] and [PG_VALUE]; ntp exposes [NTP_OFFSET] (ms) and [NTP_STRATUM]; mqtt exposes [MQTT_RETURN_CODE]; ftp exposes [FTP_BANNER].