k6-compatible scripts and load tests (UPT-002)

Watchtower runs k6-API scripts on its own engine (go/loadtest, a goja JavaScript runtime; no k6 code or binary). The same engine serves two uses: the k6 monitor kind (a scripted, multi-step synthetic check) and load tests (/loadtests in the web app).

Script surface

ES module syntax (import/export), options (vus, duration, iterations, stages, scenarios, thresholds with abortOnFail and delayAbortEval, setupTimeout, teardownTimeout, rps, userAgent, insecureSkipTLSVerify, batch), setup/teardown/handleSummary, __ENV, __VU, __ITER, open() for attached files and SharedArray.

Modules: k6 (check, group, sleep, fail, randomSeed), k6/http (all methods, request/batch/asyncRequest, params with headers, tags, timeout, cookies, redirects, responseType, auth; cookie jars; expectedStatuses; setResponseCallback; multipart http.file), k6/metrics (Counter, Gauge, Rate, Trend), k6/execution, k6/secrets (the check vault), k6/timers, k6/encoding, k6/crypto, k6/data, k6/ws, k6/net/grpc (unary invoke/asyncInvoke and new grpc.Stream for client, server and bidirectional streams; definitions from server reflection, a protoset or .proto sources compiled with client.load(importPaths, ...files)), k6/html (parseHTML with a jQuery-like Selection, plus Response.html(), submitForm and clickLink), k6/browser (Chromium through chromedp) and the k6-utils jslib helpers. All seven executors run: shared-iterations, per-vu-iterations, constant-vus, ramping-vus, constant-arrival-rate, ramping-arrival-rate and externally-controlled. Built-in metrics follow k6 names (http_req_duration, http_req_failed, checks, iterations, vus, data_sent, ...); handleSummary gets the k6 summary shape.

Extensions: instead of xk6 compile-time extensions, a script imports JavaScript modules from attached files or from https URLs pinned with a Subresource Integrity hash: import x from "https://host/lib.js#sha384-<base64>" (sha256, sha384 or sha512). Pinned modules are fetched once per runner process, verified, and their relative imports must be pinned too. An unpinned import fails with the hash to pin. Not supported: k6/x/ Go extensions.

options.hosts maps a name (or *.suffix) to an IP address, optionally with a port. Values must be IP literals. An override goes through the same ownership check as a DNS answer: the name must be a verified target, and the address must be proven. Redirects are followed under the same check, and each followed hop is recorded as its own request (http_reqs, http_req_duration, ...), as in k6.

Address proof: verifying a name (DNS TXT or the HTTP file) proves who controls the name, not who owns the address it resolves to. So a load test connects to a name's address only if that address is in a verified IP or network target, or serves the name's token at /.well-known/watchtower-verify.txt when fetched from that very address with the name as Host (HTTPS, then HTTP, no redirects). A proven address is fetched again after 10 minutes, and a failed one after 1 minute, so a changed DNS answer is proven again. Private and special addresses never pass by name. This holds at every VU count. wtctl loadtest run on your own machine is exempt, because you state its scope.

Monitor kind k6

Config: script (at most 256 KiB), vus (1 to 50), duration_s (1 to 600), env (at most 50 strings). vus and duration_s override the script's options only when set. The check is up when every threshold passes, no iteration threw and the run was not aborted. Details under k6: checks_pass_rate, http_req_duration_p50_ms, _p95_ms, _p99_ms, http_req_failed, iterations, thresholds_passed, thresholds and engine: "watchtower". {{secret.NAME}} in env resolves from the check vault.

A load test whose env reads {{secret.NAME}} needs manage_credentials to create, change or delete, as a monitor does. That role may send those values to a host of its own, so the audit log records the names and allowed_hosts, never the values.

Load tests

/v1/projects/{prj}/load-tests holds tests (a script, or kind: "check" replaying an existing uptime check as the iteration), with locations, allowed_hosts and an optional cron schedule. A run is split into one shard per location; each runner claims shards, runs its segment of every scenario, and reports snapshots every 5 s. The coordinator merges the snapshots (mergeable histograms, about 1% percentile error), evaluates thresholds, aborts all shards when an abortOnFail threshold fails, and streams live gauges (loadtest.<metric>.<stat>, resource attributes loadtest.test_id, loadtest.run_id, loadtest.location) and sampled request spans into the project's telemetry. Runs can be aborted, externally-controlled runs scaled or paused, compared with a baseline and charted as a trend. Limits per run: 1000 VUs, 1 h, 5000 requests/s per runner.

Network policy

Every connection (HTTP, WebSocket, gRPC, browser) goes through the address guard: public locations refuse private ranges, metadata addresses and names that resolve to them (DNS rebinding); private locations may reach private space but never metadata addresses. Load tests may additionally only reach the hosts in allowed_hosts. Loopback is refused on public locations because the engine dials in-process.

Target ownership

A load test may only send load to targets its project proved it controls (/v1/projects/{prj}/load-targets, the Targets section of the Load tests page):

• dns: a TXT record _watchtower-challenge.<name> with watchtower-verify=<token>. A *.name target is proved on the name and covers its subdomains.

• http: the token as the body of https://<host>/.well-known/watchtower-verify.txt (or http://), redirects not followed. Host names and IP addresses.

• probe: an address or a network (at most /16, /48) that the private probe of a private:<name> location is itself in. It is valid for runs at that location only.

Public engines check dns and http targets; private engines and pulling probes check their own probe targets. Verified targets are re-checked every 24 h, and a failed re-check revokes the target. A run freezes the verified targets when it is admitted. Every request host is checked at run time: each HTTP request and redirect hop, each websocket, gRPC and browser connection. For a target verified as a network, the dialled address must also be in it, which defeats DNS rebinding. A host that is not verified is refused and the run is aborted. The exception is a run with a single VU, which may send it uptime-check traffic: one request at a time, 2 per second. The k6 monitor kind always works at that rate for every host, because it is a check and not a load test. Use a load test for load. A replayed check must name one target host (url or host). HTTP redirects are not followed in replays.

Caps, kill switch and audit

Runs are admitted by the coordinator. Each organization may run at most max_concurrent tests at once and send max_rps requests per second in total; each run gets max_rps / max_concurrent, split across its shards. The defaults are WT_LOADTEST_ORG_CONCURRENT (2) and WT_LOADTEST_ORG_RPS (2000). An operator overrides them per org or globally:

• wtctl loadtest caps [--org <id>] [--concurrent N] [--rps N] sets the caps (-1 clears a cap).

• wtctl loadtest kill [--org <id>] --reason <text> stops every run in scope within a few seconds and admits no new ones.

• wtctl loadtest resume [--org <id>] lifts the kill.

WT_LOADTEST_DISABLED=1 keeps one engine from running shards. Every run's admission (load_test.run.admit: trigger, locations, allowed hosts, verified targets, options and caps) and its end (load_test.run.finish: state, reason, request count) are audited.

Private probes

A pulling private probe (--mode=private with WT_PROBE_URL) runs the shards of its location through the edge (POST /v1/probe/load/claim and /v1/probe/load/beat). The vault values a shard needs are sealed to the probe's key. Scheduling, admission and merging stay central. Live points and spans are not streamed from a pulling probe; its snapshots still make up the run's result.

Config: WT_LOADTEST_PARALLEL (shards one runner works on at once, default 2, max 8), WT_CHROME, WT_CHROME_NO_SANDBOX for the browser module.

Revocation and operator audit

Runs freeze their targets at admission, then every shard re-reads the project's verified targets on each 5 s beat, including shards on pulling private probes. A target deleted or failing its re-check is refused from the next request, which aborts the run, and open websocket, gRPC and browser connections to it are closed. A runner out of contact with its coordinator for 30 s stops its shard.

Replayed checks verify the dialled address at connect time for targets verified as a network, as scripts do.

wtctl loadtest kill|resume|caps require --reason and record who (--by, else $WT_OPERATOR, else the OS user), when and why in load_test_operator_audit, global changes included. Read it with wtctl loadtest log.

Pulling probes send live metrics and spans in their beats. The edge removes wt.* attributes and stamps the project from the probe token, so a probe cannot write to another tenant.