Dynamic Probes

Embed
Preview of the card
<a href="https://forumulate.com/p/watchtower/docs/dynamic-probes"><img src="https://forumulate.com/api/public/orgs/watchtower/spaces/docs/pages/aa1a60f0-e31c-4845-9b34-b1791f3e8510/social-card.svg?v=3e7ae33eec8b" alt="Dynamic Probes" width="600" height="315"></a>

<!-- status: beta language: probes -->

Dynamic probes

Add a log line, a snapshot or a metric to running code without redeploying, from an issue's stack frame ("Add probe here") or the API. Points are compiled into the service; nothing patches running code. Runtime agents for JVM, .NET, Python and Node are not built (G6-06).

Governance

  • Off by default: PUT /v1/projects/{prj}/probe-settings {"enabled": true} (project admin).

  • Create: POST /v1/projects/{prj}/probes with service, env, kind (log, snapshot, metric), location ({file, line} or {function}), condition, message, capture (locals_depth up to 3, max_bytes up to 8192), max_rate_per_min up to 60 and ttl_s up to 86400 (default 3600).

  • A probe made by an engineer in production (or with no environment) is pending until a project admin calls POST .../probes/{id}/approve.

  • At most 25 pending or active probes per project. Expiry is enforced when serving, so a late janitor never serves a dead probe. Audit events: probe.create, probe.approve, probe.delete, probe.expire.

Conditions

Comparisons (==, !=, <, <=, >, >=), &&, ||, !, in [..], string contains. Function calls are rejected.

Delivery and snapshots

SDKs poll GET /v1/probes/active?service&env&version on the edge every 30 s with If-None-Match, using an ingestion key with scope probes:read. A hit emits probe.id, code.filepath, code.lineno, code.function and code.snapshot.* (values cut to 1 KiB) with the active trace and span ids. The worker redacts them with the project's pipeline before storage, applies the probe's max_rate_per_min again and drops snapshots of expired probes.

Clients

Go: dyninst.Point("Pay", func(v dyninst.Vars) { v["order"] = o.ID }); the closure runs only while a probe targets Pay (or file:line). Browser: wt.probe("pay", () => ({ n })) with probes: { key } in the config.

Updated