Fleet control

Embed
Preview of the card
<a href="https://forumulate.com/p/watchtower/docs/fleet"><img src="https://forumulate.com/api/public/orgs/watchtower/spaces/docs/pages/93cb2165-8f8c-4abf-bc88-9ca6217f05e2/social-card.svg?v=3e7ae33eec8b" alt="Fleet control" width="600" height="315"></a>

wt-fleet is the control plane for the agents and collectors you run. It speaks OpAMP on /v1/opamp (plain HTTP polling and WebSocket), so it works with wt-agent and with upstream OpenTelemetry Collectors that ship the OpAMP extension. You give each agent the same credential you would give it for ingestion, plus the agent:fleet scope.

What you can do

  • See every connected agent: host, kind (agent or collector), version, the config it runs, health and last seen (GET /v1/projects/{p}/fleet/agents, Infrastructure > Fleet).

  • Keep configs as immutable versions. A new version is checked against the component catalog before it is stored. An unknown receiver, processor, exporter or extension is refused with 422 and the path of the problem (exporters.kafka). Exporters may only send to the gateway (${env:WT_GATEWAY_ENDPOINT}) or to a host in WT_FLEET_EXPORTER_ALLOW.

  • Use ${env:NAME} and ${secret:NAME} placeholders for values the agent resolves itself. The server never sees them.

  • Pull configs from git: create the config with source: {kind: git, url, path, ref}. The fleet server fetches it every 60 seconds (HTTPS only, through the SSRF guard) and adds a version only when the file content changed. A file that fails the catalog is skipped and logged; the running version stays.

  • Roll a version out to agents chosen by labels, os, env and zone, all at once or as a canary: canary_pct of the matching agents get it first. After bake_s seconds the canary is judged: if at least 100 - max_unhealthy_pct percent of the canaries run the new version and are healthy, it is promoted to every matching agent. Otherwise the rollout is marked rolled_back and a new rollout restores the previous version (rolled_back_from points at the failed one). An agent that stays silent for two minutes counts as unhealthy.

  • Ask an agent for a flare (diagnostics bundle) or a restart. Flares are redacted for token-like strings on the server, stored for seven days and listed per agent.

  • Register signed agent packages. The offer carries an Ed25519 signature; the agent checks it against the release key compiled into its binary (-X main.ReleaseKey=<hex>) before it downloads anything. A bad signature is reported as failed and the version is not offered to that agent again (no retry loop).

Using it

wtctl fleet push <config-id> collector.yaml --note "add filelog" --api ... --key ... --project ...
wtctl fleet rollout <config-id> --version 3 --canary 10 --bake 300 --os linux --env prod ...
wtctl fleet agents ...

Point wt-agent at it with fleet_mode: opamp in its local config (optional opamp_endpoint, default <fleet_endpoint>/v1/opamp). Local settings that decide what the server may do to the host (plugin directory, scrape allow-list) stay local and cannot be changed by a fleet config.

Limits and notes

  • Plans can cap fleet_agents. Over the cap, the fleet server refuses new agents and the API refuses new rollouts (422 limit_reached).

  • Agents are identified by their OpAMP instance uid, which must be a UUID. wt-agent uses its stored agent id.

  • Rollouts need the fleet server running; the API only records intent.

  • Not built yet: in-place binary replacement (the agent stages the verified package), OpAMP connection-settings offers, and package upload (use a URL).

Updated