wt-agent

One Go binary per host. It collects through modules and sends OTLP/HTTP to the gateway like any other client. It also registers with wt-edge and follows a remote config chosen by the agent's labels. Host, process, container and log modules arrive with P3-I2; this page covers the core, remote config, custom checks and packaging.

Install

• Linux: .deb or .rpm (systemd unit wt-agent.service, runs as user wt-agent), or install.sh.

• macOS: tarball plus infra/agent/com.watchtower.agent.plist (a LaunchDaemon that runs as the unprivileged _wtagent account, which install.sh creates).

• Windows: zip, then wt-agent service install from an elevated prompt. The service runs as NT AUTHORITY\LocalService, so the config and token files must be readable by that account.

The agent never runs as root or SYSTEM. Plugins run with its rights, so keep the token file readable by the agent account only.

install.sh --base-url https://dl.example.com/wt-agent/1.2.3 --sha256 <hex> downloads the tarball, checks the sha256 you give it and installs the binary. Read the script before running it as root; never pipe it from the network into a shell. make agent-packages builds everything into dist/.

After installing, create a credential with the scopes agent:fleet, ingest:metrics and ingest:logs (no allowed_origins: an agent:fleet credential is never a browser key), write it to /etc/wt-agent/token, set endpoint in /etc/wt-agent/agent.yaml (infra/agent/agent.example.yaml) and start the service. The endpoint must be https (loopback is exempt). The token is shown once at creation and stored only as a SHA-256 hash on the server.

Identity and fleet

Both fleet endpoints need agent:fleet; other credentials get 404 or 401.

POST /v1/agents/register on wt-edge takes {agent_id?, hostname, version, os, arch, labels} and returns {agent_id}. The project and environment come from the credential only. The id is kept in <state_dir>/identity.json; a restart presents it and keeps the same row. Labels can be changed only by the credential that first registered the id; any other credential, or another project, gets 404. Agents per project are capped at 10,000.

GET /v1/agents/{id}/config?version=N waits for a config different from version N. wait is clamped to 5 to 60 seconds (default 30). It answers 200 with a config, or 304 when nothing changed. When the agent runs a config that no longer applies (deleted, or the selector no longer matches), it answers 200 with an explicit empty config (config_id empty, version 0) and the agent clears its remote state. All waiting agents of a project share one database lookup every 5 s. A heartbeat is written at most once a minute per agent. Limits: 12 config requests a minute per agent and 12 registrations a minute per credential, separate from the error-ingest limits. GET /v1/projects/{p}/agents lists the fleet.

Remote config

Configs are managed with POST|GET /v1/projects/{p}/agent-configs, PUT|DELETE …/{c}, GET …/{c}/versions and POST …/{c}:rollback?version=N. A config is {name, selector, config}; config is {modules: {<name>: <module config>}}. Every change writes a new version (numbered per project) and an audit event. A rollback restores an earlier version's content as a new version.

The agent gets the most specific config whose selector matches its labels: most label matchers wins, ties go to the newest, and {} matches every agent.

What remote config can and cannot do

Executable plugins, their paths and arguments exist only in the agent's own file. Remote config cannot set plugin, args, env or a path: the API rejects them and the agent refuses them again. It can:

• switch a local Nagios check on or off by name and set its interval, timeout and labels;

• add Prometheus scrape targets whose host:port is in the local scrape_allow (no redirects).

A module with no remote overlay (everything else) cannot be configured remotely. A refused remote module is skipped with a log line and the local config of that module stays in force. disable_remote_config: true turns remote config off.

Integrity

• Transport: https to wt-edge; redirects are refused so the token never follows one.

• A served config carries a sha256 of the body and an HMAC-SHA256 over agent id, config id, version, selector and body, keyed from the credential. The agent verifies both, that the selector matches its labels, and that the body has no unknown keys, before it applies anything. The agent keeps the highest applied version per config in <state_dir>/applied.json and refuses a lower one, so an old signed config cannot be replayed.

• This is a tamper check, not a trust anchor. Whoever controls the server (the edge, the database, a project admin) can have any config signed, so the real boundary is the agent's local file: the worst a remote config can do is toggle checks and scrape targets you already allowed.

• Residual risk: a project admin can switch your local checks on or off and tune their timing, and can add scrape targets on the allow list (any path on an allowed host:port). Keep scrape_allow tight and the plugin directory to checks you vet.

Custom checks

Nagios (nagios module, defined locally): {checks: [{name, plugin, args, interval, timeout, enabled, labels}]}. plugin is a file name inside plugin_dir, run with args as argv (no shell) and a minimal environment without the agent's token. Use a dedicated directory; never a stock plugin directory with wrappers such as negate, urlize or check_by_ssh. Timeout default 10 s, maximum 60 s, output capped at 64 KB. On timeout the whole process group is killed (the process tree on Windows). .bat and .cmd plugins are refused on Windows. Exit 0 OK, 1 WARNING, 2 CRITICAL, 3 UNKNOWN; any other exit, a timeout or a missing plugin is UNKNOWN. Emits wt_check_status (0 to 3, attributes check, plugin, status) and, per perfdata item label=value[UOM];warn;crit;min;max, gauges wt_check_perf_value, …_warn, …_crit, …_min, …_max (attributes check, plugin, label, uom). Thresholds in Nagios range syntax such as 10:20 are not converted and are left out.

Prometheus (prometheus module): {targets: [{url, interval, labels}]}. Counters become cumulative sums, gauges and untyped values gauges, histograms _bucket{le}, _sum and _count, summaries {quantile}, _sum and _count. wt_scrape_up reports each target. A scrape is limited to 10 MiB and 50,000 samples.

Local config reference

endpoint, fleet_endpoint, token_file (or env WT_AGENT_TOKEN), state_dir, labels, plugin_dir, scrape_allow, insecure_http, disable_remote_config, modules (local module configs, always trusted). See infra/agent/agent.example.yaml.

Host, process, container, GPU and log collection (P3-I2)

Modules (all built in, pure Go, no cgo): host (runs by default; host, filesystem, disk, NIC and top-50 process metrics, OTel semconv names, via gopsutil), container (Docker Engine API over the unix socket or named pipe, plus cgroup v2 for containerd/podman), gpu (runs nvidia-smi / rocm-smi at paths set in the local file), logs (file tail with offsets per inode and multiline, journald, Windows Event Log, syslog receiver).

Remote config can only set enabled and interval of a module the agent runs; sockets, tool paths, log paths and listen addresses exist only in the local file. The syslog receiver is unauthenticated: bind it to loopback or a trusted interface.

Discovery (discovery: [{kind: filesystem|disk|nic|gpu, include, exclude}]) filters which children (mount points, devices, interfaces, GPU indexes) are collected, with glob patterns; exclude wins. No parent label is sent: the host of every child series is the resource host.name, which the worker exposes as the label host_name (this adds host_name to every series of resources that carry host.name, so existing series identities gain a label at deploy). Client wt.* attributes, including a stray wt.entity.parent, are dropped at ingest as always.

Entities: the metrics worker upserts infra_entities (created by migration 0027_infra.sql, owned by I1; I2 adds the props column there) from host.id|host.name, container.id, k8s.<kind>.uid and cloud.resource_id. Upserts run in a background task fed by a bounded queue (dropped and counted in wt_worker_entities_total{outcome="queue_full"} when full, never blocking ingest), batched into one multi-row statement with a 2 s timeout. Identity attributes (an allow-list per kind) are stored in attrs, and a change is appended to entity_history; other attributes are merged into props without history. last_seen is written at most once per 60 s per entity. A container, pod or cloud resource gets the host named by the same resource as its parent. Each project is capped at 50,000 entities; new ones beyond that are refused (outcome="refused_cap").

Offsets and cursors are persisted after records reach the agent's in-memory export buffer, so delivery is at-least-once only up to that buffer (8 MiB); a crash or a full buffer can lose records.

Known gaps: G-24 (no live agent-served process views), GPU via CLI rather than NVML, no Event Log bookmark (events during agent downtime are not replayed).