Integrations

An integration is a directory integrations/<name>/manifest.yaml (shape: integrations/schema.json) that says how to recognize a product, which agent module collects it, what the metrics mean, which dashboard to show and which alerts to start with. Manifests are compiled into the API and read by the agent from disk.

v1 set (INF-002)

apache, clickhouse, coredns, elasticsearch, etcd, haproxy, host, kafka, memcached, mongodb, mysql, nats, nginx, postgres, rabbitmq, redis (16).

Collected by Watchtower modules: postgres, mysql, redis, memcached, mongodb, kafka, rabbitmq, nats (their own modules), nginx, apache, haproxy, elasticsearch (the http_status module, kind selects the page format), host (the host module). ClickHouse, etcd and CoreDNS use the existing prometheus module against the product's own metrics endpoint.

Target list

Ranked by how many reference vendors ship an integration for it (harvest frequency, most common first). The v1 set above covers the head of this list; the rest are added as manifests plus, where no Prometheus endpoint exists, a module.

1. Linux and Windows host, Docker, Kubernetes (host and container modules)

2. Postgres, MySQL and MariaDB, Redis, MongoDB, Elasticsearch and OpenSearch

3. nginx, Apache, HAProxy, Kafka, RabbitMQ, memcached

4. NATS, etcd, ClickHouse, CoreDNS, Consul, Vault

5. SQL Server, Oracle, Cassandra, Couchbase, Traefik, Envoy

6. Jenkins, GitLab, Prometheus exporters in general (the prometheus module covers them)

Manifest

name: postgres
version: "1.0.0"
discovery: {process_names: [postgres], ports: [5432], k8s_labels: {...}}
module: {kind: postgres, config_defaults: {dsn: "...", interval: "30s"}}
metrics: [{name, unit, description}]
dashboards: [dashboard spec v1]
alerts: [{name, rule, threshold_rationale, runbook_url, default_enabled}]
depends_on: [{parent_kind: host, equal_labels: [host_name]}]

A rule uses the metric names and labels as the worker stores them: dotted OTel names selected with {__name__="postgresql.database.xid_age"}, point attributes as labels, and the resource attribute host.name as host_name. Every module also reports wt.integration.up{integration=<name>} (1 per healthy round, 0 when the product cannot be reached), which the "unreachable" alert of each pack watches.

Modules and credentials

Addresses, DSNs, passwords and tokens live only in the agent's own config file (modules: section). Remote config can switch a module on or off and set its interval, nothing else, and cannot start a module the local file does not define. The agent never stamps wt.* attributes.

Postgres reports the connected database's tables; configure one DSN per database to cover several. The XID wraparound alert fires above 1.5e9 (the freeze limit is 2e9).

Service discovery (INF-021)

The listen module reports up to 200 listening TCP sockets with their process names. GET /v1/projects/{p}/integrations/detected matches them against every manifest's discovery (a process-name match wins over a port match) and answers rows such as "postgres on :5432 on host X", each with the remote config that switches the module on and the local config the agent file needs. Credentials are filled in on the host.

Alert packs (INF-029)

POST /v1/projects/{p}/integrations/{name}:install-pack creates one alert rule per pack alert, labelled pack=<name>, enabled as default_enabled says, with the rationale and runbook in the annotations. Packs are copy-to-edit: the install remembers the hash of each rule it wrote, so running it again (the pack update) rewrites rules still as the pack left them and skips any a person edited (outcome skipped_edited). Deleting a rule and installing again recreates it.

Trigger dependencies (INF-030)

depends_on becomes inhibition rules owned by the pack, evaluated by the router's ordinary inhibition (see docs/product/notification-delivery.md). The host pack has an alert "Host down" labelled down="true": hosts that sent the agent heartbeat in the last 7 days but not in the last five minutes. A host silent for more than 7 days resolves as stale, and the inhibition of its alerts ends with it. Installing a pack with depends_on: [{parent_kind: host, equal_labels: [host_name]}] adds the rule "source pack=host, down=true; target pack=<name>; equal host_name". Installing the host pack also adds a rule that holds back every other alert carrying the downed host's host_name, so container and service alerts of that host go quiet without a pack of their own. A re-install replaces the pack's managed rules; the N2 routes answer 409 managed by pack <name>; uninstall or re-install the pack for PUT and DELETE on them (managed_by shows in the list), unless an org admin passes ?force=true; a forced change detaches the rule from the pack.

The listen module needs a local opt-in (modules.listen), so detection is empty until it is set. For a product the agent reaches over the network, host_name is the agent's host, not the product's, so the host-down inhibition follows the agent's host. Several http_status targets in one file are written by hand into the one targets list; the detected guidance shows one target to merge in.