Federated queries and HA deduplication

One logs or metrics query over several projects of an org, and over projects of other Watchtower clusters (MET-017, LOG-001).

Peers

Register a peer cluster (needs ManageOrg):

POST /v1/orgs/{org}/federation/peers
{ "name": "eu", "base_url": "https://eu.example.com", "credential": "<service API key of the peer>",
  "region": "eu-west", "allow_private": false }

• The credential is a service API key of the remote cluster. It is sealed, write-only, and never returned, logged or put in an audit event. A read-only key works.

• base_url follows the SSRF rules (https, no embedded credentials, no private, loopback or link-local address; names are re-resolved and checked at every call). An org admin may set allow_private for an in-VPC peer; the audit event records it.

• GET /v1/orgs/{org}/federation/peers, GET|DELETE …/{id}. At most 20 peers per org.

• Peers need WT_SECRET_KEY on the API (503 otherwise).

Query

POST /v1/query/federated
{ "kind": "logs" | "metrics", "org": "<org, when a peer is named>",
  "projects": [ {"project": "prj_…"}, {"peer": "eu", "project": "prj_…"} ],
  "query": "…", "from": "…", "to": "…", "step": "30s", "limit": 100, "dedup_label": "__replica__" }
→ { "data": […], "partial": false, "errors": [ {"peer": "eu/prj_…", "code": "timeout"} ] }

• Every local project needs ReadTelemetry; if any is not visible the whole request is 404 before any query runs.

• A peer receives the same request for one project over its own /v1 API. Per-peer deadline 10 s, whole fan-out 15 s. A slow or failing peer sets partial: true and appears in errors (timeout, unreachable, denied, bad_response, peer_error); the other sources still answer.

• Logs: query is free text. Entries are merged newest first, ties by source then id, the limit applies after the merge, and each entry carries source (project, or peer/project).

• Metrics: query is PromQL, step is required. Series get a project label and, from peers, a peer label. Up to 1000 series.

HA deduplication

With dedup_label set, series that are equal except for that label are replicas of one series. At each step the replica that has a sample there and the fewest missing samples within two steps on either side is kept; the label is dropped. Deduplication runs inside each source, so replicas that live in different projects or clusters are not merged (their project/peer labels differ).

Not built: a per-project dedup_label setting (the parameter is per request).