Relays: store-and-forward ingestion at the edge

A relay is wt-agent --mode relay. Run it inside a network that cannot always reach the gateway (branch sites, air-gapped VPCs, flaky links). It accepts OTLP gRPC (:4317) and HTTP (:4318: /v1/logs, /v1/traces, /v1/metrics, protobuf or JSON, optional gzip), applies edge-safe processors, queues every batch on disk and forwards it to the gateway with the project's ingestion key. Delivery is at-least-once; the gateway path deduplicates as for any sender.

Register the relay

POST /v1/projects/{project}/relays   {"name": "branch-1", "zone": "vpc-a", "peer_group": "branch"}

The response carries the relay token (wtr_...) once; only its SHA-256 is stored. The token is used for heartbeats only. Upstream the relay presents the project's ingestion key, so no new credential kind reaches the gateway. GET /v1/projects/{project}/relays lists each relay with last_seen, queue_depth and dropped_total; DELETE .../relays/{id} revokes it.

Configure

# /etc/wt-relay/relay.yaml   (WT_RELAY_KEY, WT_RELAY_TOKEN, WT_RELAY_PEER_SECRET override the files)
upstream: https://ingest.example.com:4318
key_file: /etc/wt-relay/ingest.key        # wti_...
api: https://api.example.com               # heartbeat every 30 s
token_file: /etc/wt-relay/relay.token      # wtr_...
queue: { dir: /var/lib/wt-relay/queue, max_bytes: 1073741824 }   # 1 GiB default
processors:
  version: 1
  stages:
    - { type: redact, keys: [x-internal-token], patterns: ["card-\\d+"] }
    - { type: filter, drop_if: { attr: env, eq: dev } }
    - { type: sample, rate: 0.25, by: trace_id }
    - { type: enrich, set: { site: branch-1 } }

Processors use exactly the Rust pipeline schema and semantics (redact, filter, sample, enrich); both implementations run testdata/pipeline_vectors.json. Built-in redaction is always on. They apply to logs and spans; metrics are forwarded unchanged. Two deliberate differences: the relay never emits wt.* attributes (the Rust sample stage stamps wt.sample_rate; the gateway owns that namespace) and it deletes any wt.* it receives. accept_token_file makes local senders present a bearer token.

Queue and failure behavior

• Batches go to a disk write-ahead log and survive restarts. A batch that fails stays queued.

• Retries use exponential backoff from 1 s to 60 s with jitter. Retry-After on 429/503 is honored (up to 5 min). 401/403 are retried too, so a rotated key does not lose data.

• A batch the gateway rejects with another 4xx (malformed) is dropped and counted in wt_relay_rejected_total.

• When the queue is full the oldest data goes first, in 16 MiB segments, and every batch lost is counted in wt_relay_dropped_total (also in the registry). Nothing is dropped while there is room. Depth is approximate after a failed drain and exact once drained.

• Metrics at GET /metrics: wt_relay_queue_depth, wt_relay_dropped_total, wt_relay_forwarded_total, wt_relay_rejected_total.

Peer groups: one trace, one relay

Relays that list each other in peers (or in peers_file, a JSON array or one URL per line that fleet config can write, re-read every 30 s) shard spans by trace id with the same rendezvous hashing as shuffle sharding, so the spans of a trace reach the gateway through one relay. Peers authenticate with a shared secret (peer_secret_file); a batch routed by a peer is never re-routed. If a peer is unreachable the span is kept and sent by the relay that received it. Set self to the URL peers use to reach this relay.