Prometheus metrics

Two supported paths. Both need an ingestion key (wti_...) with the ingest:metrics scope, sent as Authorization: Bearer <key>. Set WT_OTLP_ENDPOINT to your gateway HTTP URL (for example https://ingest.example.com) and WT_INGEST_KEY to the key.

The Collector below is your own OpenTelemetry Collector (contrib distribution), not a Watchtower component.

A. Scrape with your OpenTelemetry Collector, forward as OTLP

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: node
          scrape_interval: 30s
          static_configs:
            - targets: ["localhost:9100"]
exporters:
  otlphttp:
    endpoint: ${env:WT_OTLP_ENDPOINT}
    headers:
      Authorization: "Bearer ${env:WT_INGEST_KEY}"
service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [otlphttp]

The exporter appends /v1/metrics. Counters keep their OTLP monotonic-sum type.

B. Prometheus (or Agent mode) remote write

Watchtower implements Prometheus remote-write v1 at POST /api/v1/write on the gateway HTTP port (snappy-compressed protobuf).

remote_write:
  - url: https://ingest.example.com/api/v1/write
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/wt-ingest-key

Limits are the OTLP ones: 8 MiB compressed, 32 MiB decompressed (413 beyond), per-project rate limit (429 with Retry-After), bad key 401, missing scope 403.

Mapping to OTLP

Delivery is at-least-once: Prometheus retries on 429/5xx, so duplicates are possible.

C. Scrape with the Watchtower agent

The agent's prometheus module scrapes targets and forwards them as OTLP. Config lives in the agent's own file (modules.prometheus); remote config can add plain targets (inside scrape_allow) but not relabel_configs, targets_file or textfile_dir.

modules:
  prometheus:
    targets:
      - url: http://localhost:9100/metrics
        scrape_interval: 30s       # 5s to 1h, default 60s
        scrape_timeout: 10s
        sample_limit: 50000        # the scrape fails (nothing is sent) above this
        honor_labels: false        # true: a scraped label wins over a target label
        labels: {env: prod}
        relabel_configs:           # on target labels before the scrape
          - {source_labels: [__address__], regex: "(.*):9100", replacement: "$1:9101", target_label: __address__}
        metric_relabel_configs:    # on every sample after it (__name__ is the metric name)
          - {source_labels: [__name__], regex: "go_.*", action: drop}
    targets_file: /etc/wt-agent/targets.yaml   # [{targets: ["host:port"], labels: {...}}], re-read every 30 s
    textfile_dir: /var/lib/wt-agent/textfile   # *.prom files; older than 10 min are skipped (wt_textfile_stale_files)

Relabel actions follow Prometheus: replace, keep, drop, keepequal, dropequal, hashmod, labelmap, labeldrop, labelkeep, lowercase, uppercase; regexes are RE2, anchored ^(?:regex)$; default separator ; and replacement $1.

The agent asks for OpenMetrics first (application/openmetrics-text) and falls back to the Prometheus text format. OpenMetrics exemplars arrive as attributes wt.exemplar.trace_id, wt.exemplar.span_id (any exemplar label) and wt.exemplar.value on the sample.

DogStatsD

Module dogstatsd (local config only) listens on UDP 127.0.0.1:8125 (set listen: ":8125" for containers) and/or a Unix datagram socket. Types g c ms h d s, tags #k:v, sample rate |@0.1 scales counts. Timers and histograms become <name>.count|avg|min|max|95percentile. Events and service checks are ignored and counted in wt_dogstatsd_ignored_total.

Log multiline and the disk buffer

A file input takes multiline: {start_pattern, max_lines: 500, max_wait: 2s} to join stack traces. Set buffer_dir (and optionally buffer_max_bytes, default 512 MiB) in the agent file to keep batches that could not be delivered on disk in 16 MiB CRC-checked segments; they are replayed oldest first after a restart, and the oldest segment is dropped (counted) when the cap is hit. Retries back off from 1 s to 5 min with 20% jitter. Delivery is at-least-once: a crash during replay can send a batch twice.

Remote-write 2.0 and HA deduplication

POST /api/v1/write also accepts Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request (symbols table, metadata with unit and help, created_timestamp, exemplars, native histograms). Native histograms with schema -4..8 become OTel exponential histograms; others are dropped and counted in wt_gateway_rw_histograms_dropped_total. Responses carry X-Prometheus-Remote-Write-Samples-Written and -Histograms-Written.

HA pairs: set {"ha": {"cluster_label": "cluster", "replica_label": "__replica__", "failover": "30s", "max_clusters": 0}} in the project settings. The first replica seen per cluster value leads; other replicas are dropped (wt_gateway_ha_dropped_total) until the leader is silent for failover. The __replica__ label is never stored. The leader is stored in Postgres, so every gateway agrees on it (each re-checks at most every 5 s). "max_clusters": N caps the clusters a project may elect leaders for.