Archive, rehydration and pipeline replay

Watchtower copies closed days of logs and traces to a bucket you own, brings them back on request, and can replay them through your current pipeline.

Archive

PUT /v1/projects/{project}/archive/{logs|traces} with bucket, prefix, region, optional endpoint (MinIO, R2, GCS interop), delay_hours, enabled and write-only credentials. Needs the lifecycle permission.

• delay_hours must be at most (retention_days - 1) * 24, so a day is archived before retention deletes it.

• The bucket is probed at save time (a write and delete of _wt_probe); failure is 422 archive_unreachable.

• A private or loopback endpoint is refused unless allow_private_endpoint is set by a security admin; the change is audited.

• Credentials are sealed, never returned and never written to the audit log.

Layout: {prefix}/{org}/{project}/{signal}/dt=YYYY-MM-DD/part-NNNN.parquet plus _manifest.json per day (row counts and SHA-256 per part). Parts are Parquet (ClickHouse writes them), split by hour and, past 128 MiB, by hash shard. The manifest is written last. A day is archived once; a crash resumes at the first missing part. Rehydrated rows are never archived again.

Rehydration

POST /v1/projects/{project}/rehydrations with signal, from, to, optional filter (service, contains), ttl_days (1 to 30, default 14) and reprocess.

• At most 7 days per request, 200,000,000 rows, one active request per project.

• Each part's SHA-256 is verified; a corrupt part fails the request.

• Rows removed by a deletion request are never brought back.

• Rows are inserted with rehydration_id set and retention_days = ttl_days; retries are deduplicated with the token rehydrate:{id}:{day}:{part}.

• States: queued, running, done, failed, expired. DELETE removes the rows now.

Pipeline replay

With reprocess: true (logs), rows are published to wt.replay.logs and the worker runs the project's current active pipeline over them (for example a new redaction rule), writing with token replay:{id}:{seq}. Delivery is at-least-once; the token makes a redelivered batch a no-op.