Scheduled data export (PLT-028)

Embed
Preview of the card
<a href="https://forumulate.com/p/watchtower/docs/export-schedules"><img src="https://forumulate.com/api/public/orgs/watchtower/spaces/docs/pages/95eb25f0-01aa-4e16-bbd5-a359c01ac613/social-card.svg?v=3e7ae33eec8b" alt="Scheduled data export (PLT-028)" width="600" height="315"></a>

Writes closed time windows of logs or traces to your own object storage as Parquet, on a cron. Separate from streaming export-destinations, which sends events as they arrive.

  • API: POST|GET /v1/projects/{p}/export-schedules, GET|PATCH|DELETE …/{id}, POST …/{id}/run, GET …/{id}/runs. Needs the lifecycle permission. At most 20 schedules per project.

  • dest_kind: s3, gcs (S3-interop endpoint, HMAC keys) or iceberg (data files in the warehouse bucket, committed through an Iceberg REST catalog). Credentials are write-only: sealed at rest, never returned or audited. A credentials_ref (wt+secret://) can be used instead.

  • Iceberg: the archiver creates watchtower.<signal> (format v2) on the first run when it is missing, partitioned by dest.catalog.partition (up to 4 {source, transform: identity|day}); existing tables may be partitioned by identity or day. Columns are cast so Parquet types match the table (timestamps as microsecond UTC, ids as strings), and a name mapping is set because the files carry no field ids. Each window writes one file per partition, a manifest and a manifest list under {prefix}/iceberg/<signal>/metadata/, and commits one snapshot guarded by assert-ref-snapshot-id. credentials.catalog_token is the catalog's bearer token, bound to dest.catalog.uri. Only REST catalogs are accepted.

  • cron is five fields in UTC and must not fire more than every 15 minutes.

  • Window: from the last exported window end to the newest cron fire at least 5 minutes old. Object key {prefix}/signal=…/dt=…/hh=…/run-{schedule}-{window_start}.parquet; a retry overwrites the same key. Delivery is at-least-once. Windows over 50 GiB split into parts (-pN).

  • Failures retry after 1, 5 and 30 minutes, then the run is failed with its error. The cursor never moves past a failed window; a failed window is tried again at the next cron fire.

  • Erased rows (deletion ledger) are filtered out; legal holds do not block export.

  • Limits today: logs and traces only, optional columns list, no row filter.

Updated