Loading exported Parquet into a warehouse

Embed
Preview of the card
<a href="https://forumulate.com/p/watchtower/docs/warehouse-loading"><img src="https://forumulate.com/api/public/orgs/watchtower/spaces/docs/pages/0f710109-e1a1-422d-97a4-a60ee4c43d83/social-card.svg?v=3e7ae33eec8b" alt="Loading exported Parquet into a warehouse" width="600" height="315"></a>

Exports land under {prefix}/signal=<signal>/dt=<YYYY-MM-DD>/hh=<HH>/, Hive-style, so most engines discover the partitions by path.

  • Athena / Trino: CREATE EXTERNAL TABLE … PARTITIONED BY (signal string, dt string, hh string) STORED AS PARQUET LOCATION 's3://bucket/prefix/', then MSCK REPAIR TABLE.

  • BigQuery: external table over gs://bucket/prefix/* with hive partitioning mode AUTO.

  • Snowflake: external stage on the prefix, COPY INTO … FILE_FORMAT = (TYPE = PARQUET) MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE.

  • DuckDB: SELECT * FROM read_parquet('s3://bucket/prefix/**/*.parquet', hive_partitioning = true).

  • Iceberg destinations are appended as one snapshot per window; a snapshot summary property watchtower-commit-key ({schedule}:{window_start}) makes a repeated commit a no-op.

Delivery is at-least-once: a window whose cursor update was lost is rewritten to the same key, so file loads are safe to repeat; deduplicate on the full row when appending to a table from two overlapping runs.

Updated