Ingest cost shaping

Embed
Preview of the card
<a href="https://forumulate.com/p/watchtower/docs/cost-shaping"><img src="https://forumulate.com/api/public/orgs/watchtower/spaces/docs/pages/7ae34a43-0cab-495e-9849-673ddd697b96/social-card.svg?v=3e7ae33eec8b" alt="Ingest cost shaping" width="600" height="315"></a>

Cost shaping reduces stored volume with rules you can try before they change anything. Rules live under /v1/projects/{p}/shaping-rules and need the manage_pipelines permission.

Rule kinds

Kind

What it does

Parameters

drop

removes records matching match

none

drop_attrs

removes attributes

keys: exact keys or prefix* globs

sample

keeps a fraction of matching records (trace-consistent when a trace id exists)

rate 0 to 1

reduce

truncates the log body at a UTF-8 boundary

max_bytes

aggregate

metrics only: rolls matching points up per batch without drop_labels

metric glob, drop_labels, fn (sum, last, max, min)

match is a pipeline expression. Besides attrs.<key>, resource.<key> and body it can use severity (OTel number, so debug or lower is severity <= 8), service and, for metrics, metric_name. Rolled-up points carry the attribute shaping.rollup=1. Records stamped wt.self=true (our own telemetry) are never touched.

Preview first

A new rule starts in preview: the managed shaping stage counts matches (outcome preview_match in the pipeline stage statistics) and changes nothing. POST .../{id}/impact estimates the last 24 hours: {rows, bytes_est, pct_of_signal}. For drop_attrs the estimate is exact per key from the attribute byte rollup; for the other kinds it is the project's shaping stage counters (matches are not split per rule). Bytes are estimates.

Enabling

Every create, change or delete publishes one managed shaping stage as the first stage of the signal's pipeline, as a new pipeline version with the note shaping rules @ <time>. Diff and rollback work as for any version.

Reference guard. Enabling a drop_attrs rule, or an aggregate rule with drop_labels, that would remove a key an alert rule or dashboard references returns 422 with rule_breaks_reference: <keys>. ?force=true overrides the guard and needs manage_security. The guard checks attribute and label references only; it does not parse match expressions.

Adaptive suggestions

A daily janitor job proposes rules from usage data (GET .../shaping-suggestions):

  • attribute keys holding 1% or more of a signal's bytes with no reference for 30 days (drop_attrs)

  • debug-or-lower logs at 10% or more of volume with no query on severity (sample to 10%)

  • metrics with no queries for 30 days and 1% or more of samples (drop)

  • metric labels whose distinct values grew tenfold in a week (aggregate, summed; check gauges in preview)

Each stores est_bytes_saved and the reason. POST .../{id}/accept creates the rule in preview; .../dismiss hides it for good.

PUT /v1/projects/{p}/settings/shaping {"auto_apply": ...}:

Value

Behavior

off (default)

suggestions wait for you

preview

new suggestions are accepted in preview automatically

enabled

additionally, an automatically created preview rule is enabled after 7 days if the preview matched at least one record and the reference guard is clean

Automatic actions are audited with actor system:shaping.

Adaptive sink concurrency

Export sinks and ClickHouse inserts share an AIMD limiter: one more slot per 10 successes, halved on a timeout, 429 or 5xx, between 1 and 64. The gauge is wt_worker_sink_concurrency{sink}.

Limits

Aggregation windows are per delivered batch, not per interval_s. At most 200 rules per project.

Updated