Data contracts and the attribute registry

Embed
Preview of the card
<a href="https://forumulate.com/p/watchtower/docs/data-contracts"><img src="https://forumulate.com/api/public/orgs/watchtower/spaces/docs/pages/937265eb-7c71-4ad7-b69f-16ae1bf58853/social-card.svg?v=3e7ae33eec8b" alt="Data contracts and the attribute registry" width="600" height="315"></a>

Declare the attributes your services may send. The contract pipeline stage then enforces the declaration on every log, span and metric point before storage. Managing the registry needs the ManagePipelines action.

The registry

POST /v1/projects/{p}/attribute-registry declares one attribute:

Field

Values

signal

logs, traces, metrics

key

attribute key; a trailing .* declares a prefix (app.metrics.*); the longest prefix wins, an exact key beats a prefix; wt.* is reserved

value_type

string, int, double, bool, bytes (attributes are stored as text, so int means the text parses as an integer)

pii_class

none, pii, secret, sensitive

mode

warn, drop, quarantine, redact: what to do when the value has the wrong type

owner, description

free text

GET lists (cursor pagination, ?signal=), PATCH|DELETE .../{id} change or remove an entry, POST .../import takes a YAML document or JSON array of at most 2,000 entries (a duplicate rejects the whole batch). A project holds at most 5,000 entries. Every change is audited (attribute_registry.create|update|delete).

Enforcement

Add {"type": "contract", "enforce": true} to the pipeline. Per attribute:

  • pii_class = secret: the value is always removed, whatever the mode.

  • pii_class = pii with mode redact: the value is replaced by a salted SHA-256 (hex). The salt is per project, so equal values hash equally within a project (joins work) but cannot be reversed or compared across projects.

  • Wrong type: warn counts it, drop removes the attribute, quarantine sends the whole record to the DLQ with reason contract:<key>, redact replaces the value with [contract].

  • Undeclared keys follow the project setting PUT /v1/projects/{p}/settings/contracts {"undeclared": "allow" | "warn" | "drop"} (default allow).

With enforce: false nothing changes; violations are only counted, so you can watch the effect before turning enforcement on. The registry is read by the worker with the pipeline config (cached for 30 seconds).

Seeing violations

Stage stats report the outcomes contract_warn, contract_drop, contract_quarantine and contract_redact. Per-key counts are written to attr_bytes_1d.violations; GET /v1/projects/{p}/attribute-registry/violations?from&to returns the top keys.

Updated