Importing metric samples

Embed
Preview of the card
<a href="https://forumulate.com/p/watchtower/docs/metrics-import"><img src="https://forumulate.com/api/public/orgs/watchtower/spaces/docs/pages/6a07ffea-f55f-4148-9865-c44853fcad39/social-card.svg?v=3e7ae33eec8b" alt="Importing metric samples" width="600" height="315"></a>

POST /v1/projects/{prj}/metrics/import takes OpenMetrics text with a timestamp on every sample, or use the CLI:

wtctl metrics import --project <id> --file samples.om

From Prometheus or Mimir TSDB blocks: blocks are not uploaded directly. Convert them first:

promtool tsdb dump-openmetrics /path/to/data > samples.om
wtctl metrics import --project <id> --file samples.om
  • The project comes from your credential. wt.* labels are dropped. The service label becomes the series' service; an env label is kept as exported_env, because env is set by the platform.

  • Needs the manage_project permission. At most 64 MiB, 2,000,000 samples and 100,000 series per import, and at most 30 seconds' worth of the project's metrics rate quota. If the project has a daily metrics byte quota, the body size plus today's metered bytes must fit in it (422 otherwise). Split larger files.

  • Samples are queued (202) and pass through ordinary ingest: they count toward usage, the series caps and the project's metric limits. Refused samples appear in the ingest debug view. If the project sets past_grace_period_s or an out-of-order window, it applies to imported samples too, so widen it for a backfill.

  • Samples older than the project's metric retention are skipped and counted in the response.

  • Float samples are stored as plain float series (classic histograms as their _bucket, _sum and _count series).

  • Native histograms, which OpenMetrics 1.0 text cannot express, use Prometheus' test notation as the value: h{job="a"} {{schema:0 count:4 sum:5 z_bucket:1 z_bucket_w:0.001 offset:0 buckets:[1 2 1] n_offset:0 n_buckets:[]}} 1700000000. Buckets are absolute counts from offset (Prometheus numbering); counts must be whole numbers; schema -4 to 8.

  • Delivery is at-least-once: repeating an import after an error can store samples twice.

Updated