Metric query extensions and the Datadog-style dialect

Embed
Preview of the card
<a href="https://forumulate.com/p/watchtower/docs/metrics-extensions"><img src="https://forumulate.com/api/public/orgs/watchtower/spaces/docs/pages/9e883a3d-30d3-462d-ac05-81adf9af1c39/social-card.svg?v=3e7ae33eec8b" alt="Metric query extensions and the Datadog-style dialect" width="600" height="315"></a>

Applies to GET|POST /v1/projects/{p}/prometheus/api/v1/query and query_range. Queries are always scoped to the project of the credential.

Extensions (QRY-006)

Syntax

Meaning

WITH (a = expr, b = a * 2) body

Templates, at most 16, expanded textually. A binding may use earlier ones only; self or forward use is rejected (422). Names are not substituted inside strings, {..}, [..] or by (..) lists.

a default b

a or b; with a number n on the right (fills a series missing at a step, from its first appearance, with n).

a if b / a ifnot b

a and b / a unless b. on(..) / ignoring(..) may follow the keyword.

label_set(v, "k", "v", ..)

Set labels; an empty value removes the label.

label_del(v, "k", ..) / label_keep(v, "k", ..)

Remove, or keep only, the listed labels.

limit_offset(limit, offset, v)

Skip offset series (ordered by label set), return up to limit. Integer literals.

median_over_time(r)

quantile_over_time(0.5, r).

histogram_share(le, buckets)

Share of observations <= le from classic le buckets, linear inside a bucket. le is a number literal.

default, if and ifnot have the lowest precedence and associate left. and, or, unless and vector() are also available in plain PromQL.

See promql-compat.md for the supported core PromQL, subqueries and native histograms.

Limits (answer 422 unsupported promql)

Query length 8192 bytes (also after WITH expansion), syntax tree depth 64 and 2000 nodes, evaluation cost 50M units ("query too expensive"), WITH bindings 16. Range queries also stop at 11,000 steps, 10,000 series and 3,000,000 points (400 bad_data).

Datadog-style dialect (MET-016)

?dialect=dd accepts agg:metric.name{tag:a,!tag:b*} by {host}.rollup(sum, 60).

  • agg: avg sum min max count. Dots and dashes in names become _.

  • Filters: *, tag:value, !tag:value; * inside a value makes a regex match.

  • .rollup([method,] seconds), method avg sum min max count, 1 to 86400 s, becomes <method>_over_time(sel[Ns]) inside the aggregation.

  • Any other function (.as_count(), .fill(), ..) answers 422 naming it.

Example: avg:metric.name{tag:a,!tag:b*} by {host}.rollup(sum, 60) becomes avg by (host) (sum_over_time(metric_name{tag="a",tag!~"b.*"}[60s])).

Updated