PromQL compatibility

Watchtower evaluates PromQL natively (crates/wt-promql, parsing by promql-parser). Semantics follow the Prometheus 3.x documentation: left-open range windows (t-range, t], 5 minute lookback, staleness markers. Reference vectors in crates/wt-promql/tests/vectors/prometheus_docs.tsv are written from documented behavior, not copied from any test suite. Watchtower extensions (WITH, default, if, ...) are in metrics-extensions.md.

Supported

bitand bitor bitxor (precedence of *; nonnegative integers below 2^53, else NaN) and duration expressions (x[5m * 2], x[step() * 4], [range()], offset (1h + 30m), min/max of durations) are evaluated as in Prometheus 3.

Delta temporality

OTLP delta sums and histograms are stored with the label __temporality__="delta". rate, increase and irate over such series sum the samples in the window (no extrapolation) and drop the label; rollup tiers keep the sum of the deltas in a bucket. sum_over_time(x[5m]) gives the same increase directly. Joining raw delta series with raw cumulative series is refused with HTTP 422 (errorType: execution on the Prometheus API, unprocessable on native routes) and a message naming the fix: wrap both sides in rate() or increase().

anchored and smoothed

rate(x[5m] anchored) starts the window at the last sample at or before its start (within the lookback), without extrapolation; smoothed interpolates the start value between the samples around the window start and the end value between the samples around the window end, reading up to the 5 m lookback past it. A counter reset across either edge is not interpolated.

Subqueries

Inner instants are the multiples of step inside (t-range, t] (aligned to absolute time, as Prometheus does). The default step is the evaluation interval: the query step for a range query, 60 s for an instant query. Nesting depth is at most 3 (deeper answers 422), at most 11,000 inner steps per subquery (422), and every inner evaluation counts against the 50M-unit cost budget of the query ("query too expensive", 422).

An instant query whose subquery covers more than a day and is a plain function argument runs that subquery as its own range query (split and parallel), then applies the outer function (subquery spin-off). offset and @ on that subquery are honoured. Results are unchanged; anything else (nested subqueries run inside the outer one's range query) takes the normal path.

Duration expressions accept Prometheus' max_of(a, b) and min_of(a, b), e.g. x[max_of(1m, step())].

start_timestamp(v) takes a vector selector and returns, per sample, its series start in seconds: the newest start stored at or before the sample. Starts are stored for cumulative float series (counters, summary _count/_sum) from OTLP start_time_unix_nano and remote-write v2 created_timestamp (metric_start_v1). Series without a known start (gauges, delta series, histograms, senders that set none) are left out of the result.

Native histograms

OpenTelemetry exponential histograms and remote-write native histograms are stored natively (metric_nhist_v1, schema at most 8, finer ones merged down) and read through SampleSource::select_native as NativeHist (schema, zero bucket, sparse positive and negative buckets). A project may store them as classic le buckets instead, or both (metrics settings exp_histograms). A selector that matches native series yields native samples. Quantiles interpolate exponentially inside a bucket and linearly around the zero bucket; histogram_fraction interpolates linearly. rate, increase (counter-reset aware, extrapolated like float series) and sum work over native histograms; histograms of different schemas are merged at the coarser schema.

• Mixing classic and native samples for the same series, or using a native vector in a binary operator, answers 422 cannot mix classic and native histogram series in one expression naming the label set.

• A query whose final result is a native histogram answers 422; wrap it in a histogram_* function, rate or sum first.

• Custom-bucket native histograms (NHCB, schema -53, from remote write) are stored as classic le buckets and read by histogram_quantile.

• Native histograms have no rollup tier: long ranges read them raw (within raw retention).

• avg, min, max and the other aggregations over native histograms answer 422.

Limits

Query length 8192 bytes, tree depth 64, 2000 nodes, 50M cost units, 11,000 points per series for range queries. Per project (/metrics/settings): series per result (default 10,000), points per query (default 3,000,000), longest range and lookback, blocked and rate-limited queries.

Prometheus API frontend

/v1/projects/{prj}/prometheus/api/v1/query_range splits long raw ranges into parallel 24 h chunks, routes long ones to rollups and caches results (nocache=1 skips the cache). stats=all adds a stats block. status/buildinfo reports version 3.0.0 so Grafana enables Prometheus 3 features; format_query pretty-prints a query.

• Sharding. A raw range of at least 1 h whose top level is sum, count, min, max, avg, group, topk or bottomk (literal k) over a per-series expression runs on 4 series shards in parallel and is merged (avg = shard sums / shard counts). quantile, stddev, cross-series operators, nested aggregations and subqueries run unsharded.

• Fetch caps. max_fetched_series_per_query / max_fetched_samples_per_query (0 = off) stop the storage read while it runs (422 when exceeded), on top of the result caps checked afterwards.

• Remote read. POST .../prometheus/api/v1/read answers snappy prometheus.ReadRequest with raw SAMPLES (32 queries, 50,000 series per request). A client listing STREAMED_XOR_CHUNKS first gets a sorted stream of XOR and float histogram chunks of up to 120 samples, read in batches. Both response types return native histograms.

• Read consistency. X-Read-Consistency: strong waits (up to 20 s, else 503) until every metric write acknowledged before the request is stored; it skips the results cache. Default is eventual.

• Federation. X-Wt-Projects: <id>,<id> evaluates over several projects of the same org the caller can read (else 404); each series carries __project__. The header applies to query, query_range, series, labels, label/{name}/values, query_exemplars and metadata: a __project__ matcher in match[] picks projects, __project__ is listed as a label and its values are the project ids. Each project is read under the caller's own data access policy there, and the window is clamped to the shortest metrics retention among them. Federated answers are cached per set of projects and the caller's scope in each. Recording rule groups take source_projects the same way.

• Search. search/metric_names, search/label_names and search/label_values?label= take search (substring, any case) or fuzzy=true (its characters in order), limit (1 to 1000) and start / end, and answer {name, series} rows with has_more. The native API has the same as GET /metrics/search.

• Cardinality. cardinality/label_names, cardinality/label_values?label_names[]=, cardinality/active_series?selector= and cardinality/active_native_histogram_metrics follow Mimir's parameters (selector, limit, count_method=active (20 min) or inmemory (3 h)) and response shapes. The Cardinality view in Metrics uses them.

• Limits. user_limits and user_stats answer with Mimir's field names; GET /metrics/limits is the native document (quota, series caps, settings and rule limits, plus active series, bytes today and samples per second).

• Series cap across workers. quotas.max_series holds for the project as a whole, however many workers ingest: new series are claimed from one shared budget per minute. It can be passed only by series that arrive within the write delay around a minute boundary; if Postgres is unreachable, each worker decides alone until it is back.

• Plan explain. analyze?query= lists the tree as evaluated after rewrites and planner passes, one row per node, with the matchers storage receives and notes (statically empty, shared input, narrowable).

• Usage. GET /metrics/cost-attribution (samples received / discarded and active series per value of the configured cost_attribution_labels), GET /metrics/active-series/trackers (named selectors), and POST /metrics/backfills to evaluate a recording rule over a past range (90 days, 200,000 steps, 5 open per project) into storage.

• Planner. Repeated selectors in one query are read once, number arithmetic is folded, and matchers cross binary operators (a{env="p"} / on (env) b reads only b{env="p"}; not across or, fill* or into the left of unless). Series, labels and label-values answers are cached, except their last 5 minutes, which are read live, so new series show at once.

• Delayed name removal. enable_delayed_name_removal in /metrics/settings (off by default, as in Prometheus 3's promql-delayed-name-removal) keeps __name__ until the end of a query, so label_replace and by (__name__) still see it after rate and the like. It applies to the Prometheus API, recording rules, backfills and alert rules. A by (__name__) over rate with two or more names fails with a duplicate-labelset error, as in Prometheus.

• Delta temporality on long ranges. Rollup buckets of delta series hold the sum of their deltas, for floats, classic and native histograms, so rate and increase stay correct without reading raw samples.

Not supported

Each name below answers 422 unsupported_promql naming it (checked by unsupported_list_matches_doc).

<!-- unsupported:begin -->

Query planning passes

• Matcher propagation. sum by (env) (a{env="p"}) / on (env) sum by (env) (b) reads only b{env="p"}. Matchers cross binary operators and pass through parentheses, per-series functions, subqueries and by / without aggregations, in the same cases as Mimir's query engine. They do not cross or or fill*, the left of unless takes nothing from its right, and they do not pass through label_*, absent, scalar, count_values, or a grouping that removes the label.

• Difference from Prometheus. Valid queries give the same result. Prometheus checks every series on the "one" side for duplicates, also in groups that cannot match, and fails with a many-to-many error. Here (and in Mimir) those series are not read: a{env="p"} * on (env) b succeeds when only b{env="q"} has duplicates. A duplicate in a group that matches still fails.

• Narrowed selectors. In a op on (l) b, when one side reads only instant selectors that keep l, it is read first and the other side reads only the l values it saw (l=~"v1|v2"), up to 256 values. rate(x[5m]) * on (instance) group_left (node) node_uname_info reads only the x series of instances that have an info series. Not applied inside subqueries, across or or fill*, or to the left of unless.

• Split long windows. In an instant query, sum_over_time, count_over_time, min_over_time and max_over_time of a selector over more than two days (up to 120) run per UTC day and are combined; whole days are cached (60 s while the day ended within the last hour, then 1 h), except for callers under a data access policy. rate and increase are not split, because their edge extrapolation does not add up across days; long ranges of those use the rollup tiers.

• Reduced matchers. Duplicates, matchers an equality already implies (a="x", a=~"x|y") and =~".*" are not sent to storage.

• Statically empty parts are not read: contradictory matchers (a{x="1",x="2"}), vector(1) == 2, timestamp(x) < c before any visible sample, and whatever +, and or unless then makes empty. An unsupported function on a skipped side answers empty instead of 422.

• Shared inputs. sum(rate(x[5m])) / count(rate(x[5m])) computes the input once per step. histogram_count, histogram_sum and histogram_avg directly over a selector, rate, increase or sum do not read bucket arrays.