Pipe language
Embed
<a href="https://forumulate.com/p/watchtower/docs/pipe"><img src="https://forumulate.com/api/public/orgs/watchtower/spaces/docs/pages/7305240b-93e1-4b3a-bad3-778d7e5bf94b/social-card.svg?v=3e7ae33eec8b" alt="Pipe language" width="600" height="315"></a>POST /v1/projects/{p}/query/search with {"language": "pipe", "q": "logs | where level:error | stats count() by service | sort -count | limit 10"}.
A query starts with a source (logs or spans) and continues with stages separated by | (a | inside quotes does not split). Each stage reads the result of the one before. The query compiles to SQL, one CTE per stage, and runs through the tenant-safe SQL surface (crates/wt-sql): project scope, time range, allow-listed functions and memory and concurrency limits all apply.
Stages
Stage | Meaning |
|---|---|
| filter; the KQL syntax from |
| aggregate; see |
| order rows |
| keep N rows, 1 to 10,000 (default and cap of the response is 1,000 unless a larger |
| keep only these columns |
| add a column: numbers, strings, fields, |
| extract text between literals into new columns (up to 8 captures) |
| new string columns from the named groups of an RE2 regex (1 to 8 groups, regex at most 512 bytes; backslashes are kept as written) |
| merge the keys of the field (at most 32 per row) into the attributes, so |
Columns: logs have ts_ns, timestamp, service, severity_number, severity_text, body, trace_id, span_id; spans have start_ns, start, service, name, kind, duration_ns, status_code, status_message, trace_id, span_id, parent_span_id. Other names read log or span attributes (resource.name for resource attributes) until stats or fields replaces the columns. Names with special characters can be written in backticks. stats names its outputs count, sum_x, avg_x, and so on, or by as; timeslice produces bucket.
Regular expressions
match() and parse patterns are RE2 and at most 1,024 bytes. parse builds the expression itself: literal text is escaped, %{name} matches any text (the last one greedy, the others lazy).
Errors
Syntax errors return 422 with a fixed message and the character position. An unknown stage reports the position of its name, for example syntax error at 7: unknown stage; use where, stats, sort, limit, fields, eval, parse or extract. At most 16 stages and 8 KiB of query text.