Trace query language and funnels
Embed
<a href="https://forumulate.com/p/watchtower/docs/traces"><img src="https://forumulate.com/api/public/orgs/watchtower/spaces/docs/pages/28e4ae77-2f10-4d08-bb12-62fda40e7a66/social-card.svg?v=3e7ae33eec8b" alt="Trace query language and funnels" width="600" height="315"></a>POST /v1/projects/{project}/query/traces with {"q": "...", "from": "...", "to": "...", "limit": 20}. The range defaults to the last hour and is clamped to trace retention and 31 days.
Spansets
{ <condition> } selects spans. Conditions combine with &&, || and parentheses.
Field | Operators | Values |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
resource.service.name is the span's service. {} matches every span. Regular expressions must compile (at most 1 KiB) and are checked before the query runs.
Structure
{A} > {B}: the B spans whose parent is an A span.{A} >> {B}: the B spans with an A ancestor.{A} ~ {B}: the B spans that share a parent with a different A span.
Operators chain left to right: {A} >> {B} > {C}.
Pipeline
| count() > 3 and | avg(duration) > 200ms (also min, max, sum) keep a trace only when the matched spans pass the test. | select(span.http.url, name) adds those values to each matched span.
Example: { resource.service.name = "checkout" } >> { status = error } | select(span.http.url)
Funnels
funnel({ name = "cart" }, { name = "pay" }, { status = ok }) (2 to 8 steps). Step k counts traces in which spans s1..sk exist, s_i matching step i, each a descendant of the previous span or starting at or after its end. Counts never increase from step to step. Each step lists up to 5 example trace ids. Funnels work inside one trace only: there is no cross-trace or cross-session join.
Limits
Query text 4 KiB, 512 tokens, 16 spansets, 32 conditions, nesting 8. Up to 10,000 candidate traces and 1,000 spans per trace are read (300,000 spans in all); when a cap is hit the response sets truncated: true and counts are lower bounds; when the span cap is hit the oldest traces are the ones left out. A trace with more than 1,000 spans is marked spans_truncated: true because its parent links may be incomplete. Statements run with the same memory and thread limits as query/sql, share one time budget, and count against the same per-project (4) and global (8) concurrency limits (429 beyond). A result over the 64 MiB cap gives 422 result too large. Regular expressions use RE2 syntax. Syntax errors return 422 with a fixed message and an offset, never the query text.