Lucene and KQL log filters

Embed
Preview of the card
<a href="https://forumulate.com/p/watchtower/docs/lucene-kql"><img src="https://forumulate.com/api/public/orgs/watchtower/spaces/docs/pages/fb6cc063-771a-4aea-b0b0-0dc66c7e6cf6/social-card.svg?v=3e7ae33eec8b" alt="Lucene and KQL log filters" width="600" height="315"></a>

POST /v1/projects/{p}/query/search with {"language": "lucene" | "kql", "q": "...", "from", "to"} returns the newest matching log lines (at most 1,000). Both languages compile to SQL over the logs table and run through the tenant-safe SQL surface, so the project scope, the time range, the function allow-list, the memory limit and the concurrency caps are the same as for /query/sql.

Fields: service, severity_text (severity, level), severity_number, body (message, msg), trace_id, span_id, ts_ns. Any other name is a log attribute; resource.name reads a resource attribute. Words without a field search the log body, ignoring case. severity_text is compared ignoring case.

Lucene

Syntax

Meaning

error

body contains error

service:api

field equals value

"connection reset"

phrase in the body (exact value on other fields; for a substring use field:*text*)

msg:err*, msg:e?r

wildcards (* any run, ? one character); escape with \

status:[500 TO 599], n:{1 TO *}

range, [ ] inclusive, { } exclusive, * open

a AND b, a OR b, NOT a, &&, ||, !a

boolean operators (uppercase words)

+a -b c

required, prohibited, optional

f:(a OR b)

field applied to a group

_exists_:field

the field is present

Clauses written next to each other are OR (the Lucene default). When a query has required (+) clauses, the optional ones are ignored. Numeric ranges compare numbers when both bounds are numbers; otherwise they compare text.

KQL

Syntax

Meaning

service: api

field equals value

service: (api or web)

field applied to a group

level: error and not service: web

and, or, not in any case; not binds tightest, then and, then or

status >= 500, <, <=, >

comparison

user: { name: bob and id: 7 }

nested fields, meaning user.name and user.id

field: *

the field is present

match(field, "RE2")

RE2 regular expression, at most 1,024 bytes; ClickHouse runs RE2 in linear time

Limits

Queries are at most 8 KiB, 64 terms and 16 levels deep. Syntax errors return 422 with a fixed message and the character position (syntax error at 5: expected a term). Unknown fields are treated as attributes; a number field given text, or a pattern on a number field, returns 422.

Updated