Log index tuning

Embed
Preview of the card
<a href="https://forumulate.com/p/watchtower/docs/log-index-tuning"><img src="https://forumulate.com/api/public/orgs/watchtower/spaces/docs/pages/711540ee-859b-4742-85dd-f938f2654a36/social-card.svg?v=3e7ae33eec8b" alt="Log index tuning" width="600" height="315"></a>

logs_v1 carries skip indexes so free-text search reads few granules.

Index

Expression

Serves

idx_body

lower(body) tokenbf

single-token search: hasToken(lower(body), lower(term))

body_ngram

lower(body) ngrambf(4)

substring search (terms of 4+ characters)

idx_attr_keys, attr_vals

mapKeys(attrs), mapValues(attrs)

attribute filters

idx_trace

trace_id bloom

trace id lookups

Search terms that are one alphanumeric token compile to hasToken(lower(body), lower(term)); anything else becomes position(lower(body), lower(term)) > 0. The indexed expression must match exactly, so keep the lower(...) wrapper in custom queries.

Old data

New parts carry the indexes. The janitor job index_materialize builds them on older parts, one partition per tick, oldest first, skipping the newest 2 partitions, one MATERIALIZE INDEX at a time. It is safe to run repeatedly: a partition whose parts all have index marks is not touched.

Merge policy

wt_store_ch::settings::LOGS_PROFILE sets min_age_to_force_merge_seconds = 3600 and a max_bytes_to_merge_at_max_space_in_pool cap through ALTER TABLE logs_v1 MODIFY SETTING. Only settings that differ are altered, so applying twice is a no-op. Verify index use with EXPLAIN indexes = 1.

Updated