Profiling platform coverage

Embed
Preview of the card
<a href="https://forumulate.com/p/watchtower/docs/profiling-support"><img src="https://forumulate.com/api/public/orgs/watchtower/spaces/docs/pages/f8681c1f-0599-4e89-8386-ed46690d3c36/social-card.svg?v=3e7ae33eec8b" alt="Profiling platform coverage" width="600" height="315"></a>

What Watchtower accepts, what it shows for each runtime, and what it does not do. Recipes per runtime are in ingest/profilers. Ingest endpoints are in ingest/profiles.

Runtimes

Runtime

Sample types

Ingest path

Symbolization

Span links

Go (first-party go/profiler)

cpu, heap (alloc and inuse), goroutine, mutex, block

pprof push to /v1/profiles

in the binary (pprof carries names)

yes, span_id label via WithSpan

JVM

cpu, wall, alloc, lock (async-profiler)

folded stacks via wtctl profile upload

names in the output

no

Python

cpu (py-spy)

folded stacks via wtctl profile upload

names in the output

no

Ruby

cpu (rbspy)

folded stacks via wtctl profile upload

names in the output

no

.NET

cpu (dotnet-trace converted to folded)

folded stacks via wtctl profile upload

names in the output

no

Node.js

cpu (V8 .cpuprofile converted to folded)

wtctl profile convert-cpuprofile then upload

names in the output

no

Rust, C, C++, any Linux binary

cpu (perf)

folded stacks via wtctl profile upload

debug files uploaded with wtctl sourcemaps

no

Vendor and standard interop

Source

Accepted

Notes

pprof (gzip or raw)

yes

POST /v1/profiles?format=pprof

Pyroscope /ingest

yes

compatible, not Pyroscope; multipart uploads are not accepted

OTLP profiles

yes

HTTP protobuf and gRPC; JSON is not accepted yet

async-profiler collapsed, py-spy raw, rbspy collapsed, perf script

yes

as folded stacks

Datadog, Elastic and Grafana profilers

not natively

their agents speak private wire formats tied to their backends; push their pprof or folded export instead

Requirement coverage

Id

Capability

Status

PRF-001

Profile diff and comparison

query API and compare view

PRF-002

Trace-to-profile linking

span_id / trace_id sample labels

PRF-003

Profile regression issues

regression detection over the function rollup

PRF-004

Continuous profiling policy

per-project policy, rules and remote config

PRF-005

Flame graph views

flame, table, call graph

PRF-006

Symbolization and debug info

debug files uploaded per project

PRF-007

Profile ingest formats

pprof, OTLP, folded, Pyroscope

PRF-008

Whole-system eBPF profiler

planned (Linux only, privileged)

PRF-009

Memory and allocation profiling

alloc and inuse sample types, leak candidates

PRF-010

Off-CPU and lock profiling

sample types stored and queried; eBPF collection with PRF-008

PRF-011

Language profilers

Go first-party; other runtimes through the profilers above

PRF-012

Profile storage and retention

see below

PRF-013

Profiler platform coverage

this page

Retention, archive and deletion

  • Retention. ClickHouse rows expire through a TTL on the project's retention_days for profiles. A daily worker job deletes the raw originals in object storage that are older than the same number of days.

  • Archive. Set WT_PROFILE_ARCHIVE_DAYS on the worker to move profile_samples_v1 partitions older than that many days to the ClickHouse storage volume named cold. Default 0 is off. If the table's storage policy has no cold volume the job does nothing; configuring the policy is the operator's job.

  • Deletion. DELETE /v1/projects/{p}/profiles?before=<RFC 3339>[&service=<name>] (role with DeleteTelemetry) answers 202 with a deletion_id and writes an audit event. Guarantees: the audit event is recorded before anything is deleted; ClickHouse applies the row deletes as asynchronous mutations, so rows can stay visible for a short time after the 202; raw originals are deleted right after the mutations are queued, and a failed object delete is picked up by the daily retention job. Deletion does not reach backups; they age out on their own schedule.

Depth additions

  • JFR. Native recordings are parsed (POST /v1/profiles?format=jfr).

  • Sample types. gc_pause, exceptions and native_alloc join cpu, wall, alloc, inuse, offcpu and lock.

  • Per-type retention. A row in profile_type_retention_v1 lowers the retention of one sample type for a project; it never raises the project's retention.

  • Timeline and heatmap. GET /v1/projects/{p}/profiles/timeline?service=&by=thread|host|container returns value per time bucket for each label value (at most 50 rows and 2000 buckets).

  • Insights. CPU burst (over 75% for 10 s), burst saturation (a full core for 1 s) and starvation (no samples for 100 ms between busy buckets) cards.

  • Lock contention. The eBPF agent has uprobes for pthread mutex and rwlock waits (go/agent/ebpf_profiler); loader wiring and the object build are pending.

  • Hide packages. The explorer hides frames by prefix per service (browser setting); hidden time moves to the nearest visible caller.

Gaps

  • Windows ETW.

  • Symbol servers (debug files are uploaded, not fetched).

Updated