eBPF zero-code tracing and network telemetry

Embed
Preview of the card
<a href="https://forumulate.com/p/watchtower/docs/ebpf-tracing"><img src="https://forumulate.com/api/public/orgs/watchtower/spaces/docs/pages/683b5458-acfe-4e0b-91b1-5e1bcc009f3f/social-card.svg?v=3e7ae33eec8b" alt="eBPF zero-code tracing and network telemetry" width="600" height="315"></a>

Two wt-agent modules, Linux only, both disabled by default and enabled separately in the agent's local config. Requirements as for the eBPF profiler: kernel 5.8 or newer with BTF, root or CAP_BPF + CAP_PERFMON (DNS capture also needs CAP_NET_RAW). Preflight reports what is missing. Build the objects on a Linux host with make ebpf-gen (http, tls, tcp) and install them into object_dir (default /usr/lib/wt-agent/ebpf). Other platforms compile stubs.

ebpf_trace (APM-010, TRC-020)

Captures the first 512 bytes of socket reads and writes (syscall tracepoints on write, sendto, read, recvfrom) and of TLS plaintext (uprobes on OpenSSL SSL_write/SSL_read and Go crypto/tls.(*Conn).Write/Read). It parses HTTP/1.x, HTTP/2 (HPACK), gRPC, PostgreSQL, MySQL and Redis heads and emits OTLP CLIENT and SERVER spans through the agent's exporter (/v1/traces). Spans carry http.request.method, url.path, http.response.status_code, db.system, db.query.text, server.address, server.port, rpc.system, telemetry.sdk.name=wt-ebpf and wt.instrumentation=ebpf.

What never leaves the host:

  • Headers other than host, :authority, traceparent, content-type, grpc-status are skipped without being copied, so authorization and cookie are never read.

  • Query strings and fragments are cut from paths. Bodies are never captured.

  • SQL is normalized on the host with the same rules as the worker's normalize_sql (literals become ?, IN lists collapse). Parameters never leave. Redis exports the command name only.

Trace context: a W3C traceparent header, when present, supplies the trace id and parent span id, so eBPF spans join traces from instrumented services. Without traceparent a span is its own root with a fresh trace id. Spans are never stitched into a parent/child trace by timing or connection guesses; they still show up as service-graph edges through server.address.

Config keys: enabled, max_events_per_sec (default 20000; events over the cap are dropped and counted, logged once per change), ring_buffer_kb (default 4096, power of two), tls (default true), object_dir.

Limits: writev/sendmsg/recvmsg and io_uring are not traced (Node, JVM NIO); JVM and Node TLS libraries beyond OpenSSL and Go crypto/tls are not covered; the Go Read return probe finds RET instructions by byte scan on amd64 (see the shortcut comment); an HTTP/2 header block cut by the 512 byte window desyncs that direction's HPACK table; the agent's own traffic is ignored.

ebpf_net (INF-038)

A sock:inet_sock_set_state tracepoint emits one event per closed TCP connection (bytes, retransmits, smoothed RTT, connect latency, failure, reset) read from the kernel tcp_sock with CO-RE. Every interval (default 10 s) the module aggregates per (src workload, dst workload, port) and exports cumulative sums wt.net.tcp.bytes_sent, bytes_received, retransmits, resets, connections, connect_failures and gauges wt.net.tcp.rtt, wt.net.tcp.connect_latency. Attributes: src.workload, dst.workload, server.port, server.address (the evidence the infra service graph uses). Connect failures also produce WARN log events. IPv4 DNS (UDP/53) is observed with a packet socket: wt.net.dns.queries and wt.net.dns.latency by query type and response code, with failures logged.

Attribution: the connecting side is named from its pid (container/<id12>, as the eBPF profiler names services, else the process name). The accepting side is named from the host's listening-port table. A passive connection whose peer is on the same host is skipped because the connecting process reports it, so loopback traffic is counted once. Config keys: enabled, interval, max_events_per_sec, ring_buffer_kb, dns, object_dir.

Updated