eBPF whole-system profiler (ebpfprofiler)

A wt-agent module that profiles every process on a Linux host without code changes: on-CPU (PRF-008) and off-CPU (PRF-010). It is off by default.

Requirements

• Linux 5.8 or newer with BTF (/sys/kernel/btf/vmlinux).

• Root, or CAP_BPF + CAP_PERFMON (CAP_SYS_ADMIN also works). Preflight() names what is missing.

• The compiled objects cpu_bpfel.o, cpu_dwarf_bpfel.o, offcpu_bpfel.o, offcpu_dwarf_bpfel.o and rtregs_bpfel.o. Build them on a Linux host with make ebpf-gen (clang and libbpf headers) and install them into object_dir (default /usr/lib/wt-agent/ebpf). They are not built on macOS or Windows; on those the module compiles to a stub and reports "unsupported".

Enable

The module must be defined in the agent's own config; remote config can only toggle it and set the interval.

modules:
  ebpf_profiler:
    enabled: true
    sample_hz: 19            # per CPU
    upload_interval_s: 60
    offcpu: true
    offcpu_min_ms: 1         # blocked intervals shorter than this are dropped
    strip_process_args: true # default; only the executable base name is sent
    plain_pid: false         # default; pid is sent as a hash

What is sent

Every upload_interval_s the module posts one gzip pprof per service (container, otherwise executable) to /v1/profiles with source=ebpf and sample type cpu or offcpu. Labels: host, container, k8s.pod, process.executable, pid (hashed). The command line is sent only when strip_process_args: false.

User frames carry module path, GNU build id and module-relative address; the server symbolizes them. Kernel frames are resolved locally from /proc/kallsyms (root sees addresses; otherwise frames stay unresolved). JIT frames resolve from /tmp/perf-<pid>.map: Java with perf-map-agent, Node with --perf-basic-prof, Python 3.12 with the perf trampoline.

Limits

• On x86-64 and arm64, user stacks are unwound with each binary's .eh_frame / .debug_frame (dwarf_unwind, default on), so binaries built without frame pointers get whole stacks, up to 64 frames. A process gets its table from the second upload window on. On Linux 5.15+ this also covers CPU samples taken in the kernel; before that they use frame pointers, as do JIT code and hosts missing cpu_dwarf_bpfel.o.

• Off-CPU stacks use frame pointers by default. offcpu_dwarf: true walks them with the same tables on Linux 5.15+. That walk runs on every blocking context switch of a process with a table, and its overhead is unmeasured, so turn it on only where you can watch the host.

• Runtime stacks are read from process memory at 19 Hz with process_vm_readv, which needs root or CAP_SYS_PTRACE. The target is never stopped or written to, and frames that fail validation are dropped. Reads are capped per sampling tick, so a large process can yield partial samples, and each window over cpu_budget halves the cap. Overhead on hosts is unmeasured.

  Runtime	Config	Profile type	Versions	Refused
  CPython	python_stacks (default on)	python_wall	3.13 to 3.15	Free-threaded builds; builds failing the self-check
  CPython	python_stacks plus runtime_stacks: ["python_legacy"]	python_wall	3.11, 3.12	3.10 and older; builds failing the self-check
  Ruby	runtime_stacks: ["ruby"]	ruby_wall	3.3, 3.4, 4.0, all ractors	Stripped libruby (ruby_current_vm_ptr is a local symbol); other versions
  PHP	runtime_stacks: ["php"]	php_wall	8.2 to 8.5, NTS	ZTS builds; other versions
  Node	runtime_stacks: ["node"]	node_cpu	Builds with v8dbg_* symbols	Pointer-compressed builds; stripped builds
  JVM (HotSpot)	runtime_stacks: ["jvm"]	jvm_wall	JDK 11+ exporting gHotSpotVM*	JDK 8; builds missing a needed field
  .NET (JIT frame names)	runtime_stacks: ["dotnet"]	on-CPU and off-CPU stacks	.NET 8+ with its diagnostics socket in /tmp	DOTNET_EnableDiagnostics=0; a TMPDIR other than /tmp; .NET 7 and older (set DOTNET_PerfMapEnabled=1 instead)

  A refusal is logged once at info level with its reason. Node and on-CPU JVM stacks also need rtregs_bpfel.o; without it, JVM stacks cover only threads outside Java code. Node stacks follow frame pointers, so they end at C++ code built without them. JVM stacks name inlined methods from the nmethod's debug info (a layout that fails its checks falls back to the compiled outer method) and continue below a native-to-Java call from the call wrapper's saved frame; the native frames in between are not shown. For .NET, dotnet asks each process once, through the runtime's diagnostics socket, to write a perf map; frames jitted before that are named from the next upload window on. There is no reader of the runtime's own tables, so without the socket the routes are the SDK's IPC profiler and DOTNET_PerfMapEnabled=1.

• Overhead guard: if the agent's own CPU exceeds 1% of wall time over an upload window, the sample rate is halved (floor 1 Hz) and a warning is logged. The guard measures the agent process, not kernel time spent in the BPF programs; measured overhead is known gap G-16.

• Off-CPU time counts blocked (non-runnable) time only; preemption is not counted.

• Lock-contention uprobes (futex, pthread_mutex) are not built.

• Offsets: user addresses are addr - mapping start + file offset; the server maps them with the ELF program headers.