Problem
ddprof bundles blazesym 0.2.0-rc.5 (via libdatadog v26), which does not support finding split debug files via build-id paths (/usr/lib/debug/.build-id/XX/rest.debug). It only supports GNU debuglink-style paths (/usr/lib/debug/<binary-path>/<debuglink-name>).
On modern Debian/Ubuntu systems, debug symbols are distributed as automatic -dbgsym packages which exclusively use build-id paths. This is the standard mechanism since debhelper compat level 9+ (see Debian AutomaticDebugPackages and Ubuntu Debug Symbol Packages).
dh_strip creates the -dbgsym packages with debug files at:
/usr/lib/debug/.build-id/XX/YYYYYY.debug
It also sets .gnu_debuglink in the stripped binary, but the debuglink filename is the build-id hash (e.g. 2436a298407b13414d190e466b5dcd5df8f462.debug), stored only at the build-id path. There is no file at the debuglink search paths (/usr/lib/debug/usr/bin/<name>), so blazesym's debuglink search silently fails.
The result is that ddprof flame graphs show the binary name (e.g. "IndexBuilderServer") instead of function names for stripped binaries, even with DD_PROFILING_INLINED_FUNCTIONS=true. Dynamic library symbols (libc, grpc, etc.) work because they have .dynsym entries — but the main binary's own functions are only in .symtab inside the debug file that blazesym cannot find.
Fix
blazesym v0.2.3 (Jan 29, 2025) added build-id debug file lookup in libbpf/blazesym#1415 (commit 8db66bc). From the v0.2.3 changelog:
Added
- Added support for
/usr/lib/debug/.build-id debug link target directory
Updating the bundled blazesym (via libdatadog) to >= 0.2.3 would fix this issue.
Workaround
Until the update, users can create symlinks from the debuglink search path to the build-id files, e.g. by overriding dh_strip in debian/rules to add symlinks like:
/usr/lib/debug/usr/bin/<debuglink-name> → ../../.build-id/XX/rest.debug
References
Problem
ddprof bundles blazesym 0.2.0-rc.5 (via libdatadog v26), which does not support finding split debug files via build-id paths (
/usr/lib/debug/.build-id/XX/rest.debug). It only supports GNU debuglink-style paths (/usr/lib/debug/<binary-path>/<debuglink-name>).On modern Debian/Ubuntu systems, debug symbols are distributed as automatic
-dbgsympackages which exclusively use build-id paths. This is the standard mechanism since debhelper compat level 9+ (see Debian AutomaticDebugPackages and Ubuntu Debug Symbol Packages).dh_stripcreates the-dbgsympackages with debug files at:It also sets
.gnu_debuglinkin the stripped binary, but the debuglink filename is the build-id hash (e.g.2436a298407b13414d190e466b5dcd5df8f462.debug), stored only at the build-id path. There is no file at the debuglink search paths (/usr/lib/debug/usr/bin/<name>), so blazesym's debuglink search silently fails.The result is that ddprof flame graphs show the binary name (e.g. "IndexBuilderServer") instead of function names for stripped binaries, even with
DD_PROFILING_INLINED_FUNCTIONS=true. Dynamic library symbols (libc, grpc, etc.) work because they have.dynsymentries — but the main binary's own functions are only in.symtabinside the debug file that blazesym cannot find.Fix
blazesym v0.2.3 (Jan 29, 2025) added build-id debug file lookup in libbpf/blazesym#1415 (commit
8db66bc). From the v0.2.3 changelog:Updating the bundled blazesym (via libdatadog) to >= 0.2.3 would fix this issue.
Workaround
Until the update, users can create symlinks from the debuglink search path to the build-id files, e.g. by overriding
dh_stripindebian/rulesto add symlinks like:References
-dbgsympackages work