Tag: debugging

Observability tools

Noting down some helpful sources 🙂

Images

Images

PackageProvides
procpsps(1), vmstat(8), uptime(1), top(1)
util-linuxdmesg(1), lsblk(1), lscpu(1)
sysstatiostat(1), mpstat(1), pidstat(1), sar(1)
iproute2ip(8), ss(8), nstat(8), tc(8)
numactlnumastat(8)
linux-tools-common linux-tools-$(uname -r)perf(1), turbostat(8)
bcc-tools (aka bpfcc-tools)opensnoop(8), execsnoop(8), runqlat(8), runqlen(8), softirqs(8), hardirqs(8), ext4slower(8), ext4dist(8), biotop(8), biosnoop(8), biolatency(8), tcptop(8), tcplife(8), trace(8), argdist(8), funccount(8), stackcount(8), profile(8), and many more
bpftracebpftrace, basic versions of opensnoop(8), execsnoop(8), runqlat(8), runqlen(8), biosnoop(8), biolatency(8), and more
perf-tools-unstableFtrace versions of opensnoop(8), execsnoop(8), iolatency(8), iosnoop(8), bitesize(8), funccount(8), kprobe(8)
trace-cmdtrace-cmd(1)
nicstatnicstat(1)
ethtoolethtool(8)
tiptoptiptop(1)
msr-toolsrdmsr(8), wrmsr(8)
github.com/brendangregg/msr-cloud-toolsshowboost(8), cpuhot(8), cputemp(8)
github.com/brendangregg/pmc-cloud-toolspmcarch(8), cpucache(8), icache(8), tlbstat(8), resstalls(8)

  • vmstat(8): Virtual and physical memory statistics, system-wide
  • mpstat(1): Per-CPU usage
  • iostat(1): Per-disk I/O usage, reported from the block device interface
  • nstat(8): TCP/IP stack statistics
  • sar(1): Various statistics; can also archive them for historical reporting
  • ps(1): Shows process status, shows various process statistics, including memory and CPU usage.
  • top(1): Shows top processes, sorted by CPU usage or another statistic.
  • pmap(1): Lists process memory segments with usage statistics.
  • perf(1): The standard Linux profiler, which includes profiling subcommands.
  • profile(8): A BPF-based CPU profiler from the BCC repository (covered in Chapter 15, BPF) that frequency counts stack traces in kernel context.
  • Intel VTune Amplifier XE: Linux and Windows profiling, with a graphical interface including source browsing.
  • gprof(1): The GNU profiling tool, which analyzes profiling information added by compilers (e.g., gcc -pg).
  • cachegrind: A tool from the valgrind toolkit, can profile hardware cache usage (and more) and visualize profiles using kcachegrind.
  • Java Flight Recorder (JFR): Programming languages often have their own special-purpose profilers that can inspect language context. For example, JFR for Java.
  • tcpdump(8): Network packet tracing (uses libpcap)
  • biosnoop(8): Block I/O tracing (uses BCC or bpftrace)
  • execsnoop(8): New processes tracing (uses BCC or bpftrace)
  • perf(1): The standard Linux profiler, can also trace events
  • perf trace: A special perf subcommand that traces system calls system-wide
  • Ftrace: The Linux built-in tracer
  • BCC: A BPF-based tracing library and toolkit
  • bpftrace: A BPF-based tracer (bpftrace(8)) and toolkit
  • strace(1): System call tracing
  • gdb(1): A source-level debugger
  • perf stat: performance counter statistics
Images

GDB does not load debugging symbols although they are present

I have a binary which was compiled with gcc and debugging symbols enabled:

# file binary
binary: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=f5902695bdca690e84987fb377b69d16e3b47829, not stripped

Using objdump -syms binary I can also see the debugging symbols. However, GDB does not load them when I run gdb ./binary:

Reading symbols from ./binary...(no debugging symbols found)...done.
(gdb) list
No symbol table is loaded.  Use the "file" command.

Why is this happening and how can I load the debugging symbols?

Solution:

Why is this happening and how can I load the debugging symbols?

This is most likely happening because in fact the library does not have debugging symbols.

file binary
... not stripped

Above output does not indicate that the binary has debugging symbols, only that it has a symbol table. So does this: objdump -syms.

To really see debugging symbols, do this: readelf -wi binary (I predict you wouldn’t see any).

If debug symbols are in fact present, you should see something like this:

$ readelf -wi ./a.out
Contents of the .debug_info section:

  Compilation Unit @ offset 0x0:
   Length:        0x4e (32-bit)
   Version:       4
   Abbrev Offset: 0x0
   Pointer Size:  8
 <0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
    <c>   DW_AT_producer    : (indirect string, offset: 0x5): GNU C11 7.3.0 -mtune=generic -march=x86-64 -g
    <10>   DW_AT_language    : 12   (ANSI C99)
    <11>   DW_AT_name        : t.c
    <15>   DW_AT_comp_dir    : (indirect string, offset: 0x0): /tmp
    <19>   DW_AT_low_pc      : 0x5fa
    <21>   DW_AT_high_pc     : 0xb
    <29>   DW_AT_stmt_list   : 0x0
 <1><2d>: Abbrev Number: 2 (DW_TAG_subprogram)
    <2e>   DW_AT_external    : 1
    <2e>   DW_AT_name        : (indirect string, offset: 0x33): main
    <32>   DW_AT_decl_file   : 1
    <33>   DW_AT_decl_line   : 1
    <34>   DW_AT_type        : <0x4a>
    <38>   DW_AT_low_pc      : 0x5fa
    <40>   DW_AT_high_pc     : 0xb
    <48>   DW_AT_frame_base  : 1 byte block: 9c     (DW_OP_call_frame_cfa)
    <4a>   DW_AT_GNU_all_call_sites: 1
 <1><4a>: Abbrev Number: 3 (DW_TAG_base_type)
    <4b>   DW_AT_byte_size   : 4
    <4c>   DW_AT_encoding    : 5    (signed)
    <4d>   DW_AT_name        : int
 <1><51>: Abbrev Number: 0

How to switch gdb byte output grouping

When I try to see what’s inside stack space, I use the following command:

x/100x $sp

However, sometimes output is formatted like this, grouping by 4 bytes:

0xbffff0ac: 0x00000000  0xb7fbc000  0xb7fbc000  0xbffff4e8
...

While sometimes I get this:

0xbffff0ac: 00 00 00 00 00 c0 fb b7 00 c0 fb b7 e8 f4 ff bf

But I can’t determine how to switch between these formats and how gdb desides which format to use for output. Any suggestions?

Solution:

sometimes output is formatted like this
However, sometimes output is formatted like this

This is because the x command remembers the last size you used.

If you want particular size with your x, just specify it directly:

(gdb) x/100wx $sp

Documentation.