Introduce {attach|detach}_kfunc API#2726
Conversation
|
[buildbot, ok to test] |
|
[buildbot, test this please] |
|
All tests failed. All the buildbot is pretty old. The latest one in buildbot is fc28 which I do not think supports BTF. For example, the following is the error message: We will need to guard the new test with kernel version and availability of kernel BTF. Not all distro's have kernel BTF, I guess. |
|
ok, I expected some failures, but not that bad.. will check on some BTF detection for tests |
| """ | ||
|
|
||
| bpf_text_kprobe = """ | ||
| BPF_HASH(infotmp, u64, struct val_t); |
There was a problem hiding this comment.
This BPF_HASH is redundant. This will cause compilation error for kprobe case.
There was a problem hiding this comment.
ugh, I must have consused my versions and pushed the wrong one,
I recall fixing this, sry.. will fix in next version
|
@olsajiri any update on this one? |
|
Having the kretfunc retval in args[N] might get pretty confusing. Could it have a separate variable called retval for this? (I was going to suggest putting retval in args[0] and bumping the others along by one, but that would make it inconsistent with kfunc.) |
|
sry, I'm out this week, hence my lag.. I think you're right, we can add some better interface jirka |
|
Oh we could adopt libbpf convention as well. Some macros, so we do not need to add BTF analysis code although it is nice to have to double check. In the long time, I intend to change bcc to be more libbpf friendly so we can get a lot of libbpf funcitonality for free. |
ok, that might be actually easier.. will check thanks, |
|
@yonghong-song actually while working on bpftrace support, If this kind of interface is ok with you - getting the attr info from |
We need to use LIBBPF_LIBRARIES variable instead of just using bpf for libbpf linking, so it user specified library gets linked in. Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Currently we include headers from local libbpf subpackage, which does not work if user specify LIBBPF_INCLUDE_DIR. Adding HAVE_EXTERNAL_LIBBPF macro, that gets defined when user specifies libbpf header path. This macro is checked in bcc_libbpf_inc.h and proper files are included. Using bcc_libbpf_inc.h in place of libbpf includes. Signed-off-by: Jiri Olsa <jolsa@kernel.org>
The TRACING/EXT programs use attach_btf_id and attach_prog_fd fields from struct bpf_load_program_attr. The attach_prog_fd field shares space with kern_version, so by setting kern_version unconditionally we also set attach_prog_fd to bogus value and kernel fails the load because it tries to look it up. Setting kern_version only for programs other than TRACING/EXT type. Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Kernel added new probe called trampoline allowing to probe
almost any kernel function when BTF info is available in
the system.
Adding the interface to define trampoline function for given
kernel function via BPF_PROG macro, like:
KFUNC_PROBE(do_sys_open, int dfd, const char *filename, int flags, int mode)
{
...
}
which defines trampoline function with the 'kfunc__do_sys_open'
name, that instruments do_sys_open kernel function before the
function is executed.
or:
KRETFUNC_PROBE(do_sys_open, int dfd, const char *filename, int flags, int mode, int ret)
{
...
}
which defines trampoline function with the 'kfunc__do_sys_open'
name, that instruments do_sys_open kernel function after the
function is executed.
The main benefit is really lower overhead for trampolines (please
see following commit for klockstat.py with perf comparison).
Another benefit is the ability of kretfunc probe to access
function arguments, so some tools might need only one program
instead of entry/exit ones (please see following commit for
opensnoop.py changes).
Currently the interface does not allow to define function
of different name than:
kfunc__<function_name> or kretfunc__<function_name>
which is sufficient by now, and can be easily changed in future
if needed.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Adding support_kfunc function to BPF object to return state of kfunc trampolines support. Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Adding kfunc return trampoline probe if available instead of
kprobe/kretprobe probes.
The return trampoline has access to function entry arguments,
so we are good with just single eBPF program.
The kfunc trampolines are also faster - less intrusive.
Below are stats for compiling linux kernel while running
opensnoop.py on the background for kprobes and kfuncs.
Without opensnoop.py:
Performance counter stats for 'system wide':
849,741,782,765 cycles:k
162.235646336 seconds time elapsed
With opensnoop.py using kprobes:
Performance counter stats for 'system wide':
941,615,199,769 cycles:k
164.355032879 seconds time elapsed
With opensnoop.py using trampolines:
Performance counter stats for 'system wide':
913,437,005,488 cycles:k
163.742914795 seconds time elapsed
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Adding kfunc return trampoline probe if available instead of kprobe/kretprobe probes. The kfunc trampolines are faster - less intrusive. The benchmark without klockstat.py script on background: $ perf bench sched messaging -l 50000 # Running 'sched/messaging' benchmark: # 20 sender and receiver processes per group # 10 groups == 400 processes run Total time: 18.571 [sec] With kprobe tracing: $ perf bench sched messaging -l 50000 # Running 'sched/messaging' benchmark: # 20 sender and receiver processes per group # 10 groups == 400 processes run Total time: 183.395 [sec] With kfunc tracing: $ perf bench sched messaging -l 50000 # Running 'sched/messaging' benchmark: # 20 sender and receiver processes per group # 10 groups == 400 processes run Total time: 39.773 [sec] Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Adding kfunc return trampoline probe if available instead of
kprobe/kretprobe probes.
The kfunc trampolines are faster - less intrusive.
Below are stats for running perf bench sched pipe benchamark while
running vfsstat.py on the background for kprobes and kfuncs.
With kprobes:
Performance counter stats for './perf bench sched pipe -l 5000000' (3 runs):
112,520,853,574 cycles:k
48.674 +- 0.672 seconds time elapsed ( +- 1.38% )
With kfuncs:
Performance counter stats for './perf bench sched pipe -l 5000000' (3 runs):
106,304,165,424 cycles:k
46.820 +- 0.197 seconds time elapsed ( +- 0.42% )
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
This should work. In bcc, we already have some other usage with special prefixes. |
cool, changes are already in the branch |
|
LGTM. Thanks! |
|
I get the following error in opensnoop with Linux 5.5.6. Is it a problem in bcc or in the kernel? |
|
Maybe this is a kernel issue? I mean kernel is not advanced enough to have this patch? |
|
Thanks for the pointer. Indeed my kernel does not have this patch. I wonder if I just noticed that the new kfunc code does not have the |
Kernel added new probe called trampoline allowing to probe
almost any kernel function when BTF info is available in
the system.
Adding the interface under 'kfunc' name, which allows to
probe function entry and 'kretfunc' name, which allows to
trace function return.
The main benefit is really low overhead which is really low
for trampolines (please see commit for klockstat.py
with perf comparison).
Another benefit is the ability of kretfunc probe to access
function arguments, so some tools might need only one program
instead of entry/exit ones (please see commit for
opensnoop.py changes).