I don't think it makes much sense to add a library to the codebase without actually using it for anything.

We could use libbpf to eventually replace libseccomp, which can't express some logic operations we'd like to do. Also I'm not so happy how system call names map to actual system call numbers. The downside is that we would have to do everything in systemd instead of relying on a bit higher level interface which libseccomp provides, but at least it would be C (-ish).

But this is just an additional potential side benefit of using libbpf.

systemd does not usually use submodules for it's dependency and that's something that linux distro usually don't want anyway

is there a particular reason why libpf needs to be used that way ?

@boucman this is actually mentioned in the commit message.

right, my bad...

It might be OK to link against libbpf, but this PR only adds the dependency, and doesn't do anything with it. Unless there's actually code that uses the new dep I don't think we should add the dep...

Thank you for your comments.
Agree this was a bit premature, my main purpose was to collect feedback.
So the plan is to actually replace micro-assembly of either src/core/bpf-devices.{h|c} or src/core/bpf-firewall.{h|c} or both with C programs.
There is a work in parallel to make *.rpm, so we may have libbpf as a regular external dependency, depends on how long it will take to make the package.

(So, to say this clearly: in generally agree with using libbpf instead of our handgrown BPF code. Would love to review a patch. Though I generally prefer shared lib deps over drop-in deps like the current version).

Looks like libbpf is not a standalone library (yet) and built from src:linux which bumps the soname on each new major kernel release.
Are there plans to stabilize the API/ABI of libbpf?
TBH, atm I would feel uneasy depending on such a fast moving target.

Looks like libbpf is not a standalone library (yet) and built from src:linux which bumps the soname on each new major kernel release.
Are there plans to stabilize the API/ABI of libbpf?
TBH, atm I would feel uneasy depending on such a fast moving target.

Well, this PR added it as git submodule, which pins the release at some git commit, and links against the thing statically, so that we get around the .so instability.

@mbiebl Actually there were a lot of changes in the last months to make libbpf API/ABI stable and versioned:

• DSO is now built with linker version script that versions symbols
• All private symbols are hidden by corresponding -fvisibility.
• SONAME is set to libbpf.so.$(VERSION) where $(VERSION) has never been changed so far (it's the first zero in the current libbpf version that is 0.0.2 now).
• ABI version in linker script can be changed at most once per kernel development cycle, but it shouldn't affect SONAME since only the minor version is usually changed (e.g. 2 in the current 0.0.2). Changing SONAME as usual may happen only as the last resort if no other options exist for some incompatible ABI change.

I'm not sure what is src:linux, I assume it's kernel tree? All the changes I mentioned should be in both kernel tree and github mirror (just updated the mirror to bring the last ones).

Please let me know if you think something else should be fixed in libbpf to consider its API/ABI stable (in addition to git submodule approach used here).

Libbpf would introduce a dependency on LLVM, so for architectures (if any) where only GCC is available, features using BPF would have to be disabled.

@rdna I see, thanks.
Yes I was looking at the linux source which apparently is used in Debian to build the libbpf binary packages. As you can see at
https://packages.debian.org/sid/amd64/libbpf4.19
https://packages.debian.org/sid/amd64/libbpf4.19/filelist
it includes the linux version in the soname.
After some further digging, that seems to be a Debian specific change though: https://salsa.debian.org/kernel-team/linux/merge_requests/74/diffs

hmm, if libbpf indeed introduces an llvm runtime dependency this would be less than ideal. @rdna do you know the story there?

@poettering libbpf does not introduce llvm runtime dependency.
E.g. existing use cases in systemd, BPF_PROG_TYPE_CGROUP_SKB and BPF_PROG_TYPE_CGROUP_DEVICE, provide stable kernel interface so that BPF programs can be compiled once and used whenever kernel supports these program types, i.e. it's only compile time dependency.

Workflow here may look like this:

• When systemd is being compiled, all BPF programs written in restricted C are compiled by llvm (with BPF target) into object files (Elf) first.
• Then those object files can either be included in systemd binary or distributed as separate files along with systemd in its package (whichever systemd prefers). By "included" I mean: generate includable C header with array that contains hexdump of BPF object file (similar to xxd -i file.o) so that user space C code in systemd can use this header to load that array with bpf_object__open_buffer. From my experience "include" method is more convenient since it doesn't require distributing additional files.
• In runtime systemd can load BPF programs from those already compiled object files (either included as buffers in systemd itself or separate files in systemd package) using libbpf that in turn will parse Elf and prepare BPF instructions ready to load in kernel (see bpf_object__open_buffer and bpf_object__load).

By talking about run time llvm dependency people usually have tracing in mind. In tracing case BPF programs should currently be built on target host since they may depend on specific kernel configuration (e.g. field offsets in a kernel struct that is used by tracing program and those offsets e.g. depend on some #defines). But this case is:

• currently solved separately by BCC (that btw started using libbpf recently, but also provides much more functionality on top of it);
• not intended to be handled by the work based on this PR (AFAIK systemd doesn't use BPF tracing);
• will be solved in the future by "Compile Once Run Everywhere" initiative that relies on BTF, so that even for tracing BPF program will need to be compiled once and all kernel version-/configuration- specifics will be handled by BTF.

I started to experiment with this PR because I was wondering if libbpf would somehow "automagically" help to fix #9666 (which is about test-bpf failing in lxc and docker containers or, more generally, about IPAddress[Allow|Deny] that are totally broken there (presumably due a kernel bug of some kind but nobody seems to be 100% sure)). The test still fails so it seems that the "not-our-bug" label is more or less justified :-)

Anyway, so far it has been relatively easy for people who don't care much about bpf filters to just basically ignore them. With libbpf as a submodule, it seems to be necessary to also install libelf and llvm to just build systemd (which, for example, totally breaks our fuzzers, as far as I can tell). It would probably be great if there was an easy way to just turn it off.

I also opened libbpf/libbpf#27 and libbpf/libbpf#28, that were uncovered here. I'm not sure whether it's the right place to report bugs in libbpf though.

Interestingly, turns out that currently only Travis CI is aware of submodules in general. I'm not saying it's hard to tweak the build scripts a little to make it work but are we sure we can't wait for libbpf to be released so it can be used as a regular dependency (given that, as far as I understand, it should happen relatively soon)?

Plus, my understanding is that this PR is supposed to make it easier to add new eBPF programs in the future, which is great, but I'm wondering if anyone is going to add something new in the foreseeable future. Wouldn't it be better to start to rely on libbpf in the process of implementing, I don't know, some feature that would utilize the library 100% (more or less) instead of only replacing the code that I agree isn't ideal but has apparently been good enough so far :-) ?

Before I forget, in https://discuss.lgtm.com/t/lgtm-seems-to-be-failing-to-extract-the-systemd-project-due-to-a-submodule/1994 I reported the LGTM failure so in theory LGTM should be able to analyze the PR soon.

@evverx
libbpf as a git submodule looked like a quick workaround, which turned out not to be the case given the test failures.
The RPM package is coming, once it is available tweaking build scripts wouldn't make much sense. That is not one man's effort and require some coordination, so can't give the time estimate but will report about the trackable progress in this discussion.
It's better to start with moving what already presents in systemd to libbpf due to the expectations for the existing code are known at least in some degree and we already have some tests for it. Replacing macro-assembly with something more human friendly is a noble goal on it's own :)
Also the point of this PR is to build a pipe, i.e. the necessary infra for having eBPF programs in systemd. What to run in the pipe is a subject of future PRs and it is open ended – eBPF is a powerful toy.

Turns out it was me who broke the part of semaphoreci dealing with submodules a couple of months ago. I've just fixed it and judging by https://semaphoreci.com/systemd/systemd/branches/pull-request-12151/builds/7 it got past meson and ninja and finally passed.

@wat-ze-hex

Replacing macro-assembly with something more human friendly is a noble goal on it's own :)

Can't argue with that and I also agree that it's useful to have the infrastructure in place and I wouldn't say that I'm strongly opposed to the change. It's just that there're places (the fuzzers for example) where it would be useful to turn everything related to eBPF off so I guess I'm fine as long as the fuzzers are intact (right now they are failing to compile).

@poettering's proposal to provide pre-compiled BPF would be perfect, since it would allow downstream to transition at their own pace, with (hopefully) minimal overhead for upstream.

I'm glad it turned out to be possible to find a reasonable compromise.

@boucman I'm wondering if anybody is willing to bring libbpf to Buildroot? I mean, if it isn't there there will be no point in shipping the generated bytecode to make it easier to avoid LLVM.

@evverx adding libbpf to Buildroot or any other embeded distribution is trivial compared to adding llvm support. It's just a simple library that need to be compiled. There is a feature check that needs an option to overwrite the result but embedded developers are used to stuff like that ...

@michaelolbrich so, maybe, it would make sense to delay shipping the compiled code until the library is added. At this point, it seems to me that we are trying to complicate things to cover theoretical use cases.

Just to clarify, while it's important to come up with something that will hopefully work everywhere, this PR seems to be a proof of concept (unless I'm mistaken) and it would be great to keep things simple to get the code working at least (currently LLVM isn't necessary at all because the macro assembly hasn't been replaced yet and libbpf is failing to compile on i386: libbpf/libbpf#34).

@boucman I'm wondering if anybody is willing to bring libbpf to Buildroot? I mean, if it isn't there there will be no point in shipping the generated bytecode to make it easier to avoid LLVM.

I'll add it to our buildroot todo list...

@michaelolbrich

@evverx adding libbpf to Buildroot or any other embeded distribution is trivial compared to adding llvm support. It's just a simple library that need to be compiled. There is a feature check that needs an option to overwrite the result but embedded developers are used to stuff like that ...

Just having libbpf wouldn't suffice, LLVM is still required to compile BPF programs. As @evverx pointed, no programs are yet here, this PR just introduced a few libbpf user space functions.
We either need 1. LLVM to compile BPF programs with the rest of systemd and then load populated bytecode into memory or 2. committed bytecode. Having bytecode committed is possible though this approach is not clean in a way that different LLVM versions may produce different bytecode and we have to have it in both big and little endian (see @rdna #12151 (comment)).

Regarding the actual BPF programs, rather than stack commits here, I'd prefer to open another PR once this PR have broken through the wall of failing tests.libbpf submodule will stay checked out up to Subproject commit so no new funky stuff like asan failures, unless someone bumps the commit.

We have a lot of trouble with submodule and that's a poor man's problem because RPM package is not available in Fedora. But we have RPM spec: https://github.com/facebookincubator/rpm-backports/tree/master/rpms/libbpf
Any brave souls to have it published in Fedora?

So I think it's OK to commit both BE and LE versions of the bytecode. And yes, it's OK if it changes over time. It would be like the hwdb stuff which we regenerate before each release too and commit. So I am on board with adding the libbpf dep as long as the llvm thing doesn't become a hard dep and people can use the pre-generated bytecode.

Could this wait until we have a BPF compiler in GCC (which is supposedly coming in GCC 10 next year)? That way an LLVM dependency isn't required at all.