doc: add initial BPF and XDP documentation by borkmann · Pull Request #546 · cilium/cilium

borkmann · 2017-04-14T01:13:34Z

We're currently lacking a guide on this, therefore add a start of it
that we can further extend over time.

Rendered version from wip branch:

http://cilium.readthedocs.io/en/doc-bpf-wip/

Signed-off-by: Daniel Borkmann daniel@cilium.io

aalemayhu

Other than the minor nitpick this looks very good and was fun to read 🌮

aalemayhu · 2017-04-14T11:25:05Z

+generic and flexible enough that there are many kernel subsystems which use eBPF
+apart from only networking. Nowadays, the Linux kernel runs eBPF only and loaded
+cBPF bytecode is transparently translated into an eBPF representation in the
+kernel before program execution.


s/into an/into a/ ?

I think it's probably fine as is, but I'll double check to make sure.

(an eBPF representation is the correct form)

aalemayhu · 2017-04-14T11:28:44Z

+  these packet load instructions are less relevant nowadays. ``BPF_LDX`` class
+  holds instructions for byte / half-word / word / double-word loads out of
+  memory. Memory in this context is generic can could be stack memory, map value
+  data, packet data, etc.


s/can could/and could/

aalemayhu · 2017-04-14T11:36:36Z

+   included into the main programs. For example, Cilium makes heavy use of
+   this (see ``bpf/lib/``). However, this still allows for including header
+   files, for example, from the kernel or other libraries and resuse their
+   static inline functions or macros / definitions.


s/resuse/reuse/

aalemayhu · 2017-04-14T11:37:39Z

+
+   The remaining section names are specific for eBPF program code, for example,
+   the below code has been modified to contain to two program sections, ``ingress``
+   and ``egress``. The toy example code demonstrates that both can share a map


s/to contain to/to contain/ ?

aalemayhu · 2017-04-14T11:39:38Z

+  value size. This works, because during execution, eBPF programs are guaranteed
+  to never get preempted by the kernel and therefore can use the single map entry
+  as a scratch buffer for temporariy data, for example, to extend beyond the stack
+  limitation. This also works accross tail calls, since it has the same guarantees


s/temporariy/temporary/ ?

aalemayhu · 2017-04-14T11:47:30Z

+  program attached. ``ip link | grep xdp`` can thus be used to find all interfaces
+  that have XDP running. Further introspection facilities will be provided through
+  the detailed view through ``ip -d link`` once the kernel API gains support for
+  dumping additional attributes.


s/view through/view/ ? The extra through adds nothing extra IMO, but either way is fine.

Wrote it as view with.

aalemayhu · 2017-04-14T11:48:37Z

+where all subsystem users will point to the same location through symbolic
+links for the ``globals`` namespace, such that pinned eBPF maps can be reused
+among various eBPF program types in iproute2. In case the file system instace
+was mounted already and an existing structure exists already, then tc will


s/instace/instance/ ?

tgraf · 2017-04-14T08:32:17Z

+.. _bpf_guide:
+
+*****************
+BPF and XDP Guide


Maybe rename this to "BPF and XDP Reference Guide" to make it a bit clearer that this is deep dive material for developers and not required for users.

tgraf · 2017-04-14T08:33:38Z

+types such as tc and XDP, and to aide developing Cilium's eBPF templates.
+
+**This purely serves as a developer's guide and is explicitly not a requirement
+for Cilium users to read, since Cilium abstracts eBPF internals away entirely.**


Let's move this and make it the first paragraph. You can use:

.. note:: This purely serves as a ...

prefix so it will appear as a note widget so it stands out a bit more.

tgraf · 2017-04-14T08:41:19Z

+**This purely serves as a developer's guide and is explicitly not a requirement
+for Cilium users to read, since Cilium abstracts eBPF internals away entirely.**
+
+The older cBPF architecture is not covered by this document, since Cilium does


I would drop the 'since Cilium' part

tgraf · 2017-04-14T08:42:53Z

+
+eBPF does not define itself by only providing its instruction set, but also by
+offering further infrastructure around it such as maps that act as efficient
+key / value stores, helper functions to interact with the kernel, tail calls for


helper functions to interact with and leverage kernel functionaly

tgraf · 2017-04-14T09:10:54Z

+
+* In case of networking (e.g., tc and XDP), eBPF programs can be updated atomically
+  without having to restart the kernel, system services or containers, and without
+  traffic interruptions.


Let's add that any program state can be maintained throughout updates via bpf maps.

tgraf · 2017-04-14T09:23:31Z

+  calls with regard to user space applications.
+
+* eBPF programs work in concert with the kernel, they make use of existing kernel
+  infrastructure and tooling (e.g., iproute2) as well as the safety guarantees that


Would be great to add an (e.g. ...) with a couple of examples of used kernel infrastructure.

tgraf · 2017-04-14T09:24:42Z

+  infrastructure and tooling (e.g., iproute2) as well as the safety guarantees that
+  the kernel provides. Unlike kernel modules, eBPF programs are verified through an
+  in-kernel verifier in order to ensure that they cannot crash the kernel, always
+  terminate, etc. XDP programs, for example, reuse the existing in-kernel drivers


Maybe move XDP down until you have properly introduced it. The XDP concept is completely unknown to the reader at this point.

Given it's kind of used throughout the doc (also in the more generic LLVM / iproute2 section), I tried to introduce XDP and tc terms at the very beginning now, but we could later on also still rearrange this a bit if we think a strict separation seems better.

tgraf · 2017-04-14T09:25:10Z

+
+The execution of an eBPF program inside the kernel is always event driven! For example,
+a networking device that has an eBPF program attached on its ingress path will trigger
+the execution of the program once a packet is received.


Add a second example for kprobe attachment.

tgraf · 2017-04-14T09:26:59Z

+handing execution back to the kernel, the exit value is passed as a 32 bit value.
+
+Registers ``r1`` - ``r5`` are scratch registers, meaning the eBPF program needs to
+either spill them to the eBPF stack or move them to callee saved registers if these


Add a sentence to explains spilling.

We're currently lacking a guide on this, therefore add a start of it that we can further extend over time. Rendered version from wip branch: http://cilium.readthedocs.io/en/doc-bpf-wip/ Signed-off-by: Daniel Borkmann <daniel@cilium.io>

tgraf · 2017-04-15T07:33:35Z

I'll provide more feedback directly as a PR but this is in great shape to go in already.

borkmann · 2017-04-15T08:57:59Z

Perfect, thanks. I'll continue working on the remaining bits.

As the previous commit mentions, this does not occur on main, v1.16, and v1.15 due to cilium#29036. However, in v1.14 and v1.13, we need to take special care because the NameManager and SelectorCache lock can be taken while the Endpoint lock is held during Endpoint deletion. Here are the relevant stacktraces regarding the deadlock: ``` 1: sync.Mutex.Lock [75 minutes] [Created by http.(*Server).Serve in goroutine 1699 @ server.go:3086] sync sema.go:77 runtime_SemacquireMutex(*uint32(0x5), false, 43690) sync mutex.go:171 (*Mutex).lockSlow(*Mutex(cilium#1733)) sync mutex.go:90 (*Mutex).Lock(...) sync rwmutex.go:147 (*RWMutex).Lock(*RWMutex(0xb0)) fqdn name_manager.go:70 (*NameManager).Lock(0xffffffffffffffff) policy selectorcache.go:964 (*SelectorCache).RemoveSelectors(cilium#1088, {cilium#28569, 0xb, 1}, {cilium#643, cilium#32582}) policy l4.go:810 (*L4Filter).removeSelectors(cilium#32582, cilium#29992) policy l4.go:817 (*L4Filter).detach(cilium#719, cilium#29993) policy l4.go:988 L4PolicyMap.Detach(...) policy l4.go:1179 (*L4Policy).Detach(cilium#20318, cilium#1383) policy resolve.go:103 (*selectorPolicy).Detach(...) policy distillery.go:81 (*PolicyCache).delete(cilium#1354, cilium#19354) policy distillery.go:138 (*PolicyCache).LocalEndpointIdentityRemoved(cilium#523, cilium#1163) identitymanager manager.go:167 (*IdentityManager).remove(cilium#706, cilium#19354) identitymanager manager.go:147 (*IdentityManager).Remove(cilium#706, cilium#19354) identitymanager manager.go:52 Remove(...) endpoint endpoint.go:1146 (*Endpoint).leaveLocked(cilium#1883, cilium#12221, {0x30, 0}) endpoint endpoint.go:2192 (*Endpoint).Delete(cilium#1883, {0x80, 0xaa}) endpointmanager manager.go:380 (*EndpointManager).removeEndpoint(cilium#1161, 0, {0xff, 0xff}) endpointmanager manager.go:394 (*EndpointManager).RemoveEndpoint(...) cmd endpoint.go:684 (*Daemon).deleteEndpointQuiet(...) cmd endpoint.go:666 (*Daemon).deleteEndpoint(cilium#1155, cilium#1883) cmd endpoint.go:713 (*Daemon).DeleteEndpoint(cilium#1155, {cilium#27399, cilium#8108}) cmd endpoint.go:770 (*deleteEndpointID).Handle(cilium#21193, {cilium#2452, {cilium#27399, 0x4d}}) endpoint delete_endpoint_id.go:66 (*DeleteEndpointID).ServeHTTP(cilium#5934, {cilium#666, cilium#2242}, cilium#2452) middleware operation.go:28 (*Context).RoutesHandler.NewOperationExecutor.func1({cilium#666, cilium#2242}, cilium#2452) http server.go:2136 HandlerFunc.ServeHTTP(ReadCloser(cilium#130), func{cilium#2242, 0x3}) middleware router.go:78 NewRouter.func1({cilium#666, cilium#2242}, cilium#2451) http server.go:2136 HandlerFunc.ServeHTTP(ReadCloser(cilium#718), func{cilium#2242, #59}) middleware redoc.go:72 Redoc.func1({cilium#666, cilium#2242}, cilium#1251) http server.go:2136 HandlerFunc.ServeHTTP(ReadCloser(cilium#4920), func{cilium#2242, #45}) middleware spec.go:46 Spec.func1({cilium#666, cilium#2242}, cilium#4921) http server.go:2136 HandlerFunc.ServeHTTP(ReadCloser(cilium#10532), func{cilium#2242, cilium#23015}) metrics middleware.go:64 (*APIEventTSHelper).ServeHTTP(cilium#1459, {cilium#668, cilium#10533}, cilium#2451) api apipanic.go:42 (*APIPanicHandler).ServeHTTP(cilium#722, {cilium#668, cilium#10533}, cilium#4922) http server.go:2938 serverHandler.ServeHTTP(*Server(cilium#8105), cilium#668, cilium#10533, 0x6) http server.go:2009 (*conn).serve(*conn(cilium#16005), Context{cilium#673, cilium#1554}) 8: sync.Mutex.Lock [74 minutes] [Created by http.(*Server).Serve in goroutine 1699 @ server.go:3086] sync sema.go:77 runtime_SemacquireMutex(*, 0x47, cilium#1154) sync mutex.go:171 (*Mutex).lockSlow(cilium#706) sync mutex.go:90 (*Mutex).Lock(...) sync rwmutex.go:147 (*RWMutex).Lock(*) identitymanager manager.go:99 (*IdentityManager).RemoveOldAddNew(cilium#706, 0, cilium#1154) identitymanager manager.go:123 RemoveOldAddNew(...) endpoint policy.go:852 (*Endpoint).SetIdentity(*, cilium#1154, 0) endpoint endpoint.go:1932 (*Endpoint).identityLabelsChanged(*, {cilium#674, *}, 1) endpoint endpoint.go:1780 (*Endpoint).runIdentityResolver(*, {cilium#674, *}, 1, 1) endpoint endpoint.go:1720 (*Endpoint).UpdateLabels(*, {cilium#674, *}, *, *, 8) cmd endpoint.go:477 (*Daemon).createEndpoint(cilium#1155, {cilium#673, *}, {cilium#683, cilium#1155}, *) cmd endpoint.go:542 (*putEndpointID).Handle(cilium#21192, {*, *, {*, 0xe}}) endpoint put_endpoint_id.go:58 (*PutEndpointID).ServeHTTP(cilium#3961, {cilium#666, *}, *) middleware operation.go:28 (*Context).RoutesHandler.NewOperationExecutor.func1({cilium#666, *}, *) http server.go:2136 HandlerFunc.ServeHTTP(*, {cilium#666, *}, *) middleware router.go:78 NewRouter.func1({cilium#666, *}, *) http server.go:2136 HandlerFunc.ServeHTTP(*, {cilium#666, *}, *) middleware redoc.go:72 Redoc.func1({cilium#666, *}, cilium#1251) http server.go:2136 HandlerFunc.ServeHTTP(*, {cilium#666, *}, #45) middleware spec.go:46 Spec.func1({cilium#666, *}, *) http server.go:2136 HandlerFunc.ServeHTTP(*, {cilium#666, *}, *) metrics middleware.go:64 (*APIEventTSHelper).ServeHTTP(cilium#1459, {cilium#668, *}, *) api apipanic.go:42 (*APIPanicHandler).ServeHTTP(#49, {cilium#668, *}, *) http server.go:2938 serverHandler.ServeHTTP({cilium#653}, {cilium#668, *}, 6) http server.go:2009 (*conn).serve(*, {cilium#673, cilium#1554}) 5: sync.Mutex.Lock [75 minutes] [Created by eventqueue.(*EventQueue).Run in goroutine 1482 @ eventqueue.go:229] sync sema.go:77 runtime_SemacquireMutex(cilium#142, 0xe8, *) sync mutex.go:171 (*Mutex).lockSlow(cilium#1733) sync mutex.go:90 (*Mutex).Lock(...) sync rwmutex.go:147 (*RWMutex).Lock(0x68) fqdn name_manager.go:70 (*NameManager).Lock(*) policy selectorcache.go:798 (*SelectorCache).AddFQDNSelector(cilium#1088, {cilium#643, *}, {{*, 0x4d}, {0, 0}}) policy l4.go:628 (*L4Filter).cacheFQDNSelector(...) policy l4.go:623 (*L4Filter).cacheFQDNSelectors(*, {*, 4, cilium#193}, cilium#536) policy l4.go:725 createL4Filter({cilium#680, *}, {*, 1, 1}, 0, {cilium#660, *}, {{*, 4}, ...}, ...) policy l4.go:879 createL4EgressFilter(...) policy rule.go:717 mergeEgressPortProto({cilium#680, *}, #44, {*, 0xa, 0}, *, {cilium#660, *}, {{*, ...}, ...}, ...) policy rule.go:672 mergeEgress.func1({cilium#660, *}) api l4.go:284 PortRules.Iterate({*, 1, cilium#546}, *) policy rule.go:624 mergeEgress({cilium#680, *}, *, {*, 1, 1}, 0, {cilium#661, *}, {cilium#662, ...}, ...) policy rule.go:753 (*rule).resolveEgressPolicy(*, {cilium#680, *}, *, *, *, {0, 0, 0}, {0, ...}) policy rules.go:103 ruleSlice.resolveL4EgressPolicy({*, *, *}, {cilium#680, *}, *) policy repository.go:718 (*Repository).resolvePolicyLocked(cilium#1089, *) policy distillery.go:119 (*PolicyCache).updateSelectorPolicy(cilium#1354, *) policy distillery.go:153 (*PolicyCache).UpdatePolicy(...) endpoint policy.go:262 (*Endpoint).regeneratePolicy(*) endpoint bpf.go:744 (*Endpoint).runPreCompilationSteps(*, *, *) endpoint bpf.go:589 (*Endpoint).regenerateBPF(*, *) endpoint policy.go:457 (*Endpoint).regenerate(*, *) endpoint events.go:53 (*EndpointRegenerationEvent).Handle(*, *) eventqueue eventqueue.go:245 (*EventQueue).run.func1() sync once.go:74 (*Once).doSlow(*, *) sync once.go:65 (*Once).Do(...) eventqueue eventqueue.go:233 (*EventQueue).run(*) 1: select [75 minutes] [Created by eventqueue.(*EventQueue).Run in goroutine 1482 @ eventqueue.go:229] semaphore semaphore.go:60 (*Weighted).Acquire(cilium#1092, {cilium#671, cilium#722}, cilium#766) lock semaphored_mutex.go:30 (*SemaphoredMutex).Lock(...) ipcache ipcache.go:140 (*IPCache).Lock(...) ipcache cidr.go:56 (*IPCache).AllocateCIDRs(cilium#1316, {0, 0, cilium#2037}, {0, 0, 0}, 0) ipcache cidr.go:103 (*IPCache).AllocateCIDRsForIPs(0, {0, cilium#697, 0}, 0xffffffffffffffff) cmd identity.go:114 cachingIdentityAllocator.AllocateCIDRsForIPs(...) policy selectorcache.go:509 (*SelectorCache).allocateIdentityMappings(cilium#1088, {{0, 0}, {cilium#5036, 0x2b}}, #45) policy selectorcache.go:843 (*SelectorCache).AddFQDNSelector(cilium#1088, {cilium#643, cilium#27811}, {{0, 0}, {cilium#5036, 0x2b}}) policy l4.go:628 (*L4Filter).cacheFQDNSelector(...) policy l4.go:623 (*L4Filter).cacheFQDNSelectors(cilium#27811, {cilium#42936, 0x287, cilium#193}, cilium#536) policy l4.go:725 createL4Filter({cilium#680, cilium#22826}, {cilium#22828, 1, 1}, 0, {cilium#660, cilium#24510}, {{cilium#4487, 3}, ...}, ...) policy l4.go:879 createL4EgressFilter(...) policy rule.go:717 mergeEgressPortProto({cilium#680, cilium#22826}, #44, {cilium#22828, 0xa, #78536}, #79633, {cilium#660, cilium#24510}, {{cilium#4487, ...}, ...}, ...) policy rule.go:672 mergeEgress.func1({cilium#660, cilium#24510}) api l4.go:284 PortRules.Iterate({cilium#24510, 1, cilium#546}, cilium#11741) policy rule.go:624 mergeEgress({cilium#680, cilium#22826}, cilium#18687, {cilium#22828, 1, 1}, 0, {cilium#661, cilium#5624}, {cilium#662, ...}, ...) policy rule.go:753 (*rule).resolveEgressPolicy(cilium#24575, {cilium#680, cilium#22826}, cilium#18687, cilium#29345, cilium#4782, {0, 0, 0}, {0, ...}) policy rules.go:103 ruleSlice.resolveL4EgressPolicy({cilium#10690, 0xb, 5}, {cilium#680, cilium#22826}, cilium#18687) policy repository.go:718 (*Repository).resolvePolicyLocked(cilium#1089, cilium#18461) policy distillery.go:119 (*PolicyCache).updateSelectorPolicy(cilium#1354, cilium#18461) policy distillery.go:153 (*PolicyCache).UpdatePolicy(...) endpoint policy.go:262 (*Endpoint).regeneratePolicy(cilium#1748) endpoint bpf.go:744 (*Endpoint).runPreCompilationSteps(cilium#1748, cilium#27542, cilium#4781) endpoint bpf.go:589 (*Endpoint).regenerateBPF(cilium#1748, cilium#27542) endpoint policy.go:457 (*Endpoint).regenerate(cilium#1748, cilium#27542) endpoint events.go:53 (*EndpointRegenerationEvent).Handle(cilium#18609, cilium#703) eventqueue eventqueue.go:245 (*EventQueue).run.func1() sync once.go:74 (*Once).doSlow(*Once(cilium#45414), func(cilium#69)) sync once.go:65 (*Once).Do(...) eventqueue eventqueue.go:233 (*EventQueue).run(cilium#6023) ``` Generated from pp tool: https://github.com/maruel/panicparse Signed-off-by: Chris Tarazi <chris@isovalent.com>

borkmann requested review from aalemayhu, aanm, ianvernon, mchalla and tgraf April 14, 2017 01:13

aalemayhu approved these changes Apr 14, 2017

View reviewed changes

tgraf suggested changes Apr 14, 2017

View reviewed changes

borkmann force-pushed the doc-bpf-wip branch 4 times, most recently from bfe9836 to 44cd590 Compare April 14, 2017 23:35

doc: add initial BPF and XDP documentation

c5cbd77

We're currently lacking a guide on this, therefore add a start of it that we can further extend over time. Rendered version from wip branch: http://cilium.readthedocs.io/en/doc-bpf-wip/ Signed-off-by: Daniel Borkmann <daniel@cilium.io>

borkmann force-pushed the doc-bpf-wip branch from 44cd590 to c5cbd77 Compare April 15, 2017 00:00

tgraf approved these changes Apr 15, 2017

View reviewed changes

borkmann merged commit e70ff85 into master Apr 15, 2017

borkmann deleted the doc-bpf-wip branch April 15, 2017 08:58

yoursanonymous pushed a commit to yoursanonymous/cilium that referenced this pull request Jan 31, 2026

chore: add new blogs (cilium#546)

1990390

Conversation

borkmann commented Apr 14, 2017

Uh oh!

aalemayhu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tgraf commented Apr 15, 2017

Uh oh!

borkmann commented Apr 15, 2017

Uh oh!

Reviewers

Assignees

Labels

aalemayhu left a comment •

edited

Loading