Conversation
|
Thanks! I found it may be a technical obstacle for us to trace a frame's lifetime, it seems no way to link an xdp_buff to an skb: the "skb addr" is different, we can't know which skb an xdp_buff has turned to. |
|
There is a possible approach to link an I'll do a POC to confirm it later. |
|
It seems great to use Let me apply this diff. |
5ec35e9 to
7b97571
Compare
|
Thanks! I've just tried running, and got: Maybe we could ignore such progs instead of panic? |
Are you saying that both point to the same in the case XDP moves to the host (e.g., TC)? OT: could you drop the last commit, and just do |
Yes. As the output in previous reply. When to build xdp_buff from data: xdp_prepare_buff
Oh, sorry for my bad "merge". I will rebase it later. |
With these changes
|
It seems it does not remove the skb addr from Did it log I'd like to fix this code logic bug in #340. |
Noup, it didn't log. Shall I give it a go #340? |
Sure. |
|
Indeed, adding #340 to this branch resolved the last traces hiccup. Thanks! However, I still miss some traces when running this PR with #340. E.g., |
|
@Asphaltt could you rebase your PR? |
brb
left a comment
There was a problem hiding this comment.
Amazing! Could you rebase the PR? Once done, I will merge and do a new release.
In bpf verifier, it denies that FENTRY/FEXIT cannot trace BTF-not-annotated bpf progs. Or, an error "FENTRY/FEXIT program can only be attached to another program annotated with BTF" occurs. Signed-off-by: Leon Hwang <hffilwlqm@gmail.com>
It's to trace XDP progs in future commits. Signed-off-by: Leon Hwang <hffilwlqm@gmail.com>
Add an option --filter-trace-xdp to trace all XDP progs on host by fentry-ing on the progs, like the way of tracing tc-bpf 2347755. The diff from tracing tc-bpf: 1. Not support to filter mark. 2. No mark in meta output. 3. No proto in meta output. 4. Not support --output-skb. Signed-off-by: Leon Hwang <hffilwlqm@gmail.com>
As we know, the hard start of packet data is same between `struct sk_buff` and `struct xdp_buff`. So, we can track skb by it, and output it as skb address. Signed-off-by: Leon Hwang <hffilwlqm@gmail.com>
Address comments in PR cilium#340. Signed-off-by: Leon Hwang <hffilwlqm@gmail.com>
cilium#339 changed how pwru collects skb_addr, now it holds the value of skb->head instead of &skb. This patch renames value name `skb_addr` to `skb_head`, as well as map name `skb_addresses` to `skb_heads`. Signed-off-by: gray <gray.liang@isovalent.com>
After cilium#339, skb_addresses (now skb_heads) map uses skb->head as key rather than &skb. Kprobe prog on veth_convert_skb_to_xdp_buff should be adjusted to that change. Signed-off-by: gray <gray.liang@isovalent.com>
#339 changed how pwru collects skb_addr, now it holds the value of skb->head instead of &skb. This patch renames value name `skb_addr` to `skb_head`, as well as map name `skb_addresses` to `skb_heads`. Signed-off-by: gray <gray.liang@isovalent.com>
After #339, skb_addresses (now skb_heads) map uses skb->head as key rather than &skb. Kprobe prog on veth_convert_skb_to_xdp_buff should be adjusted to that change. Signed-off-by: gray <gray.liang@isovalent.com>
cilium#391 allows --track-skb to track new skb on veth, it relies on a map lookup to decide whether to track or not: ``` SEC("kprobe/veth_convert_skb_to_xdp_buff") int kprobe_veth_convert_skb_to_xdp_buff(struct pt_regs *ctx) { [...] u64 skb_head = (u64) BPF_CORE_READ(skb, head); if (bpf_map_lookup_elem(&skb_heads, &skb_head)) { [...] } return BPF_OK; } ``` However, when --track-skb-by-stackid is used along with --track-skb, the tracked skbs have risks not being recorded in skb_heads map. This is because: 1. cilium#384 no more updates skb_heads map when track reason is "by_stackid". 2. cilium#339 changes --track-skb from tracking &skb to tracking skb->head. So imagine an skb whose original skb->head = 0xa, whose value is updated to 0xb after a while. The first time pwru sees this skb, skb_heads map will insert 0xa entry. However, after skb->head is set to 0xb, pwru sees the skb being tracked due to "by_stackid", we won't insert 0xb entry into skb_heads map. Then when the skb is on veth, we can't find an entry by looking up 0xb from skb_heads map, ending up with losing track of veth skb again. This patch fixes the issue by raising the priority of track_by_filter: if an skb can be defined as tracked_by_filter and tracked_by_stackid, use tracked_by_filter over tracked_by_stackid. Another issue cilium#339 brings about is, an skb can have multiple skb->head stored in skb_heads map during its lifetime, but we only clean the latest value at kprobe_skb_lifetime_termination. This issue is beyond this patch. Signed-off-by: gray <gray.liang@isovalent.com>
cilium#391 allows --track-skb to track new skb on veth, it relies on a map lookup to decide whether to track or not: ``` SEC("kprobe/veth_convert_skb_to_xdp_buff") int kprobe_veth_convert_skb_to_xdp_buff(struct pt_regs *ctx) { [...] u64 skb_head = (u64) BPF_CORE_READ(skb, head); if (bpf_map_lookup_elem(&skb_heads, &skb_head)) { [...] } return BPF_OK; } ``` However, when --track-skb-by-stackid is used along with --track-skb, the tracked skbs have risks not being recorded in skb_heads map. This is because: 1. cilium#384 no more updates skb_heads map when track reason is "by_stackid". 2. cilium#339 changes --track-skb from using &skb to skb->head. So imagine an skb whose original skb->head = 0xa, the value is updated to 0xb after a while. The first time pwru sees this skb, skb_heads map will insert 0xa entry, this is correct. However, after skb->head being set to 0xb, pwru will verdict the skb of being tracked due to "by_stackid", we end up not inserting 0xb entry into skb_heads map. Then the skb reaches veth, pwru can't find an entry by looking up 0xb from skb_heads map, we are losing track of veth skb again. This patch fixes the issue by raising the priority of track_by_filter: if an skb can be defined as both tracked_by_filter and tracked_by_stackid, use tracked_by_filter over tracked_by_stackid. Another issue cilium#339 brings about is, an skb can have multiple skb->head stored in skb_heads map during its lifetime, but we only clean the latest value at kprobe_skb_lifetime_termination. This issue is beyond this patch. Signed-off-by: gray <gray.liang@isovalent.com>
cilium#339 changed how pwru tracks skb from using &skb to skb->head for xdp tracing. We found it causing problems such as: 1. An skb can have multiple different skb->head values during its lifetime, especically after (e.g. IPsec) encryption. Pwru now is likely to lose track of skb after encryption. 2. Some subtle issues like cilium#401. This is because some other tracking mechanism relies on &skb. This commit brings back tracking &skb instead of skb->head. As for XDP tracing, I'll introduce a new solution in the following patches. Signed-off-by: gray <gray.liang@isovalent.com>
#339 changed how pwru tracks skb from using &skb to skb->head for xdp tracing. We found it causing problems such as: 1. An skb can have multiple different skb->head values during its lifetime, especically after (e.g. IPsec) encryption. Pwru now is likely to lose track of skb after encryption. 2. Some subtle issues like #401. This is because some other tracking mechanism relies on &skb. This commit brings back tracking &skb instead of skb->head. As for XDP tracing, I'll introduce a new solution in the following patches. Signed-off-by: gray <gray.liang@isovalent.com>
cilium/pwru#339 changed how pwru collects skb_addr, now it holds the value of skb->head instead of &skb. This patch renames value name `skb_addr` to `skb_head`, as well as map name `skb_addresses` to `skb_heads`. Signed-off-by: gray <gray.liang@isovalent.com>
After cilium/pwru#339, skb_addresses (now skb_heads) map uses skb->head as key rather than &skb. Kprobe prog on veth_convert_skb_to_xdp_buff should be adjusted to that change. Signed-off-by: gray <gray.liang@isovalent.com>
cilium/pwru#339 changed how pwru tracks skb from using &skb to skb->head for xdp tracing. We found it causing problems such as: 1. An skb can have multiple different skb->head values during its lifetime, especically after (e.g. IPsec) encryption. Pwru now is likely to lose track of skb after encryption. 2. Some subtle issues like cilium/pwru#401. This is because some other tracking mechanism relies on &skb. This commit brings back tracking &skb instead of skb->head. As for XDP tracing, I'll introduce a new solution in the following patches. Signed-off-by: gray <gray.liang@isovalent.com>
cilium/pwru#339 changed how pwru collects skb_addr, now it holds the value of skb->head instead of &skb. This patch renames value name `skb_addr` to `skb_head`, as well as map name `skb_addresses` to `skb_heads`. Signed-off-by: gray <gray.liang@isovalent.com>
After cilium/pwru#339, skb_addresses (now skb_heads) map uses skb->head as key rather than &skb. Kprobe prog on veth_convert_skb_to_xdp_buff should be adjusted to that change. Signed-off-by: gray <gray.liang@isovalent.com>
cilium/pwru#339 changed how pwru tracks skb from using &skb to skb->head for xdp tracing. We found it causing problems such as: 1. An skb can have multiple different skb->head values during its lifetime, especically after (e.g. IPsec) encryption. Pwru now is likely to lose track of skb after encryption. 2. Some subtle issues like cilium/pwru#401. This is because some other tracking mechanism relies on &skb. This commit brings back tracking &skb instead of skb->head. As for XDP tracing, I'll introduce a new solution in the following patches. Signed-off-by: gray <gray.liang@isovalent.com>
cilium/pwru#339 changed how pwru collects skb_addr, now it holds the value of skb->head instead of &skb. This patch renames value name `skb_addr` to `skb_head`, as well as map name `skb_addresses` to `skb_heads`. Signed-off-by: gray <gray.liang@isovalent.com>
After cilium/pwru#339, skb_addresses (now skb_heads) map uses skb->head as key rather than &skb. Kprobe prog on veth_convert_skb_to_xdp_buff should be adjusted to that change. Signed-off-by: gray <gray.liang@isovalent.com>
cilium/pwru#339 changed how pwru tracks skb from using &skb to skb->head for xdp tracing. We found it causing problems such as: 1. An skb can have multiple different skb->head values during its lifetime, especically after (e.g. IPsec) encryption. Pwru now is likely to lose track of skb after encryption. 2. Some subtle issues like cilium/pwru#401. This is because some other tracking mechanism relies on &skb. This commit brings back tracking &skb instead of skb->head. As for XDP tracing, I'll introduce a new solution in the following patches. Signed-off-by: gray <gray.liang@isovalent.com>
Fix #239
Support tracing XDP
Add an option --filter-trace-xdp to trace all XDP progs on host by
fentry-ing on the progs, like the way tracing tc-bpf
Asphaltt@2347755.
The diff from tracing tc-bpf:
Example with
--filter-trace-xdp --filter-trace-tc: