ci: skip plain-text TCP RST packets in bpftrace#36962
Merged
Conversation
In #35485, we identified a leaked TCP RST packet generated from the kernel client in response to a FIN-ACK coming from the envoy proxy at the server-side. We noticed the behavior is due to a set of timers in the proxy that make it attempt closing (already closed) connections after specific intervals, causing the client to reply with a reset packet. The main issue is that such kernel-level packets don't contain the MARK we rely on to forward traffic from/to proxy, therefore letting the plain text packet through the default path. Let's tolerate such RST packets with this patch in all the cases we'd trace it as plain-text. That means: 1. $src_is_pod && $dst_is_pod 2. $pod_to_pod_via_proxy && ($src_is_pod || $dst_is_pod) Depending on the Cilium config, we observe leaked packets with source address either pod IP or ingress IP. The patch included both the two cases. Fixes: #35485 Signed-off-by: Simone Magnani <simone.magnani@isovalent.com>
Contributor
Author
|
/test |
jschwinger233
approved these changes
Jan 14, 2025
Member
There was a problem hiding this comment.
LGTM.
Since we have a scenario to reproduce: pod-to-pod-with-l7-policy-encryption for ipv6, maybe we can draft a separate test pr, add a commit to run ipv6 pod-to-pod-with-l7-policy-encryption on top of this, to verify the issue is really gone.
#35485 will be closed by merging this PR. I'll open a new issue (#37009) with our current understandings and proposed long term solution.
Contributor
Author
|
Plain TCP RST problem in CI is solved, we can merge this. |
aanm
approved these changes
Jan 17, 2025
Member
|
I think it would be good to bring this back into all releases to stop the CI bleed. From the looks of it we'll also need #36364 to make this apply cleanly. |
Contributor
Author
Yep, I've got this on my todos, adding |
Member
Bundled backport PRs are fine, if you're solving a class of issues or bring dependencies along (for instance #33575). |
Contributor
Author
That's an exact example of what I was looking for, many thanks. |
19 tasks
6 tasks
4 tasks
14 tasks
smagnani96
added a commit
that referenced
this pull request
Dec 9, 2025
This substantially reverts #41765, re-applying #36962 (modified code for readability). During our CLI tests, we observe some auto-generated TCP RST packets from the kernel in response to TCP-FIN packets sent by, most likely, some idle's timeout expiring. Envoy would generate TCP-FIN, but the kernel replies with a RST, given there's not a socket anymore listening to that port. We thought that after VinE we were able to catch such packets in our to-netdev program, but we kept observing them while adding unrelated tests to our CLI suite (see https://github.com/cilium/cilium/actions/runs/19833684454). Thus, we revert the latest change, and keep ignoring TCP RST packets. Signed-off-by: Simone Magnani <simone.magnani@isovalent.com>
aanm
pushed a commit
that referenced
this pull request
Dec 15, 2025
This substantially reverts #41765, re-applying #36962 (modified code for readability). During our CLI tests, we observe some auto-generated TCP RST packets from the kernel in response to TCP-FIN packets sent by, most likely, some idle's timeout expiring. Envoy would generate TCP-FIN, but the kernel replies with a RST, given there's not a socket anymore listening to that port. We thought that after VinE we were able to catch such packets in our to-netdev program, but we kept observing them while adding unrelated tests to our CLI suite (see https://github.com/cilium/cilium/actions/runs/19833684454). Thus, we revert the latest change, and keep ignoring TCP RST packets. Signed-off-by: Simone Magnani <simone.magnani@isovalent.com>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Dec 16, 2025
This substantially reverts #41765, re-applying #36962 (modified code for readability). During our CLI tests, we observe some auto-generated TCP RST packets from the kernel in response to TCP-FIN packets sent by, most likely, some idle's timeout expiring. Envoy would generate TCP-FIN, but the kernel replies with a RST, given there's not a socket anymore listening to that port. We thought that after VinE we were able to catch such packets in our to-netdev program, but we kept observing them while adding unrelated tests to our CLI suite (see https://github.com/cilium/cilium/actions/runs/19833684454). Thus, we revert the latest change, and keep ignoring TCP RST packets. Signed-off-by: Simone Magnani <simone.magnani@isovalent.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In #35485, we identified a leaked TCP RST packet generated from the kernel client in response to a FIN-ACK coming from the envoy proxy at the server-side. We noticed the behavior is due to a set of timers in the proxy that make it attempt closing (already closed) connections after specific intervals, causing the client to reply with a reset packet. The main issue is that such kernel-level packets don't contain the MARK we rely on to forward traffic from/to proxy, therefore letting the plain text packet through the default path.
Let's tolerate such RST packets with this patch in all the cases we'd trace it as plain-text. That means:
Depending on the Cilium config, we observe leaked packets with source address either pod IP or ingress IP. The patch included both the two cases.
Fixes: #35485