Skip to content

testing: final conversion 4.19 CI to kube-proxy free#12045

Merged
borkmann merged 14 commits intomasterfrom
pr/fix-4.19-verifier2
Jun 16, 2020
Merged

testing: final conversion 4.19 CI to kube-proxy free#12045
borkmann merged 14 commits intomasterfrom
pr/fix-4.19-verifier2

Conversation

@borkmann
Copy link
Copy Markdown
Member

@borkmann borkmann commented Jun 12, 2020 •
edited
Loading

Re-enabling probe mode on 4.19.

Fixes: #11175

@maintainer-s-little-helper
Copy link
Copy Markdown

maintainer-s-little-helper Bot commented Jun 12, 2020

Please set the appropriate release note label.

@borkmann
Copy link
Copy Markdown
Member Author

borkmann commented Jun 12, 2020

retest-4.19

@borkmann borkmann mentioned this pull request Jun 12, 2020
Merged
@borkmann borkmann changed the title [no not merge] testing only testing: final conversion 4.19 CI to kube-proxy free Jun 12, 2020
@borkmann borkmann force-pushed the pr/fix-4.19-verifier2 branch from 65c3fe4 to 994fe15 Compare June 12, 2020 14:05
@borkmann borkmann added the release-note/misc This PR makes changes that have no direct user impact. label Jun 12, 2020
@borkmann borkmann added area/CI Continuous Integration testing issue or flake and removed pending-review labels Jun 12, 2020
@coveralls
Copy link
Copy Markdown

coveralls commented Jun 12, 2020 •
edited
Loading

Coverage Status

Coverage decreased (-0.04%) to 36.999% when pulling beb8ee0 on pr/fix-4.19-verifier2 into e4bd54c on master.

@borkmann borkmann force-pushed the pr/fix-4.19-verifier2 branch 4 times, most recently from 0bd37b3 to 248b222 Compare June 12, 2020 22:17
@borkmann
Copy link
Copy Markdown
Member Author

borkmann commented Jun 12, 2020

test-focus K8sUpdates

@borkmann borkmann requested review from joestringer and pchaigno June 12, 2020 22:24
@borkmann
Copy link
Copy Markdown
Member Author

borkmann commented Jun 12, 2020

test-me-please

@borkmann borkmann added needs-backport/1.8 area/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. labels Jun 12, 2020
@borkmann
Copy link
Copy Markdown
Member Author

borkmann commented Jun 13, 2020

test-focus K8sUpdates

@borkmann
Copy link
Copy Markdown
Member Author

borkmann commented Jun 13, 2020

test-me-please

pchaigno
pchaigno reviewed Jun 13, 2020
View reviewed changes
Comment thread bpf/bpf_host.c
@pchaigno pchaigno mentioned this pull request Jun 15, 2020
Closed
@borkmann borkmann force-pushed the pr/fix-4.19-verifier2 branch from fa455e4 to ba58905 Compare June 15, 2020 09:11
borkmann and others added 12 commits June 16, 2020 13:11
@borkmann
cilium: bail out in native XDP mode when using multi-device setup
9e06255 
In XDP, we can only xmit via XDP_TX, so a multi-device setup would
otherwise be broken. Tell the user to specify the device manually
in case there were multiple devices detected.

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
@borkmann
cilium: don't warn if nodeport dev for default route cannot be found
fa7cb68 
It is otherwise just confusing to users to figure out whether they need
to react to this or not. Only really warn if we cannot auto-derive it
from the k8s Node IP since this case would be unexpected.

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
@borkmann
docs: add notes on XDP on multi-device nodes
fd4570b 
Add few notes on how to deal with nodes that have multiple devices which
are auto-detected. Also make it clear as a limitation.

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
@borkmann
docs: add cilium status info to hostport section
e72c119 
Add a small paragraph for helping users to verify whether enabled or not.

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
@borkmann
docs: link to map sizing guide from XDP section
4c6c599 
Make users aware that they might need to tweak their map sizes and
link to the guide which has more details on it.

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
@borkmann
bpf: ensure we only xlate external ip when netns cookie unavailable
cc0d7df 
Commit b2c9b07 ("bpf, sock: reduce xlation for externalTrafficPolicy=Local
to host_id") skips service translation for remote externalTrafficPolicy=Local
services (aka not being HOST_ID ipcache scope).

However, later on, we relaxed bpf_lxc to compile legacy service xlation back in
for services with externalIPs via 6426c86 ("bpf, external ip: fix service
xlation for containers"), which would then go and try to xlate such services
locally.

Issue is that here we don't differentiate between the different service types
so we would go and attempt to xlate. Given we don't populate service maps with
remote backends that traffic gets blackholed on bpf_lxc.

Instead, only xlate externalIP services at that layer and skip the rest, so we
let externalTrafficPolicy=Local dsts pass through and be xlated at the remote
LB. We /already/ have this kind of behavior when we have netns cookies since
the bpf_lxc xlation is compiled out entirely there, but not for older kernels,
thus, fix it to make both consistent.

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
@pchaigno @borkmann
bpf: fix program size issue for complex configs on 4.19
9b1ca79 
On Linux 4.19 (and any kernel with 4096 BPF instructions limit), when
running with kube-proxy-replacement=strict, IPv4, IPv6, fragment support
(and in debug mode), Cilium fails to load bpf_overlay and bpf_host with:

  Prog section '2/19' rejected: Argument list too long (7)
   - Type:         3
   - Attach Type:  0
   - Instructions: 4203 (107 over limit)
   - License:      GPL
  Error filling program arrays!

2/19 is section CILIUM_CALL_NODEPORT_REVNAT. We can reduce the number of
instructions in that section by breaking the IPv4 and IPv6 cases with
tail calls. Instead of introducing a new tail call on each path, we can
turn CILIUM_CALL_NODEPORT_REVNAT into CILIUM_CALL_IPV{4,6}_NODEPORT_REVNAT.

Signed-off-by: Paul Chaignon <paul@cilium.io>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
@borkmann
bpf: bump snat retries for older kernels
3430da7 
Given v4/v6 has been split off into different tail calls, it enables
us to significantly bump the collision retries on older kernels.

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
@borkmann
test: specify single device for XDP nodeport tests
5be1b3f 
... as otherwise it won't work on multi-dev node. Revert back to auto-detection
after XDP tests have run.

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
@borkmann
test: re-enable kube-proxy replacement probe mode in 4.19
85553a7 
Re-enable it such that we can run CI with kube-proxy-free BPF code alongside
kube-proxy.

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
@borkmann
test: use new image to clean up prior cilium state in upgrade test
04c7dc2 
Prior tests have been run with the latest Cilium image. Using the old one
for the initial cleanup is broken since it will not clean-up everything.

For example the 1.7 cleanup method will try to remove tc attached filters
with the name bpf_netdev.o, but for 1.8/latest they are names bpf_netdev_<dev>.o
which will then break connectivity after flushing all files from BPF fs as
tail call maps are purged which gets us into drop-all due to missed tail calls.

Run new (and old) to make sure there are no quirks from old version left behind
that the new one didn't catch.

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
@pchaigno @borkmann
test: disable running with kube-proxy in 4.19 CI build
beb8ee0 
... and instead rely on kube-proxy's strict replacement.

Also disable tests that are not working on 4.19 due to missing kernel
features on NodePort side.

Signed-off-by: Paul Chaignon <paul@cilium.io>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
@borkmann borkmann force-pushed the pr/fix-4.19-verifier2 branch from 78e6f32 to beb8ee0 Compare June 16, 2020 11:14
@borkmann
Copy link
Copy Markdown
Member Author

borkmann commented Jun 16, 2020

test-me-please

@borkmann
Copy link
Copy Markdown
Member Author

borkmann commented Jun 16, 2020

retest-4.9

@borkmann borkmann mentioned this pull request Jun 16, 2020
Merged
@borkmann
Copy link
Copy Markdown
Member Author

borkmann commented Jun 16, 2020

4.19 has a single test that is flaking (K8sDatapathConfig Etcd Check connectivity). Was green on prior test run.

@borkmann borkmann merged commit 0fa8ac7 into master Jun 16, 2020
@borkmann borkmann deleted the pr/fix-4.19-verifier2 branch June 16, 2020 13:17
@borkmann borkmann mentioned this pull request Jun 16, 2020
Merged
qmonnet added a commit that referenced this pull request Jun 17, 2020
@qmonnet
test/K8sServices: re-enable IPv4 fragment tests on kernel 4.19
8f04915 
The test for fragment tracking support got a fix with commit
0e772e7 ("test: Fix fragment tracking test under KUBEPROXY=1"),
where the pattern for searching entries in the Conntrack table accounts
for DNAT not happening if kube-proxy is present.

Following recent changes in the datapath and tests for the Linux 4.19
kernel, DNAT is now used even with kube-proxy, provided bpf_sock is in
use. This led to CI failures, and the test was disabled for 4.19 kernels
with commit 1120aed ("test/K8sServices: disable fragment tracking
test for kernel 4.19").

Now that complexity issues are fixed (see #11977 and #12045), let's
enable the test on 4.19 again. Ignore DNAT only if kube-proxy is present
and bpf_sock (host-reachable services) is not in use. This is also the
case for net-next kernels (this didn't fail in CI before because we do
not test with kube-proxy on net-next).

Note that (as far as I know) both 4.19 and net-next always use bpf_sock
in CI runs, so the check on hostReachableServices is currently
superfluous. Let's have it all the same, in case something changes in
the future, to avoid unexpected breakage.

Signed-off-by: Quentin Monnet <quentin@isovalent.com>
@qmonnet qmonnet mentioned this pull request Jun 17, 2020
Merged
qmonnet added a commit that referenced this pull request Jun 18, 2020
@qmonnet
test/K8sServices: re-enable IPv4 fragment tests on kernel 4.19
da70faf 
The test for fragment tracking support got a fix with commit
0e772e7 ("test: Fix fragment tracking test under KUBEPROXY=1"),
where the pattern for searching entries in the Conntrack table accounts
for DNAT not happening if kube-proxy is present.

Following recent changes in the datapath and tests for the Linux 4.19
kernel, DNAT is now used even with kube-proxy, provided bpf_sock is in
use. This led to CI failures, and the test was disabled for 4.19 kernels
with commit 1120aed ("test/K8sServices: disable fragment tracking
test for kernel 4.19").

Now that complexity issues are fixed (see #11977 and #12045), let's
enable the test on 4.19 again. Ignore DNAT only if kube-proxy is present
and bpf_sock (host-reachable services) is not in use. This is also the
case for net-next kernels (this didn't fail in CI before because we do
not test with kube-proxy on net-next).

Note that (as far as I know) both 4.19 and net-next always use bpf_sock
in CI runs, so the check on hostReachableServices is currently
superfluous. Let's have it all the same, in case something changes in
the future, to avoid unexpected breakage.

Signed-off-by: Quentin Monnet <quentin@isovalent.com>
aanm pushed a commit that referenced this pull request Jun 21, 2020
@qmonnet @aanm
test/K8sServices: re-enable IPv4 fragment tests on kernel 4.19
5b9503f 
The test for fragment tracking support got a fix with commit
0e772e7 ("test: Fix fragment tracking test under KUBEPROXY=1"),
where the pattern for searching entries in the Conntrack table accounts
for DNAT not happening if kube-proxy is present.

Following recent changes in the datapath and tests for the Linux 4.19
kernel, DNAT is now used even with kube-proxy, provided bpf_sock is in
use. This led to CI failures, and the test was disabled for 4.19 kernels
with commit 1120aed ("test/K8sServices: disable fragment tracking
test for kernel 4.19").

Now that complexity issues are fixed (see #11977 and #12045), let's
enable the test on 4.19 again. Ignore DNAT only if kube-proxy is present
and bpf_sock (host-reachable services) is not in use. This is also the
case for net-next kernels (this didn't fail in CI before because we do
not test with kube-proxy on net-next).

Note that (as far as I know) both 4.19 and net-next always use bpf_sock
in CI runs, so the check on hostReachableServices is currently
superfluous. Let's have it all the same, in case something changes in
the future, to avoid unexpected breakage.

Signed-off-by: Quentin Monnet <quentin@isovalent.com>
aanm pushed a commit that referenced this pull request Jun 21, 2020
@qmonnet @aanm
test/K8sServices: re-enable IPv4 fragment tests on kernel 4.19
5b68b01 
[ upstream commit 5b9503f ]

The test for fragment tracking support got a fix with commit
0e772e7 ("test: Fix fragment tracking test under KUBEPROXY=1"),
where the pattern for searching entries in the Conntrack table accounts
for DNAT not happening if kube-proxy is present.

Following recent changes in the datapath and tests for the Linux 4.19
kernel, DNAT is now used even with kube-proxy, provided bpf_sock is in
use. This led to CI failures, and the test was disabled for 4.19 kernels
with commit 1120aed ("test/K8sServices: disable fragment tracking
test for kernel 4.19").

Now that complexity issues are fixed (see #11977 and #12045), let's
enable the test on 4.19 again. Ignore DNAT only if kube-proxy is present
and bpf_sock (host-reachable services) is not in use. This is also the
case for net-next kernels (this didn't fail in CI before because we do
not test with kube-proxy on net-next).

Note that (as far as I know) both 4.19 and net-next always use bpf_sock
in CI runs, so the check on hostReachableServices is currently
superfluous. Let's have it all the same, in case something changes in
the future, to avoid unexpected breakage.

Signed-off-by: Quentin Monnet <quentin@isovalent.com>
Signed-off-by: André Martins <andre@cilium.io>
aanm pushed a commit that referenced this pull request Jun 22, 2020
@qmonnet @aanm
test/K8sServices: re-enable IPv4 fragment tests on kernel 4.19
4ff8ea5 
[ upstream commit 5b9503f ]

The test for fragment tracking support got a fix with commit
0e772e7 ("test: Fix fragment tracking test under KUBEPROXY=1"),
where the pattern for searching entries in the Conntrack table accounts
for DNAT not happening if kube-proxy is present.

Following recent changes in the datapath and tests for the Linux 4.19
kernel, DNAT is now used even with kube-proxy, provided bpf_sock is in
use. This led to CI failures, and the test was disabled for 4.19 kernels
with commit 1120aed ("test/K8sServices: disable fragment tracking
test for kernel 4.19").

Now that complexity issues are fixed (see #11977 and #12045), let's
enable the test on 4.19 again. Ignore DNAT only if kube-proxy is present
and bpf_sock (host-reachable services) is not in use. This is also the
case for net-next kernels (this didn't fail in CI before because we do
not test with kube-proxy on net-next).

Note that (as far as I know) both 4.19 and net-next always use bpf_sock
in CI runs, so the check on hostReachableServices is currently
superfluous. Let's have it all the same, in case something changes in
the future, to avoid unexpected breakage.

Signed-off-by: Quentin Monnet <quentin@isovalent.com>
Signed-off-by: André Martins <andre@cilium.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pchaigno pchaigno pchaigno left review comments

@joestringer joestringer joestringer approved these changes

Assignees

No one assigned

Labels

area/CI Continuous Integration testing issue or flake area/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. release-note/misc This PR makes changes that have no direct user impact.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Reduce BPF program complexity for kernel 4.19 (ipv4+ipv6+debug=false+vxlan+kubeProxyReplacement=strict VS. fragment support)

4 participants

@borkmann @coveralls @joestringer @pchaigno