Hi @dujun1990. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with @k8s-bot ok to test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://github.com/kubernetes/kubernetes/wiki/CLA-FAQ to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.

@fisherxu needs to sign cla?

@k8s-ci-robot have signed the cla, thank you

@k8s-bot ok to test

Thanks. Let's fix test failures. @dhilipkumars @fisherxu

@dujun1990 and @fisherxu

Sorry i re-introduced the tests, we need them for sure. But just disabled it when run by non-root user and if the initialization failed.

The initialization process also should take care of loading the ip_vs kernel module?

how about moving below code into ipvs.InitIpvsInterface() from NewProxier

// The system command "modeprobe" is hard coded here.
	for _, module := range ipvsModules {
		_, err := exec.Command("modprobe", module).CombinedOutput()
		if err != nil {
			glog.Warningf("Warning: Can not load module: %s in lvs proxier", module)
		}
	}

@dhilipkumars

I suggest keep modprobe ip_vs in NewProxier() which is called whiling initializing ipvs proxy instance.

We should keep it as ipvs.InitIpvsInterface() can be called by other proxy mode(userspace and iptables) in set up for the purpose of cleaning up ipvs rules.

Besides, UT failed again, please check?

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://github.com/kubernetes/kubernetes/wiki/CLA-FAQ to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.

@k8s-bot pull-kubernetes-unit test this

@thockin

Sorry for keep bothering you, but I am waiting for your feedback :)

Reviewed 2 of 18 files at r4, 1 of 10 files at r5, 1 of 11 files at r7, 4 of 5 files at r12, 5 of 8 files at r13, 7 of 9 files at r14, 9 of 9 files at r15, 4 of 5 files at r17.
Review status: 95 of 96 files reviewed at latest revision, 30 unresolved discussions.

Comments from Reviewable

/lgtm

Good luck with resyncing this to head of the iptables. I suggest maybe replaying each commit to the iptables proxier against your own. It's not going to be fun, but it needs to be done.

Don't forget all the other todos in code and review process.

Rebased again to HEAD of master branch due to conflicted with other PR(#47263). Changes only happened in pkg/features/kube_features.go

/lgtm

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dujun1990, thockin

Associated issue: 17470

The full list of commands accepted by this bot can be found here.

/test all [submit-queue is verifying that this PR is safe to merge]

Automatic merge from submit-queue (batch tested with PRs 51377, 46580, 50998, 51466, 49749)

@dujun1990: The following test failed, say /retest to rerun them all:

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Good luck with resyncing this to head of the iptables. I suggest maybe replaying each commit to the iptables proxier against your own. It's not going to be fun, but it needs to be done.

Don't forget all the other todos in code and review process.

@thockin

I created an issue for tracking all these tasks, see #51602. I will ping you when they are ready for review or I needs your suggestions :) Thanks!

Was adding module-init-tools to the debian-iptables image needed here? I don't think we even rebuilt the iptables image, so I'm guessing not? @rphillips

also cc @tallclair re: my comment above ^

@ixdy

That's because IPVS kube-proxy needs the util modprobe...

@dujun1990 Did not found any information for how to upgrade kube-proxy working more from iptables to ipvs in CHANGELOG, any comments/suggestions for this?

@gyliu513 Please xref: https://github.com/kubernetes/kubernetes/tree/master/pkg/proxy/ipvs

@m1093782566 I mean how to upgrade? If I already have a cluster running with kube-proxy iptables , how can I enable it with ipvs without impacting current workloads?

@gyliu513 drain the node to evict all pods ala:

kubectl drain --force --ignore-daemonsets --delete-local-data $NODE_NAME_HERE

Then you can stop the kube-proxy service, reconfigure it, and restart it. It very much depends on how you've deployed kube-proxy as it might be trickier if you've deployed it via a daemonset.

@gyliu513

Actually, IPVS mode is NOT an upgrade from iptables mode - not the relationship of etcd2 -> etcd3.

If your "upgrade" means upgrading the software and switch the proxy mode, it very depends on your deployment and kube-proxy provides the --proxy-mode=ipvs option. @SEJeff gave a good example.

Since ipvs rules is totally different from iptables rules and IPVS kube-proxy will clean up all iptables rules before running, service VIP will go down during "upgrade".

got it, thanks @SEJeff @m1093782566