Skip to content

kube-proxy ipvs: fix to prevent concurrent map read and map write#107748

Merged
k8s-ci-robot merged 1 commit into
kubernetes:masterfrom
cyclinder:fix_concurrent_map
Jan 26, 2022
Merged

kube-proxy ipvs: fix to prevent concurrent map read and map write#107748
k8s-ci-robot merged 1 commit into
kubernetes:masterfrom
cyclinder:fix_concurrent_map

Conversation

@cyclinder

Copy link
Copy Markdown
Contributor

Signed-off-by: cyclinder qifeng.guo@daocloud.io

What type of PR is this?

/kind bug

What this PR does / why we need it:

fix to prevent concurrent map read and map write in kube-proxy

Which issue(s) this PR fixes:

Fixes #103260

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/bug Categorizes issue or PR as related to a bug. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 25, 2022
@k8s-ci-robot

Copy link
Copy Markdown
Contributor

@cyclinder: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Jan 25, 2022
@k8s-ci-robot k8s-ci-robot added sig/network Categorizes an issue or PR as relevant to SIG Network. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jan 25, 2022
@cyclinder

Copy link
Copy Markdown
Contributor Author

/sig network

Comment thread pkg/proxy/ipvs/graceful_termination.go Outdated
@aojea

aojea commented Jan 25, 2022

Copy link
Copy Markdown
Member

/assign @uablrek @andrewsykim

Comment thread pkg/proxy/ipvs/graceful_termination_test.go Outdated
Comment thread pkg/proxy/ipvs/graceful_termination.go Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just delete this function now or are there other calls to q.remove now?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't delete q.remove(), there other still calls to q.remove(), for example:

m.rsList.remove(rsToDelete)

@cyclinder cyclinder force-pushed the fix_concurrent_map branch 2 times, most recently from 452d199 to 5568d47 Compare January 26, 2022 01:48

@andrewsykim andrewsykim left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

deferring lgtm to @uablrek

Comment thread pkg/proxy/ipvs/graceful_termination_test.go Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So does running this test without the fix result in a panic?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, if not fixed, running this test will cause panic.

cyclinder@CycLinder-3 ~/Desktop/code/kubernetes/pkg/proxy/ipvs$ go test -race -timeout 60s -run ^Test_RaceTerminateRSList$  -v                                         ✹ ✭fix_concurrent_map 
=== RUN   Test_RaceTerminateRSList
--- PASS: Test_RaceTerminateRSList (0.00s)
==================
WARNING: DATA RACE
Write at 0x00c0001a85a0 by goroutine 21:
  runtime.mapassign_faststr()
      /Users/cyclinder/Downloads/goland/go1.17/src/runtime/map_faststr.go:202 +0x0
  k8s.io/kubernetes/pkg/proxy/ipvs.(*graceTerminateRSList).add()
      /Users/cyclinder/Desktop/code/kubernetes/pkg/proxy/ipvs/graceful_termination.go:66 +0x2a4
  k8s.io/kubernetes/pkg/proxy/ipvs.Test_RaceTerminateRSList.func1()
      /Users/cyclinder/Desktop/code/kubernetes/pkg/proxy/ipvs/graceful_termination_test.go:412 +0x4f

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

@k8s-ci-robot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andrewsykim, cyclinder

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 26, 2022
Signed-off-by: cyclinder <qifeng.guo@daocloud.io>
@cyclinder

Copy link
Copy Markdown
Contributor Author

/test pull-kubernetes-integration

@aojea

aojea commented Jan 26, 2022

Copy link
Copy Markdown
Member

/test pull-kubernetes-integration

🤔 it seems it timed out

@aojea

aojea commented Jan 26, 2022

Copy link
Copy Markdown
Member

LGTM too
defer to Lars too

@uablrek

uablrek commented Jan 26, 2022

Copy link
Copy Markdown
Contributor

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 26, 2022
@cyclinder

cyclinder commented Jan 26, 2022

Copy link
Copy Markdown
Contributor Author

/test pull-kubernetes-integration

🤔 it seems it timed out

I don't know why it time out...

@cyclinder

Copy link
Copy Markdown
Contributor Author

Thanks @aojea @andrewsykim @uablrek

@aojea

aojea commented Jan 26, 2022

Copy link
Copy Markdown
Member

@liggitt it seems we are hitting again timeouts on the integration job? did you notice something?

/test pull-kubernetes-integration

@liggitt

liggitt commented Jan 26, 2022

Copy link
Copy Markdown
Member

#107765 just started populating openapi in all integration in order to support server side apply.

Openapi construction is expensive, is that contributing to the timeout?

cc @apelisse @Jefftree

@k8s-ci-robot k8s-ci-robot merged commit e6cbcae into kubernetes:master Jan 26, 2022
@k8s-ci-robot k8s-ci-robot added this to the v1.24 milestone Jan 26, 2022
@MadhavJivrajani

Copy link
Copy Markdown
Contributor

Thanks for working on this @cyclinder!

@JornShen

Copy link
Copy Markdown
Member

@cyclinder consider cherrying pick to the lower version? cc @aojea

@cyclinder

Copy link
Copy Markdown
Contributor Author

This PR fixed for 1.24, Maybe need also fix for 1.22 and 1.23, I'm ok, but need ask for other.
/cc @aojea

@k8s-ci-robot k8s-ci-robot requested a review from aojea May 17, 2022 14:10
@aojea

aojea commented May 17, 2022

Copy link
Copy Markdown
Member

if there are users affected by the bug I think that is legit asking for the backport,

@JornShen

Copy link
Copy Markdown
Member

This PR fixed for 1.24, Maybe need also fix for 1.22 and 1.23, I'm ok, but need ask for other. /cc @aojea

+1

@cyclinder

Copy link
Copy Markdown
Contributor Author

Ok, I'll try to cp it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/ipvs cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note-none Denotes a PR that doesn't merit a release note. sig/network Categorizes an issue or PR as relevant to SIG Network. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ipvs: Concurrent map read and map write exist in kube-proxy

8 participants