Skip to content

agent: Fix data race when accessing d.monitorAgent#11823

Merged
borkmann merged 1 commit intomasterfrom
pr/tgraf/fix-monitor-agent-race
Jun 2, 2020
Merged

agent: Fix data race when accessing d.monitorAgent#11823
borkmann merged 1 commit intomasterfrom
pr/tgraf/fix-monitor-agent-race

Conversation

@tgraf
Copy link
Copy Markdown
Contributor

@tgraf tgraf commented Jun 2, 2020

Fixes:

2020-04-15T08:20:51.171924491Z WARNING: DATA RACE
2020-04-15T08:20:51.171929471Z Write at 0x00c0002d89c0 by main goroutine:
2020-04-15T08:20:51.171932068Z   github.com/cilium/cilium/daemon/cmd.NewDaemon()
2020-04-15T08:20:51.171934565Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:521 +0x3811
2020-04-15T08:20:51.171937059Z   github.com/cilium/cilium/daemon/cmd.(*Daemon).init()
2020-04-15T08:20:51.171939762Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:210 +0x4a4
2020-04-15T08:20:51.171942223Z   github.com/cilium/cilium/daemon/cmd.NewDaemon()
2020-04-15T08:20:51.171944604Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:508 +0x2fa5
2020-04-15T08:20:51.171947111Z   github.com/cilium/cilium/daemon/cmd.NewDaemon()
2020-04-15T08:20:51.171949522Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:411 +0x463e
2020-04-15T08:20:51.171951975Z   github.com/cilium/cilium/daemon/cmd.runDaemon()
2020-04-15T08:20:51.171954378Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:1194 +0x345
2020-04-15T08:20:51.171956805Z   github.com/cilium/cilium/daemon/cmd.glob..func1()
2020-04-15T08:20:51.171959248Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:107 +0x108
2020-04-15T08:20:51.171961654Z   github.com/cilium/cilium/daemon/cmd.initEnv()
2020-04-15T08:20:51.171964065Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:1073 +0x2c56
2020-04-15T08:20:51.171966471Z   github.com/cilium/cilium/daemon/cmd.glob..func1()
2020-04-15T08:20:51.171968921Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:105 +0xee
2020-04-15T08:20:51.171971339Z   github.com/spf13/cobra.(*Command).execute()
2020-04-15T08:20:51.171973711Z       /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:830 +0x8e0
2020-04-15T08:20:51.171983921Z   github.com/spf13/cobra.(*Command).ExecuteC()
2020-04-15T08:20:51.17198653Z       /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:914 +0x41a
2020-04-15T08:20:51.171989059Z   github.com/spf13/cobra.(*Command).Execute()
2020-04-15T08:20:51.171991461Z       /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:864 +0x251
2020-04-15T08:20:51.171993892Z   github.com/cilium/cilium/daemon/cmd.Execute()
2020-04-15T08:20:51.171996301Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:138 +0x232
2020-04-15T08:20:51.171998765Z   main.main()
2020-04-15T08:20:51.172001107Z       /go/src/github.com/cilium/cilium/daemon/main.go:22 +0x2f
2020-04-15T08:20:51.172003564Z
2020-04-15T08:20:51.172005844Z Previous read at 0x00c0002d89c0 by goroutine 82:
2020-04-15T08:20:51.172008245Z   github.com/cilium/cilium/daemon/cmd.(*Daemon).SendNotification()
2020-04-15T08:20:51.172010648Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:633 +0xc0
2020-04-15T08:20:51.172013085Z   github.com/cilium/cilium/pkg/datapath/ipcache.(*BPFListener).notifyMonitor()
2020-04-15T08:20:51.172015524Z       /go/src/github.com/cilium/cilium/pkg/datapath/ipcache/listener.go:113 +0x288
2020-04-15T08:20:51.172019826Z   github.com/cilium/cilium/pkg/datapath/ipcache.(*BPFListener).OnIPIdentityCacheChange()
2020-04-15T08:20:51.172022388Z       /go/src/github.com/cilium/cilium/pkg/datapath/ipcache/listener.go:142 +0x21e
2020-04-15T08:20:51.172024963Z   github.com/cilium/cilium/pkg/ipcache.(*IPCache).Upsert()
2020-04-15T08:20:51.172027361Z       /go/src/github.com/cilium/cilium/pkg/ipcache/ipcache.go:278 +0xaa0
2020-04-15T08:20:51.172029767Z   github.com/cilium/cilium/daemon/cmd.(*Daemon).syncEndpointsAndHostIPs()
2020-04-15T08:20:51.17203218Z       /go/src/github.com/cilium/cilium/daemon/cmd/datapath.go:275 +0xcd0
2020-04-15T08:20:51.172034647Z   github.com/cilium/cilium/daemon/cmd.(*Daemon).init.func2()
2020-04-15T08:20:51.17203703Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:224 +0x41
2020-04-15T08:20:51.172039455Z   github.com/cilium/cilium/pkg/controller.(*Controller).runController()
2020-04-15T08:20:51.172041885Z       /go/src/github.com/cilium/cilium/pkg/controller/controller.go:205 +0xc71

Fixes: #10987

@tgraf
agent: Fix data race when accessing d.monitorAgent
717d083 
Fixes:
```
2020-04-15T08:20:51.171924491Z WARNING: DATA RACE
2020-04-15T08:20:51.171929471Z Write at 0x00c0002d89c0 by main goroutine:
2020-04-15T08:20:51.171932068Z   github.com/cilium/cilium/daemon/cmd.NewDaemon()
2020-04-15T08:20:51.171934565Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:521 +0x3811
2020-04-15T08:20:51.171937059Z   github.com/cilium/cilium/daemon/cmd.(*Daemon).init()
2020-04-15T08:20:51.171939762Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:210 +0x4a4
2020-04-15T08:20:51.171942223Z   github.com/cilium/cilium/daemon/cmd.NewDaemon()
2020-04-15T08:20:51.171944604Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:508 +0x2fa5
2020-04-15T08:20:51.171947111Z   github.com/cilium/cilium/daemon/cmd.NewDaemon()
2020-04-15T08:20:51.171949522Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:411 +0x463e
2020-04-15T08:20:51.171951975Z   github.com/cilium/cilium/daemon/cmd.runDaemon()
2020-04-15T08:20:51.171954378Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:1194 +0x345
2020-04-15T08:20:51.171956805Z   github.com/cilium/cilium/daemon/cmd.glob..func1()
2020-04-15T08:20:51.171959248Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:107 +0x108
2020-04-15T08:20:51.171961654Z   github.com/cilium/cilium/daemon/cmd.initEnv()
2020-04-15T08:20:51.171964065Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:1073 +0x2c56
2020-04-15T08:20:51.171966471Z   github.com/cilium/cilium/daemon/cmd.glob..func1()
2020-04-15T08:20:51.171968921Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:105 +0xee
2020-04-15T08:20:51.171971339Z   github.com/spf13/cobra.(*Command).execute()
2020-04-15T08:20:51.171973711Z       /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:830 +0x8e0
2020-04-15T08:20:51.171983921Z   github.com/spf13/cobra.(*Command).ExecuteC()
2020-04-15T08:20:51.17198653Z       /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:914 +0x41a
2020-04-15T08:20:51.171989059Z   github.com/spf13/cobra.(*Command).Execute()
2020-04-15T08:20:51.171991461Z       /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:864 +0x251
2020-04-15T08:20:51.171993892Z   github.com/cilium/cilium/daemon/cmd.Execute()
2020-04-15T08:20:51.171996301Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:138 +0x232
2020-04-15T08:20:51.171998765Z   main.main()
2020-04-15T08:20:51.172001107Z       /go/src/github.com/cilium/cilium/daemon/main.go:22 +0x2f
2020-04-15T08:20:51.172003564Z
2020-04-15T08:20:51.172005844Z Previous read at 0x00c0002d89c0 by goroutine 82:
2020-04-15T08:20:51.172008245Z   github.com/cilium/cilium/daemon/cmd.(*Daemon).SendNotification()
2020-04-15T08:20:51.172010648Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:633 +0xc0
2020-04-15T08:20:51.172013085Z   github.com/cilium/cilium/pkg/datapath/ipcache.(*BPFListener).notifyMonitor()
2020-04-15T08:20:51.172015524Z       /go/src/github.com/cilium/cilium/pkg/datapath/ipcache/listener.go:113 +0x288
2020-04-15T08:20:51.172019826Z   github.com/cilium/cilium/pkg/datapath/ipcache.(*BPFListener).OnIPIdentityCacheChange()
2020-04-15T08:20:51.172022388Z       /go/src/github.com/cilium/cilium/pkg/datapath/ipcache/listener.go:142 +0x21e
2020-04-15T08:20:51.172024963Z   github.com/cilium/cilium/pkg/ipcache.(*IPCache).Upsert()
2020-04-15T08:20:51.172027361Z       /go/src/github.com/cilium/cilium/pkg/ipcache/ipcache.go:278 +0xaa0
2020-04-15T08:20:51.172029767Z   github.com/cilium/cilium/daemon/cmd.(*Daemon).syncEndpointsAndHostIPs()
2020-04-15T08:20:51.17203218Z       /go/src/github.com/cilium/cilium/daemon/cmd/datapath.go:275 +0xcd0
2020-04-15T08:20:51.172034647Z   github.com/cilium/cilium/daemon/cmd.(*Daemon).init.func2()
2020-04-15T08:20:51.17203703Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:224 +0x41
2020-04-15T08:20:51.172039455Z   github.com/cilium/cilium/pkg/controller.(*Controller).runController()
2020-04-15T08:20:51.172041885Z       /go/src/github.com/cilium/cilium/pkg/controller/controller.go:205 +0xc71
```

Fixes: #10987

Signed-off-by: Thomas Graf <thomas@cilium.io>
@tgraf tgraf added kind/bug This is a bug in the Cilium logic. priority/release-blocker release-note/misc This PR makes changes that have no direct user impact. labels Jun 2, 2020
@tgraf tgraf requested review from a team June 2, 2020 13:01
borkmann
borkmann approved these changes Jun 2, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@borkmann borkmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing!

@aanm
Copy link
Copy Markdown
Member

aanm commented Jun 2, 2020

test-me-please

@coveralls
Copy link
Copy Markdown

coveralls commented Jun 2, 2020

Coverage Status

Coverage increased (+0.03%) to 36.913% when pulling 717d083 on pr/tgraf/fix-monitor-agent-race into e57ae1c on master.

@maintainer-s-little-helper maintainer-s-little-helper Bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Jun 2, 2020
@borkmann borkmann merged commit 870894a into master Jun 2, 2020
@borkmann borkmann deleted the pr/tgraf/fix-monitor-agent-race branch June 2, 2020 20:46
@tklauser tklauser mentioned this pull request Jun 3, 2020
Merged
@tklauser tklauser mentioned this pull request Jun 3, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@borkmann borkmann borkmann approved these changes

Assignees

No one assigned

Labels

kind/bug This is a bug in the Cilium logic. ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/misc This PR makes changes that have no direct user impact.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

DATA RACE: d.monitorAgent unsafely accessed

6 participants

@tgraf @aanm @coveralls @borkmann @tklauser @joestringer