Skip to content

DATA RACE: pkg/node potentially caused by premature initialization of k8s watchers #11836

@aanm

Description

@aanm
2020-06-02T15:48:23.400056446Z ==================
2020-06-02T15:48:23.400149248Z WARNING: DATA RACE
2020-06-02T15:48:23.400161115Z Write at 0x000004ec90a0 by main goroutine:
2020-06-02T15:48:23.400198996Z   github.com/cilium/cilium/pkg/node.SetExternalIPv4()
2020-06-02T15:48:23.400228373Z       /go/src/github.com/cilium/cilium/pkg/node/address.go:180 +0x836
2020-06-02T15:48:23.400309217Z   github.com/cilium/cilium/pkg/k8s.GetNodeSpec()
2020-06-02T15:48:23.400316393Z       /go/src/github.com/cilium/cilium/pkg/k8s/init.go:224 +0x86c
2020-06-02T15:48:23.400319791Z   github.com/cilium/cilium/daemon/cmd.NewDaemon()
2020-06-02T15:48:23.400322838Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:403 +0x446d
2020-06-02T15:48:23.400344681Z   github.com/cilium/cilium/daemon/cmd.runDaemon()
2020-06-02T15:48:23.400352433Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:1281 +0x354
2020-06-02T15:48:23.400442153Z   github.com/cilium/cilium/daemon/cmd.glob..func1()
2020-06-02T15:48:23.400448392Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:116 +0xab
2020-06-02T15:48:23.400451689Z   github.com/cilium/cilium/daemon/cmd.glob..func1()
2020-06-02T15:48:23.400454698Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:114 +0x91
2020-06-02T15:48:23.400476627Z   github.com/spf13/cobra.(*Command).execute()
2020-06-02T15:48:23.4004836Z       /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:846 +0x8e0
2020-06-02T15:48:23.400551868Z   github.com/spf13/cobra.(*Command).ExecuteC()
2020-06-02T15:48:23.400558041Z       /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:950 +0x499
2020-06-02T15:48:23.400561363Z   github.com/spf13/cobra.(*Command).Execute()
2020-06-02T15:48:23.400564368Z       /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:887 +0x1eb
2020-06-02T15:48:23.400585966Z   github.com/cilium/cilium/daemon/cmd.Execute()
2020-06-02T15:48:23.400592944Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:147 +0x1cc
2020-06-02T15:48:23.400671182Z   main.main()
2020-06-02T15:48:23.400677133Z       /go/src/github.com/cilium/cilium/daemon/main.go:22 +0x2f
2020-06-02T15:48:23.400680516Z 
2020-06-02T15:48:23.400753266Z Previous read at 0x000004ec90a0 by goroutine 143:
2020-06-02T15:48:23.400810125Z   github.com/cilium/cilium/pkg/datapath/ipcache.(*BPFListener).OnIPIdentityCacheChange()
2020-06-02T15:48:23.400816405Z       /go/src/github.com/cilium/cilium/pkg/datapath/ipcache/listener.go:164 +0xc19
2020-06-02T15:48:23.400819705Z   github.com/cilium/cilium/pkg/ipcache.(*IPCache).Upsert()
2020-06-02T15:48:23.400822656Z       /go/src/github.com/cilium/cilium/pkg/ipcache/ipcache.go:383 +0xa7a
2020-06-02T15:48:23.400856045Z   github.com/cilium/cilium/pkg/node/manager.(*Manager).NodeUpdated()
2020-06-02T15:48:23.400881855Z       /go/src/github.com/cilium/cilium/pkg/node/manager/manager.go:368 +0x1077
2020-06-02T15:48:23.400944261Z   github.com/cilium/cilium/pkg/k8s/watchers.(*K8sWatcher).ciliumNodeInit.func2()
2020-06-02T15:48:23.400950985Z       /go/src/github.com/cilium/cilium/pkg/k8s/watchers/cilium_node.go:73 +0x28f
2020-06-02T15:48:23.400954332Z   k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate()
2020-06-02T15:48:23.400957332Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:225 +0xa5
2020-06-02T15:48:23.401033217Z   k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnUpdate()
2020-06-02T15:48:23.40104109Z       <autogenerated>:1 +0x2e
2020-06-02T15:48:23.401107817Z   github.com/cilium/cilium/pkg/k8s/informer.NewInformerWithStore.func1()
2020-06-02T15:48:23.401115157Z       /go/src/github.com/cilium/cilium/pkg/k8s/informer/informer.go:113 +0x372
2020-06-02T15:48:23.401190846Z   k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop()
2020-06-02T15:48:23.401198158Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:492 +0x396
2020-06-02T15:48:23.401201586Z   k8s.io/client-go/tools/cache.(*controller).processLoop()
2020-06-02T15:48:23.401204607Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:173 +0x83
2020-06-02T15:48:23.401245609Z   k8s.io/client-go/tools/cache.(*controller).processLoop-fm()
2020-06-02T15:48:23.401253591Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:171 +0x41
2020-06-02T15:48:23.401322164Z   k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1()
2020-06-02T15:48:23.401329422Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x75
2020-06-02T15:48:23.401332785Z   k8s.io/apimachinery/pkg/util/wait.BackoffUntil()
2020-06-02T15:48:23.401335821Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb3
2020-06-02T15:48:23.401384723Z   k8s.io/apimachinery/pkg/util/wait.JitterUntil()
2020-06-02T15:48:23.401393068Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x10d
2020-06-02T15:48:23.401452286Z   k8s.io/apimachinery/pkg/util/wait.Until()
2020-06-02T15:48:23.401458838Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4a1
2020-06-02T15:48:23.40146613Z   k8s.io/client-go/tools/cache.(*controller).Run()
2020-06-02T15:48:23.401469501Z       /go/src/github.com/cilium/cilium/vendor/k8s.io/client-go/tools/cache/controller.go:145 +0x443
2020-06-02T15:48:23.401510568Z 
2020-06-02T15:48:23.401575704Z Goroutine 143 (running) created at:
2020-06-02T15:48:23.401582125Z   github.com/cilium/cilium/pkg/k8s/watchers.(*K8sWatcher).ciliumNodeInit()
2020-06-02T15:48:23.401585513Z       /go/src/github.com/cilium/cilium/pkg/k8s/watchers/cilium_node.go:103 +0x24e

Commit sha that potentially introduced race: 05163ed

Metadata

Metadata

Assignees

Labels

kind/bugThis is a bug in the Cilium logic.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions