Skip to content

hubble: enable metrics before starting server#11846

Merged
borkmann merged 1 commit intomasterfrom
pr/fix-datarace-hubble
Jun 3, 2020
Merged

hubble: enable metrics before starting server#11846
borkmann merged 1 commit intomasterfrom
pr/fix-datarace-hubble

Conversation

@aanm
Copy link
Copy Markdown
Member

@aanm aanm commented Jun 3, 2020

The server might depend on the metrics module to be initialized first so
we should enable this module before starting the hubble server.

Fixes: 56825e0 ("vendor: Add hubble as a dependency")
Signed-off-by: André Martins andre@cilium.io

Fixes:

2020-06-02T21:02:33.675986491Z WARNING: DATA RACE
2020-06-02T21:02:33.675989472Z Read at 0x000004ec9f80 by goroutine 28:
2020-06-02T21:02:33.675992513Z   github.com/cilium/cilium/pkg/hubble/metrics.ProcessFlow()
2020-06-02T21:02:33.675997457Z       /go/src/github.com/cilium/cilium/pkg/hubble/metrics/metrics.go:43 +0x5c5
2020-06-02T21:02:33.676001194Z   github.com/cilium/cilium/pkg/hubble/observer.(*LocalObserverServer).Start()
2020-06-02T21:02:33.676004662Z       /go/src/github.com/cilium/cilium/pkg/hubble/observer/local_observer.go:176 +0x5b5
2020-06-02T21:02:33.67600855Z 
2020-06-02T21:02:33.676012345Z Previous write at 0x000004ec9f80 by main goroutine:
2020-06-02T21:02:33.676015767Z   github.com/cilium/cilium/pkg/hubble/metrics.Init()
2020-06-02T21:02:33.676019149Z       /go/src/github.com/cilium/cilium/pkg/hubble/metrics/metrics.go:54 +0xc5
2020-06-02T21:02:33.676022508Z   github.com/cilium/cilium/pkg/hubble/metrics.EnableMetrics()
2020-06-02T21:02:33.676025697Z       /go/src/github.com/cilium/cilium/pkg/hubble/metrics/metrics.go:74 +0x81
2020-06-02T21:02:33.676029115Z   github.com/cilium/cilium/daemon/cmd.(*Daemon).launchHubble()
2020-06-02T21:02:33.676037024Z       /go/src/github.com/cilium/cilium/daemon/cmd/hubble.go:162 +0x1495
2020-06-02T21:02:33.676040507Z   github.com/cilium/cilium/daemon/cmd.runDaemon()
2020-06-02T21:02:33.67604377Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:1421 +0x141f
2020-06-02T21:02:33.676047518Z   github.com/cilium/cilium/daemon/cmd.NewDaemon()
2020-06-02T21:02:33.676051165Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:494 +0x2b5a
2020-06-02T21:02:33.676054539Z   github.com/cilium/cilium/daemon/cmd.runDaemon()
2020-06-02T21:02:33.676057758Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:1273 +0x345
2020-06-02T21:02:33.676060811Z   github.com/cilium/cilium/daemon/cmd.glob..func1()
2020-06-02T21:02:33.676063995Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:116 +0xab
2020-06-02T21:02:33.67606716Z   github.com/cilium/cilium/daemon/cmd.glob..func1()
2020-06-02T21:02:33.676070694Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:114 +0x91
2020-06-02T21:02:33.676074062Z   github.com/spf13/cobra.(*Command).execute()
2020-06-02T21:02:33.676077611Z       /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:846 +0x8e0
2020-06-02T21:02:33.676080874Z   github.com/spf13/cobra.(*Command).ExecuteC()
2020-06-02T21:02:33.676084451Z       /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:950 +0x499
2020-06-02T21:02:33.676088578Z   github.com/spf13/cobra.(*Command).Execute()
2020-06-02T21:02:33.676092009Z       /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:887 +0x1eb
2020-06-02T21:02:33.676095374Z   github.com/cilium/cilium/daemon/cmd.Execute()
2020-06-02T21:02:33.676098881Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:147 +0x1cc
2020-06-02T21:02:33.676102351Z   main.main()
2020-06-02T21:02:33.676105896Z       /go/src/github.com/cilium/cilium/daemon/main.go:22 +0x2f
2020-06-02T21:02:33.676109263Z 
2020-06-02T21:02:33.676112467Z Goroutine 28 (running) created at:
2020-06-02T21:02:33.676115684Z   github.com/cilium/cilium/daemon/cmd.(*Daemon).launchHubble()
2020-06-02T21:02:33.676119688Z       /go/src/github.com/cilium/cilium/daemon/cmd/hubble.go:107 +0x774
2020-06-02T21:02:33.676123364Z   github.com/cilium/cilium/daemon/cmd.runDaemon()
2020-06-02T21:02:33.676126674Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:1421 +0x141f
2020-06-02T21:02:33.676130032Z   github.com/cilium/cilium/daemon/cmd.NewDaemon()
2020-06-02T21:02:33.676133557Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon.go:494 +0x2b5a
2020-06-02T21:02:33.676137035Z   github.com/cilium/cilium/daemon/cmd.runDaemon()
2020-06-02T21:02:33.676140631Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:1273 +0x345
2020-06-02T21:02:33.676179998Z   github.com/cilium/cilium/daemon/cmd.glob..func1()
2020-06-02T21:02:33.676187842Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:116 +0xab
2020-06-02T21:02:33.676191173Z   github.com/cilium/cilium/daemon/cmd.glob..func1()
2020-06-02T21:02:33.676194182Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:114 +0x91
2020-06-02T21:02:33.676197227Z   github.com/spf13/cobra.(*Command).execute()
2020-06-02T21:02:33.676200185Z       /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:846 +0x8e0
2020-06-02T21:02:33.67620324Z   github.com/spf13/cobra.(*Command).ExecuteC()
2020-06-02T21:02:33.676206276Z       /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:950 +0x499
2020-06-02T21:02:33.676209324Z   github.com/spf13/cobra.(*Command).Execute()
2020-06-02T21:02:33.676212301Z       /go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:887 +0x1eb
2020-06-02T21:02:33.676215635Z   github.com/cilium/cilium/daemon/cmd.Execute()
2020-06-02T21:02:33.676218858Z       /go/src/github.com/cilium/cilium/daemon/cmd/daemon_main.go:147 +0x1cc
2020-06-02T21:02:33.676222245Z   main.main()
2020-06-02T21:02:33.676225405Z       /go/src/github.com/cilium/cilium/daemon/main.go:22 +0x2f

@aanm
hubble: enable metrics before starting server
5c41c17 
The server might depend on the metrics module to be initialized first so
we should enable this module before starting the hubble server.

Fixes: 56825e0 ("vendor: Add hubble as a dependency")
Signed-off-by: André Martins <andre@cilium.io>
@aanm aanm requested a review from a team June 3, 2020 07:15
@maintainer-s-little-helper
Copy link
Copy Markdown

maintainer-s-little-helper Bot commented Jun 3, 2020

Please set the appropriate release note label.

3 similar comments
@maintainer-s-little-helper
Copy link
Copy Markdown

maintainer-s-little-helper Bot commented Jun 3, 2020

Please set the appropriate release note label.

@maintainer-s-little-helper
Copy link
Copy Markdown

maintainer-s-little-helper Bot commented Jun 3, 2020

Please set the appropriate release note label.

@maintainer-s-little-helper
Copy link
Copy Markdown

maintainer-s-little-helper Bot commented Jun 3, 2020

Please set the appropriate release note label.

@aanm aanm added the release-note/misc This PR makes changes that have no direct user impact. label Jun 3, 2020
@aanm
Copy link
Copy Markdown
Member Author

aanm commented Jun 3, 2020

test-me-please

@coveralls
Copy link
Copy Markdown

coveralls commented Jun 3, 2020

Coverage Status

Coverage decreased (-0.002%) to 36.92% when pulling 5c41c17 on pr/fix-datarace-hubble into 0b49599 on master.

@maintainer-s-little-helper maintainer-s-little-helper Bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Jun 3, 2020
@borkmann borkmann merged commit 798f178 into master Jun 3, 2020
@borkmann borkmann deleted the pr/fix-datarace-hubble branch June 3, 2020 11:49
@tklauser tklauser mentioned this pull request Jun 4, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rolinh rolinh rolinh approved these changes

Assignees

No one assigned

Labels

ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/misc This PR makes changes that have no direct user impact.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@aanm @coveralls @rolinh @tklauser @borkmann