Skip to content

[Metricbeat] Data race with monitoring metrics for metricsets #22639

@andrewkroh

Description

@andrewkroh

There's a data race accessing the monitoring registry for a given namespace. Multiple threads are calling Namespace.GetRegistry at

registry := monitoring.GetNamespace("dataset").GetRegistry()
defer registry.Remove(metricsPath)
defer releaseStats(msw.stats)
defer wg.Done()
defer msw.close()
registry.Add(metricsPath, msw.Metrics(), monitoring.Full)

Race Detector
==================
WARNING: DATA RACE
Read at 0x00c000f66730 by goroutine 107:
  github.com/elastic/beats/v7/libbeat/monitoring.(*Namespace).GetRegistry()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/monitoring/namespace.go:50 +0xe0
  github.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start.func1()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:137 +0xcb

Previous write at 0x00c000f66730 by goroutine 61:
  github.com/elastic/beats/v7/libbeat/monitoring.(*Namespace).GetRegistry()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/monitoring/namespace.go:51 +0x521
  github.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start.func1()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:137 +0xcb

Goroutine 107 (running) created at:
  github.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:135 +0x1e1
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Start.func1()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner.go:77 +0x87
  sync.(*Once).doSlow()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:66 +0x103
  sync.(*Once).Do()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:57 +0x68
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Start()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner.go:76 +0x61
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Start.func1()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner_group.go:43 +0x9e
  sync.(*Once).doSlow()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:66 +0x103
  sync.(*Once).Do()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:57 +0x68
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Start()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner_group.go:41 +0x61
  github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Run()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:208 +0x10f
  github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:463 +0xa6a
  github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:190 +0x746
  github.com/elastic/beats/v7/libbeat/cmd/instance.Run()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:191 +0x17b
  github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/run.go:36 +0xef
  github.com/spf13/cobra.(*Command).execute()
      /Users/akroh/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830 +0x8e0
  github.com/spf13/cobra.(*Command).ExecuteC()
      /Users/akroh/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x41a
  github.com/spf13/cobra.(*Command).Execute()
      /Users/akroh/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864 +0x4c
  main.main()
      /Users/akroh/go/src/github.com/elastic/beats/x-pack/metricbeat/main.go:21 +0x2b

Goroutine 61 (running) created at:
  github.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:135 +0x1e1
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Start.func1()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner.go:77 +0x87
  sync.(*Once).doSlow()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:66 +0x103
  sync.(*Once).Do()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:57 +0x68
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Start()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner.go:76 +0x61
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Start.func1()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner_group.go:43 +0x9e
  sync.(*Once).doSlow()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:66 +0x103
  sync.(*Once).Do()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:57 +0x68
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Start()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner_group.go:41 +0x61
  github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Run()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:208 +0x10f
  github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:463 +0xa6a
  github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:190 +0x746
  github.com/elastic/beats/v7/libbeat/cmd/instance.Run()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:191 +0x17b
  github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/run.go:36 +0xef
  github.com/spf13/cobra.(*Command).execute()
      /Users/akroh/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830 +0x8e0
  github.com/spf13/cobra.(*Command).ExecuteC()
      /Users/akroh/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x41a
  github.com/spf13/cobra.(*Command).Execute()
      /Users/akroh/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864 +0x4c
  main.main()
      /Users/akroh/go/src/github.com/elastic/beats/x-pack/metricbeat/main.go:21 +0x2b
==================
==================
WARNING: DATA RACE
Read at 0x00c000f4fa70 by goroutine 107  github.com/elastic/beats/v7/libbeat/monitoring.(*Registry).Add()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/monitoring/registry.go:148 +0x5b
  github.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start.func1()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:144 +0x2b6

Previous write at 0x00c000f4fa70 by goroutine 61:
  github.com/elastic/beats/v7/libbeat/monitoring.NewRegistry()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/monitoring/registry.go:51 +0x464
  github.com/elastic/beats/v7/libbeat/monitoring.(*Namespace).GetRegistry()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/monitoring/namespace.go:51 +0x511
  github.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start.func1()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:137 +0xcb

Goroutine 107 (running) created at:
  github.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:135 +0x1e1
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Start.func1()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner.go:77 +0x87
  sync.(*Once).doSlow()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:66 +0x103
  sync.(*Once).Do()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:57 +0x68
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Start()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner.go:76 +0x61
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Start.func1()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner_group.go:43 +0x9e
  sync.(*Once).doSlow()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:66 +0x103
  sync.(*Once).Do()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:57 +0x68
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Start()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner_group.go:41 +0x61
  github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Run()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:208 +0x10f
  github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:463 +0xa6a
  github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:190 +0x746
  github.com/elastic/beats/v7/libbeat/cmd/instance.Run()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:191 +0x17b
  github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/run.go:36 +0xef
  github.com/spf13/cobra.(*Command).execute()
      /Users/akroh/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830 +0x8e0
  github.com/spf13/cobra.(*Command).ExecuteC()
      /Users/akroh/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x41a
  github.com/spf13/cobra.(*Command).Execute()
      /Users/akroh/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864 +0x4c
  main.main()
      /Users/akroh/go/src/github.com/elastic/beats/x-pack/metricbeat/main.go:21 +0x2b

Goroutine 61 (running) created at:
  github.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:135 +0x1e1
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Start.func1()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner.go:77 +0x87
  sync.(*Once).doSlow()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:66 +0x103
  sync.(*Once).Do()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:57 +0x68
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Start()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner.go:76 +0x61
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Start.func1()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner_group.go:43 +0x9e
  sync.(*Once).doSlow()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:66 +0x103
  sync.(*Once).Do()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:57 +0x68
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Start()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner_group.go:41 +0x61
  github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Run()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:208 +0x10f
  github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:463 +0xa6a
  github.com/elastic/beats/v7  github.com/elastic/beats/v7/libbeat/cmd/instance.Run()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:191 +0x17b
  github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/run.go:36 +0xef
  github.com/spf13/cobra.(*Command).execute()
      /Users/akroh/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830 +0x8e0
  github.com/spf13/cobra.(*Command).ExecuteC()
      /Users/akroh/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x41a
  github.com/spf13/cobra.(*Command).Execute()
      /Users/akroh/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864 +0x4c
  main.main()
      /Users/akroh/go/src/github.com/elastic/beats/x-pack/metricbeat/main.go:21 +0x2b
==================
==================
WARNING: DATA RACE
Read at 0x00c000f3cdcd by goroutine 107:
  github.com/elastic/beats/v7/libbeat/monitoring.(*Registry).Add()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/monitoring/registry.go:149 +0x7b
  github.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start.func1()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:144 +0x2b6

Previous write at 0x00c000f3cdcd by goroutine 61:
  github.com/elastic/beats/v7/libbeat/monitoring.ensureOptions()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/monitoring/opts.go:89 +0x247
  github.com/elastic/beats/v7/libbeat/monitoring.applyOpts()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/monitoring/opts.go:74 +0x1ca
  github.com/elastic/beats/v7/libbeat/monitoring.NewRegistry()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/monitoring/registry.go:52 +0x428
  github.com/elastic/beats/v7/libbeat/monitoring.(*Namespace).GetRegistry()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/monitoring/namespace.go:51 +0x511
  github.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start.func1()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:137 +0xcb

Goroutine 107 (running) created at:
  github.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:135 +0x1e1
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Start.func1()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner.go:77 +0x87
  sync.(*Once).doSlow()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:66 +0x103
  sync.(*Once).Do()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:57 +0x68
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Start()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner.go:76 +0x61
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Start.func1()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner_group.go:43 +0x9e
  sync.(*Once).doSlow()
      /Users/akroh/.gvm/versions/go1.14.4.darw  sync.(*Once).Do()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:57 +0x68
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Start()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner_group.go:41 +0x61
  github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Run()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:208 +0x10f
  github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:463 +0xa6a
  github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:190 +0x746
  github.com/elastic/beats/v7/libbeat/cmd/instance.Run()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:191 +0x17b
  github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/run.go:36 +0xef
  github.com/spf13/cobra.(*Command).execute()
      /Users/akroh/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830 +0x8e0
  github.com/spf13/cobra.(*Command).ExecuteC()
      /Users/akroh/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x41a
  github.com/spf13/cobra.(*Command).Execute()
      /Users/akroh/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864 +0x4c
  main.main()
      /Users/akroh/go/src/github.com/elastic/beats/x-pack/metricbeat/main.go:21 +0x2b

Goroutine 61 (running) created at:
  github.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:135 +0x1e1
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Start.func1()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner.go:77 +0x87
  sync.(*Once).doSlow()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:66 +0x103
  sync.(*Once).Do()
      /  github.com/elastic/beats/v7/metricbeat/mb/module.(*runner).Start()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner.go:76 +0x61
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Start.func1()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner_group.go:43 +0x9e
  sync.(*Once).doSlow()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:66 +0x103
  sync.(*Once).Do()
      /Users/akroh/.gvm/versions/go1.14.4.darwin.amd64/src/sync/once.go:57 +0x68
  github.com/elastic/beats/v7/metricbeat/mb/module.(*runnerGroup).Start()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/mb/module/runner_group.go:41 +0x61
  github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Run()
      /Users/akroh/go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:208 +0x10f
  github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:463 +0xa6a
  github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:190 +0x746
  github.com/elastic/beats/v7/libbeat/cmd/instance.Run()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:191 +0x17b
  github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1()
      /Users/akroh/go/src/github.com/elastic/beats/libbeat/cmd/run.go:36 +0xef
  github.com/spf13/cobra.(*Command).execute()
      /Users/akroh/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830 +0x8e0
  github.com/spf13/cobra.(*Command).ExecuteC()
      /Users/akroh/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x41a
  github.com/spf13/cobra.(*Command).Execute()
      /Users/akroh/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864 +0x4c
  main.main()
      /Users/akroh/go/src/github.com/elastic/beats/x-pack/metricbeat/main.go:21 +0x2b
==================

Metadata

Metadata

Assignees

No one assigned

    Labels

    MetricbeatMetricbeatTeam:Services(Deprecated) Label for the former Integrations-Services teambuglibbeat

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions