Skip to content

data race in TestOnlyProviderStaleTargetsAreDropped #15538

New issue
New issue
Open
#15637
Open
data race in TestOnlyProviderStaleTargetsAreDropped#15538
#15637
Assignees
bwplotka
@machine424

Description

@machine424
machine424
opened on Dec 4, 2024

What did you do?

the test failed on unrelated change in #15529

This isn't the first occurrence of the data race

What did you expect to see?

No response

What did you see instead? Under which circumstances?

==================
WARNING: DATA RACE
Read at 0x00c0004fe320 by goroutine 761:
  github.com/prometheus/prometheus/scrape.(*scrapePool).sync()
      /__w/prometheus/prometheus/scrape/scrape.go:479 +0x18f
  github.com/prometheus/prometheus/scrape.(*scrapePool).Sync()
      /__w/prometheus/prometheus/scrape/scrape.go:462 +0x564
  github.com/prometheus/prometheus/scrape.(*Manager).reload.func1()
      /__w/prometheus/prometheus/scrape/manager.go:202 +0x4d
  github.com/prometheus/prometheus/scrape.(*Manager).reload.gowrap1()
      /__w/prometheus/prometheus/scrape/manager.go:204 +0x6e

Previous write at 0x00c0004fe320 by goroutine 710:
  github.com/prometheus/prometheus/config.(*ScrapeConfig).Validate()
      /__w/prometheus/prometheus/config/config.go:795 +0x1e5
  github.com/prometheus/prometheus/config.(*Config).GetScrapeConfigs()
      /__w/prometheus/prometheus/config/config.go:313 +0x204
  github.com/prometheus/prometheus/scrape.applyConfig()
      /__w/prometheus/prometheus/scrape/manager_test.go:1113 +0x14a
  github.com/prometheus/prometheus/scrape.TestOnlyProviderStaleTargetsAreDropped()
      /__w/prometheus/prometheus/scrape/manager_test.go:1339 +0x75d
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44

Goroutine 761 (running) created at:
  github.com/prometheus/prometheus/scrape.(*Manager).reload()
      /__w/prometheus/prometheus/scrape/manager.go:201 +0xb93
  github.com/prometheus/prometheus/scrape.(*Manager).reloader()
      /__w/prometheus/prometheus/scrape/manager.go:161 +0x111
  github.com/prometheus/prometheus/scrape.(*Manager).Run.gowrap1()
      /__w/prometheus/prometheus/scrape/manager.go:122 +0x33

Goroutine 710 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1743 +0x825
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:2168 +0x85
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.runTests()
      /usr/local/go/src/testing/testing.go:2166 +0x8be
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:2034 +0xf17
  go.uber.org/goleak.VerifyTestMain()
      /go/pkg/mod/go.uber.org/goleak@v1.3.0/testmain.go:53 +0x64
  github.com/prometheus/prometheus/util/testutil.TolerantVerifyLeak()
      /__w/prometheus/prometheus/util/testutil/testing.go:34 +0x310
  github.com/prometheus/prometheus/scrape.TestMain()
      /__w/prometheus/prometheus/scrape/scrape_test.go:65 +0x172
  main.main()
      _testmain.go:241 +0x16d
==================
==================
WARNING: DATA RACE
Read at 0x00c0004fe328 by goroutine 761:
  github.com/prometheus/prometheus/scrape.(*scrapePool).sync()
      /__w/prometheus/prometheus/scrape/scrape.go:480 +0x1d8
  github.com/prometheus/prometheus/scrape.(*scrapePool).Sync()
      /__w/prometheus/prometheus/scrape/scrape.go:462 +0x564
  github.com/prometheus/prometheus/scrape.(*Manager).reload.func1()
      /__w/prometheus/prometheus/scrape/manager.go:202 +0x4d
  github.com/prometheus/prometheus/scrape.(*Manager).reload.gowrap1()
      /__w/prometheus/prometheus/scrape/manager.go:204 +0x6e

Previous write at 0x00c0004fe328 by goroutine 710:
  github.com/prometheus/prometheus/config.(*ScrapeConfig).Validate()
      /__w/prometheus/prometheus/config/config.go:798 +0x23c
  github.com/prometheus/prometheus/config.(*Config).GetScrapeConfigs()
      /__w/prometheus/prometheus/config/config.go:313 +0x204
  github.com/prometheus/prometheus/scrape.applyConfig()
      /__w/prometheus/prometheus/scrape/manager_test.go:1113 +0x14a
  github.com/prometheus/prometheus/scrape.TestOnlyProviderStaleTargetsAreDropped()
      /__w/prometheus/prometheus/scrape/manager_test.go:1339 +0x75d
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44

Goroutine 761 (running) created at:
  github.com/prometheus/prometheus/scrape.(*Manager).reload()
      /__w/prometheus/prometheus/scrape/manager.go:201 +0xb93
  github.com/prometheus/prometheus/scrape.(*Manager).reloader()
      /__w/prometheus/prometheus/scrape/manager.go:161 +0x111
  github.com/prometheus/prometheus/scrape.(*Manager).Run.gowrap1()
      /__w/prometheus/prometheus/scrape/manager.go:122 +0x33

Goroutine 710 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1743 +0x825
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:2168 +0x85
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.runTests()
      /usr/local/go/src/testing/testing.go:2166 +0x8be
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:2034 +0xf17
  go.uber.org/goleak.VerifyTestMain()
      /go/pkg/mod/go.uber.org/goleak@v1.3.0/testmain.go:53 +0x64
  github.com/prometheus/prometheus/util/testutil.TolerantVerifyLeak()
      /__w/prometheus/prometheus/util/testutil/testing.go:34 +0x310
  github.com/prometheus/prometheus/scrape.TestMain()
      /__w/prometheus/prometheus/scrape/scrape_test.go:65 +0x172
  main.main()
      _testmain.go:241 +0x16d
==================
==================
WARNING: DATA RACE
Read at 0x00c0004fe338 by goroutine 761:
  github.com/prometheus/prometheus/scrape.(*scrapePool).sync()
      /__w/prometheus/prometheus/scrape/scrape.go:484 +0x2d1
  github.com/prometheus/prometheus/scrape.(*scrapePool).Sync()
      /__w/prometheus/prometheus/scrape/scrape.go:462 +0x564
  github.com/prometheus/prometheus/scrape.(*Manager).reload.func1()
      /__w/prometheus/prometheus/scrape/manager.go:202 +0x4d
  github.com/prometheus/prometheus/scrape.(*Manager).reload.gowrap1()
      /__w/prometheus/prometheus/scrape/manager.go:204 +0x6e

Previous write at 0x00c0004fe338 by goroutine 710:
  github.com/prometheus/prometheus/config.(*ScrapeConfig).Validate()
      /__w/prometheus/prometheus/config/config.go:804 +0x2ea
  github.com/prometheus/prometheus/config.(*Config).GetScrapeConfigs()
      /__w/prometheus/prometheus/config/config.go:313 +0x204
  github.com/prometheus/prometheus/scrape.applyConfig()
      /__w/prometheus/prometheus/scrape/manager_test.go:1113 +0x14a
  github.com/prometheus/prometheus/scrape.TestOnlyProviderStaleTargetsAreDropped()
      /__w/prometheus/prometheus/scrape/manager_test.go:1339 +0x75d
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44

Goroutine 761 (running) created at:
  github.com/prometheus/prometheus/scrape.(*Manager).reload()
      /__w/prometheus/prometheus/scrape/manager.go:201 +0xb93
  github.com/prometheus/prometheus/scrape.(*Manager).reloader()
      /__w/prometheus/prometheus/scrape/manager.go:161 +0x111
  github.com/prometheus/prometheus/scrape.(*Manager).Run.gowrap1()
      /__w/prometheus/prometheus/scrape/manager.go:122 +0x33

Goroutine 710 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1743 +0x825
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:2168 +0x85
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.runTests()
      /usr/local/go/src/testing/testing.go:2166 +0x8be
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:2034 +0xf17
  go.uber.org/goleak.VerifyTestMain()
      /go/pkg/mod/go.uber.org/goleak@v1.3.0/testmain.go:53 +0x64
  github.com/prometheus/prometheus/util/testutil.TolerantVerifyLeak()
      /__w/prometheus/prometheus/util/testutil/testing.go:34 +0x310
  github.com/prometheus/prometheus/scrape.TestMain()
      /__w/prometheus/prometheus/scrape/scrape_test.go:65 +0x172
  main.main()
      _testmain.go:241 +0x16d
==================
==================
WARNING: DATA RACE
Read at 0x00c0004fe340 by goroutine 761:
  github.com/prometheus/prometheus/scrape.(*scrapePool).sync()
      /__w/prometheus/prometheus/scrape/scrape.go:485 +0x34e
  github.com/prometheus/prometheus/scrape.(*scrapePool).Sync()
      /__w/prometheus/prometheus/scrape/scrape.go:462 +0x564
  github.com/prometheus/prometheus/scrape.(*Manager).reload.func1()
      /__w/prometheus/prometheus/scrape/manager.go:202 +0x4d
  github.com/prometheus/prometheus/scrape.(*Manager).reload.gowrap1()
      /__w/prometheus/prometheus/scrape/manager.go:204 +0x6e

Previous write at 0x00c0004fe340 by goroutine 710:
  github.com/prometheus/prometheus/config.(*ScrapeConfig).Validate()
      /__w/prometheus/prometheus/config/config.go:807 +0x344
  github.com/prometheus/prometheus/config.(*Config).GetScrapeConfigs()
      /__w/prometheus/prometheus/config/config.go:313 +0x204
  github.com/prometheus/prometheus/scrape.applyConfig()
      /__w/prometheus/prometheus/scrape/manager_test.go:1113 +0x14a
  github.com/prometheus/prometheus/scrape.TestOnlyProviderStaleTargetsAreDropped()
      /__w/prometheus/prometheus/scrape/manager_test.go:1339 +0x75d
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44

Goroutine 761 (running) created at:
  github.com/prometheus/prometheus/scrape.(*Manager).reload()
      /__w/prometheus/prometheus/scrape/manager.go:201 +0xb93
  github.com/prometheus/prometheus/scrape.(*Manager).reloader()
      /__w/prometheus/prometheus/scrape/manager.go:161 +0x111
  github.com/prometheus/prometheus/scrape.(*Manager).Run.gowrap1()
      /__w/prometheus/prometheus/scrape/manager.go:122 +0x33

Goroutine 710 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1743 +0x825
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:2168 +0x85
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.runTests()
      /usr/local/go/src/testing/testing.go:2166 +0x8be
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:2034 +0xf17
  go.uber.org/goleak.VerifyTestMain()
      /go/pkg/mod/go.uber.org/goleak@v1.3.0/testmain.go:53 +0x64
  github.com/prometheus/prometheus/util/testutil.TolerantVerifyLeak()
      /__w/prometheus/prometheus/util/testutil/testing.go:34 +0x310
  github.com/prometheus/prometheus/scrape.TestMain()
      /__w/prometheus/prometheus/scrape/scrape_test.go:65 +0x172
  main.main()
      _testmain.go:241 +0x16d
==================
==================
WARNING: DATA RACE
Read at 0x00c0004fe348 by goroutine 761:
  github.com/prometheus/prometheus/scrape.(*scrapePool).sync()
      /__w/prometheus/prometheus/scrape/scrape.go:486 +0x3cc
  github.com/prometheus/prometheus/scrape.(*scrapePool).Sync()
      /__w/prometheus/prometheus/scrape/scrape.go:462 +0x564
  github.com/prometheus/prometheus/scrape.(*Manager).reload.func1()
      /__w/prometheus/prometheus/scrape/manager.go:202 +0x4d
  github.com/prometheus/prometheus/scrape.(*Manager).reload.gowrap1()
      /__w/prometheus/prometheus/scrape/manager.go:204 +0x6e

Previous write at 0x00c0004fe348 by goroutine 710:
  github.com/prometheus/prometheus/config.(*ScrapeConfig).Validate()
      /__w/prometheus/prometheus/config/config.go:810 +0x39b
  github.com/prometheus/prometheus/config.(*Config).GetScrapeConfigs()
      /__w/prometheus/prometheus/config/config.go:313 +0x204
  github.com/prometheus/prometheus/scrape.applyConfig()
      /__w/prometheus/prometheus/scrape/manager_test.go:1113 +0x14a
  github.com/prometheus/prometheus/scrape.TestOnlyProviderStaleTargetsAreDropped()
      /__w/prometheus/prometheus/scrape/manager_test.go:1339 +0x75d
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44

Goroutine 761 (running) created at:
  github.com/prometheus/prometheus/scrape.(*Manager).reload()
      /__w/prometheus/prometheus/scrape/manager.go:201 +0xb93
  github.com/prometheus/prometheus/scrape.(*Manager).reloader()
      /__w/prometheus/prometheus/scrape/manager.go:161 +0x111
  github.com/prometheus/prometheus/scrape.(*Manager).Run.gowrap1()
      /__w/prometheus/prometheus/scrape/manager.go:122 +0x33

Goroutine 710 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1743 +0x825
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:2168 +0x85
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.runTests()
      /usr/local/go/src/testing/testing.go:2166 +0x8be
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:2034 +0xf17
  go.uber.org/goleak.VerifyTestMain()
      /go/pkg/mod/go.uber.org/goleak@v1.3.0/testmain.go:53 +0x64
  github.com/prometheus/prometheus/util/testutil.TolerantVerifyLeak()
      /__w/prometheus/prometheus/util/testutil/testing.go:34 +0x310
  github.com/prometheus/prometheus/scrape.TestMain()
      /__w/prometheus/prometheus/scrape/scrape_test.go:65 +0x172
  main.main()
      _testmain.go:241 +0x16d
==================
==================
WARNING: DATA RACE
Read at 0x00c0004fe368 by goroutine 761:
  github.com/prometheus/prometheus/scrape.(*scrapePool).sync()
      /__w/prometheus/prometheus/scrape/scrape.go:499 +0x68d
  github.com/prometheus/prometheus/scrape.(*scrapePool).Sync()
      /__w/prometheus/prometheus/scrape/scrape.go:462 +0x564
  github.com/prometheus/prometheus/scrape.(*Manager).reload.func1()
      /__w/prometheus/prometheus/scrape/manager.go:202 +0x4d
  github.com/prometheus/prometheus/scrape.(*Manager).reload.gowrap1()
      /__w/prometheus/prometheus/scrape/manager.go:204 +0x6e

Previous write at 0x00c0004fe368 by goroutine 710:
  github.com/prometheus/prometheus/config.(*ScrapeConfig).Validate()
      /__w/prometheus/prometheus/config/config.go:842 +0x82f
  github.com/prometheus/prometheus/config.(*Config).GetScrapeConfigs()
      /__w/prometheus/prometheus/config/config.go:313 +0x204
  github.com/prometheus/prometheus/scrape.applyConfig()
      /__w/prometheus/prometheus/scrape/manager_test.go:1113 +0x14a
  github.com/prometheus/prometheus/scrape.TestOnlyProviderStaleTargetsAreDropped()
      /__w/prometheus/prometheus/scrape/manager_test.go:1339 +0x75d
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44

Goroutine 761 (running) created at:
  github.com/prometheus/prometheus/scrape.(*Manager).reload()
      /__w/prometheus/prometheus/scrape/manager.go:201 +0xb93
  github.com/prometheus/prometheus/scrape.(*Manager).reloader()
      /__w/prometheus/prometheus/scrape/manager.go:161 +0x111
  github.com/prometheus/prometheus/scrape.(*Manager).Run.gowrap1()
      /__w/prometheus/prometheus/scrape/manager.go:122 +0x33

Goroutine 710 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1743 +0x825
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:2168 +0x85
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.runTests()
      /usr/local/go/src/testing/testing.go:2166 +0x8be
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:2034 +0xf17
  go.uber.org/goleak.VerifyTestMain()
      /go/pkg/mod/go.uber.org/goleak@v1.3.0/testmain.go:53 +0x64
  github.com/prometheus/prometheus/util/testutil.TolerantVerifyLeak()
      /__w/prometheus/prometheus/util/testutil/testing.go:34 +0x310
  github.com/prometheus/prometheus/scrape.TestMain()
      /__w/prometheus/prometheus/scrape/scrape_test.go:65 +0x172
  main.main()
      _testmain.go:241 +0x16d
==================
==================
WARNING: DATA RACE
Read at 0x00c0004fe330 by goroutine 761:
  github.com/prometheus/prometheus/scrape.(*scrapePool).refreshTargetLimitErr()
      /__w/prometheus/prometheus/scrape/scrape.go:599 +0x8a
  github.com/prometheus/prometheus/scrape.(*scrapePool).sync()
      /__w/prometheus/prometheus/scrape/scrape.go:580 +0x179d
  github.com/prometheus/prometheus/scrape.(*scrapePool).Sync()
      /__w/prometheus/prometheus/scrape/scrape.go:462 +0x564
  github.com/prometheus/prometheus/scrape.(*Manager).reload.func1()
      /__w/prometheus/prometheus/scrape/manager.go:202 +0x4d
  github.com/prometheus/prometheus/scrape.(*Manager).reload.gowrap1()
      /__w/prometheus/prometheus/scrape/manager.go:204 +0x6e

Previous write at 0x00c0004fe330 by goroutine 710:
  github.com/prometheus/prometheus/config.(*ScrapeConfig).Validate()
      /__w/prometheus/prometheus/config/config.go:801 +0x293
  github.com/prometheus/prometheus/config.(*Config).GetScrapeConfigs()
      /__w/prometheus/prometheus/config/config.go:313 +0x204
  github.com/prometheus/prometheus/scrape.applyConfig()
      /__w/prometheus/prometheus/scrape/manager_test.go:1113 +0x14a
  github.com/prometheus/prometheus/scrape.TestOnlyProviderStaleTargetsAreDropped()
      /__w/prometheus/prometheus/scrape/manager_test.go:1339 +0x75d
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1743 +0x44

Goroutine 761 (running) created at:
  github.com/prometheus/prometheus/scrape.(*Manager).reload()
      /__w/prometheus/prometheus/scrape/manager.go:201 +0xb93
  github.com/prometheus/prometheus/scrape.(*Manager).reloader()
      /__w/prometheus/prometheus/scrape/manager.go:161 +0x111
  github.com/prometheus/prometheus/scrape.(*Manager).Run.gowrap1()
      /__w/prometheus/prometheus/scrape/manager.go:122 +0x33

Goroutine 710 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1743 +0x825
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:2168 +0x85
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1690 +0x226
  testing.runTests()
      /usr/local/go/src/testing/testing.go:2166 +0x8be
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:2034 +0xf17
  go.uber.org/goleak.VerifyTestMain()
      /go/pkg/mod/go.uber.org/goleak@v1.3.0/testmain.go:53 +0x64
  github.com/prometheus/prometheus/util/testutil.TolerantVerifyLeak()
      /__w/prometheus/prometheus/util/testutil/testing.go:34 +0x310
  github.com/prometheus/prometheus/scrape.TestMain()
      /__w/prometheus/prometheus/scrape/scrape_test.go:65 +0x172
  main.main()
      _testmain.go:241 +0x16d
==================
--- FAIL: TestOnlyProviderStaleTargetsAreDropped (0.30s)
    testing.go:1399: race detected during execution of test
2024/12/04 16:25:33 http: TLS handshake error from 127.0.0.1:55474: remote error: tls: bad certificate
FAIL
FAIL	github.com/prometheus/prometheus/scrape	24.160s

System information

No response

Prometheus version

No response

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions