race condition between watcheabler.Map.Close and Subscribe #5505
Description
Description:
Looks like there's some race condition stemming from the use of telepresenceio/watchable in Envoy Gateway which ends up calling WaitGroup.Add from a goroutine and WaitGroup.Wait from another goroutine.
Repro steps:
- Checkout this branch of AI Gateway: cli: scaffolds 'aigw run' command ai-gateway#498
go test ./cmd/aigw/... -race -count=100
Unfortunately, this is flaky and you might need to run multiple times.
Logs:
WARNING: DATA RACE
Write at 0x00c001db0ef8 by goroutine 128:
runtime.racewrite()
<autogenerated>:1 +0x10
github.com/telepresenceio/watchable.(*Map[go.shape.string,go.shape.*uint8]).Close()
/Users/mathetake/go/pkg/mod/github.com/telepresenceio/watchable@v0.0.0-20220726211108-9bb86f92afa7/map.go:229 +0x18c
github.com/envoyproxy/gateway/internal/cmd.setupRunners()
/Users/mathetake/go/pkg/mod/github.com/envoyproxy/gateway@v0.5.0-rc.1.0.20250314011001-d8d3ca4c1b01/internal/cmd/server.go:229 +0xbe4
github.com/envoyproxy/gateway/internal/cmd.server.func1()
/Users/mathetake/go/pkg/mod/github.com/envoyproxy/gateway@v0.5.0-rc.1.0.20250314011001-d8d3ca4c1b01/internal/cmd/server.go:59 +0x88
github.com/envoyproxy/gateway/internal/envoygateway/config/loader.(*Loader).runHook.func1()
/Users/mathetake/go/pkg/mod/github.com/envoyproxy/gateway@v0.5.0-rc.1.0.20250314011001-d8d3ca4c1b01/internal/envoygateway/config/loader/configloader.go:110 +0x7c
github.com/envoyproxy/gateway/internal/envoygateway/config/loader.(*Loader).runHook.gowrap1()
/Users/mathetake/go/pkg/mod/github.com/envoyproxy/gateway@v0.5.0-rc.1.0.20250314011001-d8d3ca4c1b01/internal/envoygateway/config/loader/configloader.go:113 +0x54
Previous read at 0x00c001db0ef8 by goroutine 150:
runtime.raceread()
<autogenerated>:1 +0x10
github.com/telepresenceio/watchable.(*Map[go.shape.string,go.shape.*uint8]).SubscribeSubset()
/Users/mathetake/go/pkg/mod/github.com/telepresenceio/watchable@v0.0.0-20220726211108-9bb86f92afa7/map.go:280 +0xa8
github.com/telepresenceio/watchable.(*Map[go.shape.string,go.shape.*uint8]).Subscribe()
/Users/mathetake/go/pkg/mod/github.com/telepresenceio/watchable@v0.0.0-20220726211108-9bb86f92afa7/map.go:262 +0xb4
github.com/envoyproxy/gateway/internal/xds/server/runner.(*Runner).subscribeAndTranslate()
/Users/mathetake/go/pkg/mod/github.com/envoyproxy/gateway@v0.5.0-rc.1.0.20250314011001-d8d3ca4c1b01/internal/xds/server/runner/runner.go:146 +0x58
github.com/envoyproxy/gateway/internal/xds/server/runner.(*Runner).Start.gowrap2()
/Users/mathetake/go/pkg/mod/github.com/envoyproxy/gateway@v0.5.0-rc.1.0.20250314011001-d8d3ca4c1b01/internal/xds/server/runner/runner.go:103 +0x4c