ui: race condition during initialization with login enabled #25771
Description
When login is enabled, there seems to be a race condition during initialization, manifesting as the error "settings/cluster/settings.go:164 Version() was called before having been initialized". It seems to occur when an existing admin ui browser tab is open during node startup, issuing requests such as health checks. It was previously theorized that #24945 fixed this issue, but it looks like it's come up since.
Stack trace for my repro just now:
$ COCKROACH_EXPERIMENTAL_REQUIRE_WEB_LOGIN=true ./cockroach start --certs-dir clusters/login-test/certs --store clusters/login-test/cockroach-data/
F180521 18:20:17.173359 127 settings/cluster/settings.go:164 Version() was called before having been initialized
goroutine 127 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0xc4204fa500, 0xc4204fa540, 0x6fe6c00, 0x1c)
/Users/couch/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:956 +0xcf
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x76ab5e0, 0xc400000004, 0x6fe6cb6, 0x1c, 0xa4, 0xc4207f6400, 0x33)
/Users/couch/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:832 +0x753
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x63f4aa0, 0xc420054098, 0x4, 0x2, 0x0, 0x0, 0xc420b96c08, 0x1, 0x1)
/Users/couch/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:154 +0x2e5
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x63f4aa0, 0xc420054098, 0x1, 0xc400000004, 0x0, 0x0, 0xc420b96c08, 0x1, 0x1)
/Users/couch/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:55 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.Fatal(0x63f4aa0, 0xc420054098, 0xc420b96c08, 0x1, 0x1)
/Users/couch/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:158 +0x6c
github.com/cockroachdb/cockroach/pkg/settings/cluster.(*ExposedClusterVersion).Version(0xc420705830, 0x0, 0x0, 0x0, 0x5ea7e60)
/Users/couch/go/src/github.com/cockroachdb/cockroach/pkg/settings/cluster/settings.go:164 +0x161
github.com/cockroachdb/cockroach/pkg/settings/cluster.(*ExposedClusterVersion).IsActive(0xc420705830, 0x7, 0x608d904)
/Users/couch/go/src/github.com/cockroachdb/cockroach/pkg/settings/cluster/settings.go:199 +0x2b
github.com/cockroachdb/cockroach/pkg/kv.(*RangeDescriptorCache).performRangeLookup(0xc42052a420, 0x63f4b20, 0xc42024f260, 0xc4202b1c90, 0xe, 0x10, 0x6417800, 0xc4202868c0, 0xc4206eb000, 0x1000, ...)
/Users/couch/go/src/github.com/cockroachdb/cockroach/pkg/kv/range_cache.go:419 +0x103
github.com/cockroachdb/cockroach/pkg/kv.(*RangeDescriptorCache).lookupRangeDescriptorInternal.func3(0x0, 0x0, 0x0, 0x0)
/Users/couch/go/src/github.com/cockroachdb/cockroach/pkg/kv/range_cache.go:310 +0x132
github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall(0xc42052a460, 0xc42025e780, 0xc420312820, 0x14, 0xc4206b7d00)
/Users/couch/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:118 +0x2e
created by github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).DoChan
/Users/couch/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:111 +0x2d0