-
Notifications
You must be signed in to change notification settings - Fork 4.1k
setting ‹"cluster.preserve_downgrade_option"› to ‹"20.2"› failed: cannot set cluster.preserve_downgrade_option to ‹20.2› (cluster version is 21.1) #68335
Copy link
Copy link
Closed
Labels
A-configurabilityPertains to cluster settings, CLI flags, env vars etcPertains to cluster settings, CLI flags, env vars etcC-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.O-communityOriginated from the communityOriginated from the communityT-multitenantIssues owned by the multi-tenant virtual teamIssues owned by the multi-tenant virtual teamX-blathers-triagedblathers was able to find an ownerblathers was able to find an owner
Description
Describe the problem
After upgrading our clusters from 20.2 to 21.1, all nodes now appear to want to adjust cluster.preserve_downgrade_option when starting.
+ exec /cockroach/cockroach start --logtostderr=WARNING --certs-dir /cockroach/cockroach-certs --advertise-addr cockroachdb-0.cockroachdb.***.svc.cluster.local --http-addr 0.0.0.0 --joi
n cockroachdb-0.cockroachdb,cockroachdb-1.cockroachdb,cockroachdb-2.cockroachdb --cache 25% --max-sql-memory 25%
Flag --logtostderr has been deprecated, use --log instead to specify 'sinks: {stderr: {filter: ...}}'.
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 setting ‹"cluster.preserve_downgrade_option"› to ‹"20.2"› failed: cannot set cluster.preserve_downgrade_option to ‹20.2› (cluster version
is 21.1)
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +(1) attached stack trace
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + -- stack trace:
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | github.com/cockroachdb/cockroach/pkg/clusterversion.registerPreserveDowngradeVersionSetting.func1
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | /go/src/github.com/cockroachdb/cockroach/pkg/clusterversion/setting.go:259
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | github.com/cockroachdb/cockroach/pkg/settings.(*StringSetting).Validate
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | /go/src/github.com/cockroachdb/cockroach/pkg/settings/string.go:65
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | github.com/cockroachdb/cockroach/pkg/settings.(*StringSetting).set
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | /go/src/github.com/cockroachdb/cockroach/pkg/settings/string.go:79
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | github.com/cockroachdb/cockroach/pkg/settings.updater.Set
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | /go/src/github.com/cockroachdb/cockroach/pkg/settings/updater.go:92
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | github.com/cockroachdb/cockroach/pkg/server.processSystemConfigKVs.func1
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | /go/src/github.com/cockroachdb/cockroach/pkg/server/settingsworker.go:47
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | github.com/cockroachdb/cockroach/pkg/server.processSystemConfigKVs
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | /go/src/github.com/cockroachdb/cockroach/pkg/server/settingsworker.go:53
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | github.com/cockroachdb/cockroach/pkg/server.(*Server).refreshSettings
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | /go/src/github.com/cockroachdb/cockroach/pkg/server/settingsworker.go:69
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1494
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | github.com/cockroachdb/cockroach/pkg/cli.runStart.func4.2
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | /go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:587
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | github.com/cockroachdb/cockroach/pkg/cli.runStart.func4
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | /go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:710
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | runtime.goexit
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 + | /usr/local/go/src/runtime/asm_amd64.s:1374
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +Wraps: (2) cannot set cluster.preserve_downgrade_option to ‹20.2› (cluster version is 21.1)
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +Error types: (1) *withstack.withStack (2) *errutil.leafError
W210801 11:35:17.095099 66 2@gossip/gossip.go:1491 ⋮ [n?] 2 no incoming or outgoing connections
CockroachDB node starting at 2021-08-01 11:35:22.216472524 +0000 UTC (took 5.4s)
build: CCL v21.1.6 @ 2021/07/20 15:30:39 (go1.15.11)
webui: https://0.0.0.0:8080
sql: postgresql://root@cockroachdb-0.cockroachdb.***.svc.cluster.local:26257?sslmode=verify-full&sslrootcert=%2Fcockroach%2Fcockroach-certs%2Fca.crt
RPC client flags: /cockroach/cockroach <client cmd> --host=cockroachdb-0.cockroachdb.***.svc.cluster.local:26257 --certs-dir=/cockroach/cockroach-certs
logs: /cockroach/cockroach-data/logs
temp dir: /cockroach/cockroach-data/cockroach-temp515401059
external I/O path: /cockroach/cockroach-data/extern
store[0]: path=/cockroach/cockroach-data
storage engine: pebble
status: restarted pre-existing node
clusterID: 88707280-6f4a-4158-8dda-0ffcf42d40ba
nodeID: 1
To Reproduce
We upgraded our cockroachdb clusters. When upgrading each cluster, we used the following steps:
SET CLUSTER SETTING cluster.preserve_downgrade_option = '20.2';executed prior to upgrade- Upgraded all nodes in cluster to v21.1
- Waited 24h+
RESET CLUSTER SETTING cluster.preserve_downgrade_option;executed
Expected behavior
The node seems to want to adjust the preserve_downgrade_option with no good reason.
Additional data / screenshots
Environment:
- CockroachDB version: v21.1.6
- Server OS: Linux
Additional context
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
A-configurabilityPertains to cluster settings, CLI flags, env vars etcPertains to cluster settings, CLI flags, env vars etcC-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.O-communityOriginated from the communityOriginated from the communityT-multitenantIssues owned by the multi-tenant virtual teamIssues owned by the multi-tenant virtual teamX-blathers-triagedblathers was able to find an ownerblathers was able to find an owner