Skip to content

setting ‹"cluster.preserve_downgrade_option"› to ‹"20.2"› failed: cannot set cluster.preserve_downgrade_option to ‹20.2› (cluster version is 21.1) #68335

@nick-jones

Description

@nick-jones

Describe the problem

After upgrading our clusters from 20.2 to 21.1, all nodes now appear to want to adjust cluster.preserve_downgrade_option when starting.

+ exec /cockroach/cockroach start --logtostderr=WARNING --certs-dir /cockroach/cockroach-certs --advertise-addr cockroachdb-0.cockroachdb.***.svc.cluster.local --http-addr 0.0.0.0 --joi
n cockroachdb-0.cockroachdb,cockroachdb-1.cockroachdb,cockroachdb-2.cockroachdb --cache 25% --max-sql-memory 25%
Flag --logtostderr has been deprecated, use --log instead to specify 'sinks: {stderr: {filter: ...}}'.
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1  setting ‹"cluster.preserve_downgrade_option"› to ‹"20.2"› failed: cannot set cluster.preserve_downgrade_option to ‹20.2› (cluster version
 is 21.1)
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +(1) attached stack trace
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  -- stack trace:
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  | github.com/cockroachdb/cockroach/pkg/clusterversion.registerPreserveDowngradeVersionSetting.func1
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  |    /go/src/github.com/cockroachdb/cockroach/pkg/clusterversion/setting.go:259
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  | github.com/cockroachdb/cockroach/pkg/settings.(*StringSetting).Validate
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  |    /go/src/github.com/cockroachdb/cockroach/pkg/settings/string.go:65
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  | github.com/cockroachdb/cockroach/pkg/settings.(*StringSetting).set
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  |    /go/src/github.com/cockroachdb/cockroach/pkg/settings/string.go:79
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  | github.com/cockroachdb/cockroach/pkg/settings.updater.Set
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  |    /go/src/github.com/cockroachdb/cockroach/pkg/settings/updater.go:92
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  | github.com/cockroachdb/cockroach/pkg/server.processSystemConfigKVs.func1
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  |    /go/src/github.com/cockroachdb/cockroach/pkg/server/settingsworker.go:47
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  | github.com/cockroachdb/cockroach/pkg/server.processSystemConfigKVs
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  |    /go/src/github.com/cockroachdb/cockroach/pkg/server/settingsworker.go:53
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  | github.com/cockroachdb/cockroach/pkg/server.(*Server).refreshSettings
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  |    /go/src/github.com/cockroachdb/cockroach/pkg/server/settingsworker.go:69
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  | github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  |    /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1494
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  | github.com/cockroachdb/cockroach/pkg/cli.runStart.func4.2
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  |    /go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:587
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  | github.com/cockroachdb/cockroach/pkg/cli.runStart.func4
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  |    /go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:710
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  | runtime.goexit
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +  |    /usr/local/go/src/runtime/asm_amd64.s:1374
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +Wraps: (2) cannot set cluster.preserve_downgrade_option to ‹20.2› (cluster version is 21.1)
W210801 11:35:17.090732 66 server/settingsworker.go:48 ⋮ [n?] 1 +Error types: (1) *withstack.withStack (2) *errutil.leafError
W210801 11:35:17.095099 66 2@gossip/gossip.go:1491 ⋮ [n?] 2  no incoming or outgoing connections
CockroachDB node starting at 2021-08-01 11:35:22.216472524 +0000 UTC (took 5.4s)
build:               CCL v21.1.6 @ 2021/07/20 15:30:39 (go1.15.11)
webui:               https://0.0.0.0:8080
sql:                 postgresql://root@cockroachdb-0.cockroachdb.***.svc.cluster.local:26257?sslmode=verify-full&sslrootcert=%2Fcockroach%2Fcockroach-certs%2Fca.crt
RPC client flags:    /cockroach/cockroach <client cmd> --host=cockroachdb-0.cockroachdb.***.svc.cluster.local:26257 --certs-dir=/cockroach/cockroach-certs
logs:                /cockroach/cockroach-data/logs
temp dir:            /cockroach/cockroach-data/cockroach-temp515401059
external I/O path:   /cockroach/cockroach-data/extern
store[0]:            path=/cockroach/cockroach-data
storage engine:      pebble
status:              restarted pre-existing node
clusterID:           88707280-6f4a-4158-8dda-0ffcf42d40ba
nodeID:              1

To Reproduce

We upgraded our cockroachdb clusters. When upgrading each cluster, we used the following steps:

  • SET CLUSTER SETTING cluster.preserve_downgrade_option = '20.2'; executed prior to upgrade
  • Upgraded all nodes in cluster to v21.1
  • Waited 24h+
  • RESET CLUSTER SETTING cluster.preserve_downgrade_option; executed

Expected behavior

The node seems to want to adjust the preserve_downgrade_option with no good reason.

Additional data / screenshots

Environment:

  • CockroachDB version: v21.1.6
  • Server OS: Linux

Additional context

Jira issue: CRDB-8992
Epic: CRDB-6671

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-configurabilityPertains to cluster settings, CLI flags, env vars etcC-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.O-communityOriginated from the communityT-multitenantIssues owned by the multi-tenant virtual teamX-blathers-triagedblathers was able to find an owner

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions