Skip to content

release: audit unit/integration upgrade tests #100552

@srosenberg

Description

@srosenberg

All unit tests which invoke MakeTestingClusterSettingsWithVersions and pass a non-default binaryMinSupportedVersion (i.e., anything other than clusterversion.TestingBinaryMinSupportedVersion) should be audited. The reason is explained in detail in [1].

TL;DR Before [1], the test cluster would be bootstrapped at the current version unless the test passed BootstrapVersionKeyOverride. Thus, migrations were not being tested other than their idempotency. After [1], the test cluster is correctly bootstrapped at the default binaryMinSupportedVersion unless the test overrides it. This roughly translates to,

NOTE: the above only concerns the unit tests. Roachtests use a different bootstrapping mechanism which ensures an upgrade test is always bootstrapped at some previous version.

  • Node is bootstrapped and joins the test cluster at the default binaryMinSupportedVersion
  • StartServer uses s.BinaryVersionOverride() to SET CLUSTER SETTING version which triggers migrations from the bootstrapped version to the (overridden) BinaryVersion

Below are all unit tests, grouped by package, which should be audited. This set is conservative. The ones which say FAIL pass a non-default binaryMinSupportedVersion, but may still be correct. The ones which don't say FAIL are likely to be correct, but it wouldn't hurt to double-check them.

pkg/ccl/kvccl/kvtenantccl

@cockroachdb/multi-tenant

TestTenantUpgradeFailure
TestTenantUpgradeInterlock

pkg/ccl/serverccl

@cockroachdb/multi-tenant

TestServerStartupGuardrails (FAIL)
TestBumpTenantClusterVersion (FAIL)
TestValidateTargetTenantClusterVersion (FAIL)

pkg/kv/kvclient/kvcoord

TestBiDirectionalRangefeedNotUsedUntilUpgradeFinalilzed (FAIL)

pkg/kv/kvserver

TestMigrateWaitsForApplication (FAIL)
TestLeaseUpgradeVersionGate (FAIL)
TestRangeMigration (FAIL)
TestLoadBasedRebalancingObjective
TestRebalanceObjectiveManager
TestStoreConfig

pkg/kv/kvserver/batcheval

TestDeclareKeysResolveIntent

pkg/server/settingswatcher

TestVersionGuard

pkg/storage

TestPebbleIterator_ExternalCorruption #100592

pkg/upgrade/upgrademanager

@cockroachdb/sql-schema and @cockroachdb/jobs

TestAlreadyRunningJobsAreHandledProperly - #100830
TestConcurrentMigrationAttempts - #100873
TestMigrateUpdatesReplicaVersion - #100873
TestPauseMigration - #100830

pkg/upgrade/upgrades

@cockroachdb/sql-schema

TestDatabaseRoleSettingsUserIDMigration1500Users (FAIL) seems like a mistake that this is in this list
TestDeleteDescriptorsOfDroppedFunctions (FAIL) #100726
TestExternalConnectionsUserIDMigration10Users (FAIL) seems like a mistake that this is in this list
TestFixUserfileRelatedDescriptorCorruptionUpgrade (FAIL) #100726
TestPreconditionBeforeStartingAnUpgrade (FAIL) #100726
TestMigrationWithFailuresMultipleAltersOnSameColumn fixed by #100644
TestSystemJobInfoMigration #100726 #100696
TestSystemPrivilegesIndexMigration seems like a mistake that this is in this list
TestSystemActivityMigration #100726
TestWaitForDelRangeInGCJob (FAIL) #100726
TestWebSessionsUserIDMigrationNoUsers seems like a mistake that this is in this list

pkg/ccl/backupccl/datadriven_test.go

@cockroachdb/disaster-recovery

testdata/backup-restore/restore-schema-only-mixed-version:new-cluster name=s1 beforeVersion=Start22_2 disable-tenant #100696
testdata/backup-restore/in-progress-import-rollback:new-cluster name=s1 beforeVersion=23_1_MVCCTombstones disable-tenant #100696
testdata/backup-restore/restore-mixed-version:new-cluster name=s1 beforeVersion=Start22_2 disable-tenant #100696
testdata/backup-restore/restore-mixed-version:new-cluster name=s2 beforeVersion=Start22_2 share-io-dir=s1 disable-tenant #100696

[1] #99082

Jira issue: CRDB-26487

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)GA-blockerbranch-masterFailures and bugs on the master branch.branch-release-23.1Used to mark GA and release blockers, technical advisories, and bugs for 23.1

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions