Skip to content

roachtest: add mixed version testing for the schema changer corpuses#87922

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
fqazi:roachTestForMixed
Sep 28, 2022
Merged

roachtest: add mixed version testing for the schema changer corpuses#87922
craig[bot] merged 1 commit intocockroachdb:masterfrom
fqazi:roachTestForMixed

Conversation

@fqazi
Copy link
Copy Markdown
Collaborator

@fqazi fqazi commented Sep 13, 2022

Fixes: #82643

Previously, we had no automated tests for validating mixed version compatibility. This new roachtest validates elements generated on the previous release against the current. It validates elements generated in a mixed version state on the current release against previous version. It also sanity checks that any existing corpus stays compatible with the current version. The corpuses uses are automatically generated on nightly builds and stored on google cloud storage.

Release justification: low risk and enhances backwards / forwards compatibility test coverage for declarative schema changer.
Release note: None

@fqazi fqazi requested a review from a team September 13, 2022 18:50
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@fqazi fqazi force-pushed the roachTestForMixed branch 2 times, most recently from 824ce6c to 9004014 Compare September 15, 2022 01:42
@fqazi fqazi force-pushed the roachTestForMixed branch 2 times, most recently from 89ca166 to fbd0379 Compare September 21, 2022 17:19
@fqazi fqazi marked this pull request as ready for review September 21, 2022 17:21
@fqazi
Copy link
Copy Markdown
Collaborator Author

fqazi commented Sep 21, 2022

RFAL - Infrastructure is in place for this now

Copy link
Copy Markdown
Contributor

@ajwerner ajwerner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just nits. Sorry for the late review.

Reviewed 3 of 3 files at r1, all commit messages.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @fqazi)


pkg/cmd/roachtest/tests/mixed_version_decl_schemachange_compat.go line 42 at r1 (raw file):

	ctx context.Context, logger *logger.Logger, c cluster.Cluster, versionNumber string,
) string {
	const tmpPath = "/tmp/"

this should use os.MkdirTemp


pkg/cmd/roachtest/tests/mixed_version_decl_schemachange_compat.go line 48 at r1 (raw file):

			tmpPath))
	if err != nil {
		logger.Printf("Missing validation corpus for %v", versionNumber)

delete the tempdir if there's an error? Or rather, should we propagate the error up and fail the test?


pkg/cmd/roachtest/tests/mixed_version_decl_schemachange_compat.go line 68 at r1 (raw file):

		t.Fatal(err)
	}
	failureRegex := regexp.MustCompile(`failed to validate.*`)

should we have the debug command exit with a non-zero exit code if the validation fails? Maybe it's a bit late for that.

@fqazi fqazi requested a review from a team as a code owner September 27, 2022 20:19
@fqazi fqazi force-pushed the roachTestForMixed branch 4 times, most recently from d3d3d12 to 143cbd9 Compare September 27, 2022 21:02
Copy link
Copy Markdown
Collaborator Author

@fqazi fqazi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner)


pkg/cmd/roachtest/tests/mixed_version_decl_schemachange_compat.go line 42 at r1 (raw file):

Previously, ajwerner wrote…

this should use os.MkdirTemp

Done.


pkg/cmd/roachtest/tests/mixed_version_decl_schemachange_compat.go line 48 at r1 (raw file):

Previously, ajwerner wrote…

delete the tempdir if there's an error? Or rather, should we propagate the error up and fail the test?

Done.

Moved it so that the test passed into these functions for a direct failure.


pkg/cmd/roachtest/tests/mixed_version_decl_schemachange_compat.go line 68 at r1 (raw file):

Previously, ajwerner wrote…

should we have the debug command exit with a non-zero exit code if the validation fails? Maybe it's a bit late for that.

Done.

Modified the command to do this, and added support in the test detect both ways of failing. In the future we can drop the first method of checking.

@fqazi fqazi requested a review from ajwerner September 27, 2022 21:07
Copy link
Copy Markdown
Contributor

@ajwerner ajwerner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 2 of 2 files at r2, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @fqazi)


pkg/cmd/roachtest/tests/mixed_version_decl_schemachange_compat.go line 114 at r2 (raw file):

		t.Fatal(err)
	}
	// Validate the mixed version of the corpus, against the previous

nit: this could be refactored to be more table-driven, the core of the logic seems duplicated

Code quote:

	// Validate the mixed version of the corpus, against the previous
	// version.
	binaryName := uploadVersion(ctx, t, c, c.All(), predecessorVersion)
	mixedVersionCorpus := fmt.Sprintf("%d.%d-mixed", buildVersion.Major(), buildVersion.Minor())
	corpusPath, cleanupFn := fetchCorpusToTmpDir(ctx, t, c, mixedVersionCorpus)
	if err != nil {
		t.Fatal("failed to locate backwards compatible corpus (%v)", err)
	}
	validateCorpusFile(ctx, t, c, binaryName, corpusPath)
	cleanupFn()
	// Forward compatibility testing of elements generated in the previous
	// version.
	binaryName = uploadVersion(ctx, t, c, c.All(), "")
	versionRegex := regexp.MustCompile(`(\d+\.\d+)`)
	versionNumber := versionRegex.FindStringSubmatch(predecessorVersion)[0]
	corpusPath, cleanupFn = fetchCorpusToTmpDir(ctx, t, c, versionNumber)
	if err != nil {
		t.Fatal("failed to locate backwards compatible corpus (%v)", err)
	}
	validateCorpusFile(ctx, t, c, binaryName, corpusPath)
	cleanupFn()
	// Validate the corpus generated on the same version as well.
	versionNumber = versionRegex.FindStringSubmatch(buildVersion.String())[0]
	corpusPath, cleanupFn = fetchCorpusToTmpDir(ctx, t, c, versionNumber)
	if err != nil {
		t.Fatal("failed to locate same version corpus (%v)", err)
	}
	validateCorpusFile(ctx, t, c, binaryName, corpusPath)
	cleanupFn()

Fixes: cockroachdb#82643

Previously, we had no automated tests for validating
mixed version compatibility. This new roachtest validates
elements generated on the previous release against the
current. It validates elements generated in a mixed
version state on the current release against previous
version. It also sanity checks that any existing corpus
stays compatible with the current version.

Release justification: low risk and enhances backwards / forwards
compatibility test coverage for declarative schema changer.
Release note: None
Copy link
Copy Markdown
Collaborator Author

@fqazi fqazi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ajwerner TFTR!

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @ajwerner)

@fqazi
Copy link
Copy Markdown
Collaborator Author

fqazi commented Sep 28, 2022

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Sep 28, 2022

Build succeeded:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sql/schemachanger: mixed version compatibility testing

3 participants