roachtest: add mixed version testing for the schema changer corpuses#87922
roachtest: add mixed version testing for the schema changer corpuses#87922craig[bot] merged 1 commit intocockroachdb:masterfrom
Conversation
824ce6c to
9004014
Compare
89ca166 to
fbd0379
Compare
|
RFAL - Infrastructure is in place for this now |
aa589a9 to
8c5b7e9
Compare
8c5b7e9 to
0892a29
Compare
ajwerner
left a comment
There was a problem hiding this comment.
Just nits. Sorry for the late review.
Reviewed 3 of 3 files at r1, all commit messages.
Reviewable status:complete! 0 of 0 LGTMs obtained (waiting on @fqazi)
pkg/cmd/roachtest/tests/mixed_version_decl_schemachange_compat.go line 42 at r1 (raw file):
ctx context.Context, logger *logger.Logger, c cluster.Cluster, versionNumber string, ) string { const tmpPath = "/tmp/"
this should use os.MkdirTemp
pkg/cmd/roachtest/tests/mixed_version_decl_schemachange_compat.go line 48 at r1 (raw file):
tmpPath)) if err != nil { logger.Printf("Missing validation corpus for %v", versionNumber)
delete the tempdir if there's an error? Or rather, should we propagate the error up and fail the test?
pkg/cmd/roachtest/tests/mixed_version_decl_schemachange_compat.go line 68 at r1 (raw file):
t.Fatal(err) } failureRegex := regexp.MustCompile(`failed to validate.*`)
should we have the debug command exit with a non-zero exit code if the validation fails? Maybe it's a bit late for that.
0892a29 to
fd097dc
Compare
d3d3d12 to
143cbd9
Compare
fqazi
left a comment
There was a problem hiding this comment.
Reviewable status:
complete! 0 of 0 LGTMs obtained (waiting on @ajwerner)
pkg/cmd/roachtest/tests/mixed_version_decl_schemachange_compat.go line 42 at r1 (raw file):
Previously, ajwerner wrote…
this should use
os.MkdirTemp
Done.
pkg/cmd/roachtest/tests/mixed_version_decl_schemachange_compat.go line 48 at r1 (raw file):
Previously, ajwerner wrote…
delete the tempdir if there's an error? Or rather, should we propagate the error up and fail the test?
Done.
Moved it so that the test passed into these functions for a direct failure.
pkg/cmd/roachtest/tests/mixed_version_decl_schemachange_compat.go line 68 at r1 (raw file):
Previously, ajwerner wrote…
should we have the debug command exit with a non-zero exit code if the validation fails? Maybe it's a bit late for that.
Done.
Modified the command to do this, and added support in the test detect both ways of failing. In the future we can drop the first method of checking.
ajwerner
left a comment
There was a problem hiding this comment.
Reviewed 2 of 2 files at r2, all commit messages.
Reviewable status:complete! 1 of 0 LGTMs obtained (waiting on @fqazi)
pkg/cmd/roachtest/tests/mixed_version_decl_schemachange_compat.go line 114 at r2 (raw file):
t.Fatal(err) } // Validate the mixed version of the corpus, against the previous
nit: this could be refactored to be more table-driven, the core of the logic seems duplicated
Code quote:
// Validate the mixed version of the corpus, against the previous
// version.
binaryName := uploadVersion(ctx, t, c, c.All(), predecessorVersion)
mixedVersionCorpus := fmt.Sprintf("%d.%d-mixed", buildVersion.Major(), buildVersion.Minor())
corpusPath, cleanupFn := fetchCorpusToTmpDir(ctx, t, c, mixedVersionCorpus)
if err != nil {
t.Fatal("failed to locate backwards compatible corpus (%v)", err)
}
validateCorpusFile(ctx, t, c, binaryName, corpusPath)
cleanupFn()
// Forward compatibility testing of elements generated in the previous
// version.
binaryName = uploadVersion(ctx, t, c, c.All(), "")
versionRegex := regexp.MustCompile(`(\d+\.\d+)`)
versionNumber := versionRegex.FindStringSubmatch(predecessorVersion)[0]
corpusPath, cleanupFn = fetchCorpusToTmpDir(ctx, t, c, versionNumber)
if err != nil {
t.Fatal("failed to locate backwards compatible corpus (%v)", err)
}
validateCorpusFile(ctx, t, c, binaryName, corpusPath)
cleanupFn()
// Validate the corpus generated on the same version as well.
versionNumber = versionRegex.FindStringSubmatch(buildVersion.String())[0]
corpusPath, cleanupFn = fetchCorpusToTmpDir(ctx, t, c, versionNumber)
if err != nil {
t.Fatal("failed to locate same version corpus (%v)", err)
}
validateCorpusFile(ctx, t, c, binaryName, corpusPath)
cleanupFn()Fixes: cockroachdb#82643 Previously, we had no automated tests for validating mixed version compatibility. This new roachtest validates elements generated on the previous release against the current. It validates elements generated in a mixed version state on the current release against previous version. It also sanity checks that any existing corpus stays compatible with the current version. Release justification: low risk and enhances backwards / forwards compatibility test coverage for declarative schema changer. Release note: None
143cbd9 to
9aa68b5
Compare
fqazi
left a comment
There was a problem hiding this comment.
@ajwerner TFTR!
Reviewable status:
complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @ajwerner)
|
bors r+ |
|
Build succeeded: |
Fixes: #82643
Previously, we had no automated tests for validating mixed version compatibility. This new roachtest validates elements generated on the previous release against the current. It validates elements generated in a mixed version state on the current release against previous version. It also sanity checks that any existing corpus stays compatible with the current version. The corpuses uses are automatically generated on nightly builds and stored on google cloud storage.
Release justification: low risk and enhances backwards / forwards compatibility test coverage for declarative schema changer.
Release note: None