roachtest: rewrite multitenant-upgrade to use the mixedversion framework#131847
Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom Oct 4, 2024
Merged
Conversation
Member
|
This change is |
This commit rewrites the `multitenant-upgrade` roachtest to use the `mixedversion` framework. Previously, this test would create a fixed set of tenants and run basic queries on them. With this change, a lot of aspects of the test are randomized: the number of tenants created, the amount of SQL server processes for each, the timing of tenant creation. We also run a `tpcc` workload on the tenants instead of the previous basic sanity checks. Fixes: cockroachdb#117673 Release note: None
1c3e509 to
e5dcb49
Compare
Author
|
Test passed 10 times on this branch (1 SSH flake). |
srosenberg
reviewed
Oct 4, 2024
| } | ||
| type createTenantOpt func(*createTenantOptions) | ||
|
|
||
| func createTenantNodeInternal( |
Author
|
@cockroachdb/disaster-recovery FYI, as owners of this test. It seems pretty stable, but letting you know in case you see a weird failure in the coming days. TFTR! bors r=srosenberg |
Contributor
rimadeodhar
added a commit
to rimadeodhar/cockroach
that referenced
this pull request
Jan 13, 2025
The multitenant upgrade test enforces different test scenarios while upgrading tenants in a mixed version state. The test enforces the following cases: 1. Start storage cluster with binary version: x, cluster version: x 2. Create some tenants with binary version: x and ensure they can connect to the cluster and run a workload. 3. Using the mixed version test framework, upgrade the storage cluster with binary version: x+1, cluster version: x. In this mixed version state, create remaining tenants with binary version: x and run a workload. 4. Finalize the storage cluster. At this point, the storage cluster has binary version: x+1 and cluster version: x+1 5. Upgrade tenants with binary version: x+1 and confirm tenants can connect to the storage cluster and run a workload. In cockroachdb#131847, the test was rewritten using the new mixed version test framework. However, this change exposed this test to a scenario that can cause this test to fail at step 3 above. The MVT framework also runs the mixed version test (i.e. with the tenant at the older binary version) when the cluster is in the finalizing stage. This scenario is run with a prefixed probability. However, if we attempt to start the tenants with the previous version (i.e. the version the cluster is being upgraded from) when the cluster is being finalized, the tenants rightfully fail to connect which the test incorrectly interprets as a failure. As a result, we would see this test fail occassionally since the test was updated to use the new MVT. This PR modifies the test to ensure that in the finalizing state, we start the tenants with the right version. Epic: none Fixes: cockroachdb#136447 Release note: None
craig bot
pushed a commit
that referenced
this pull request
Jan 13, 2025
138233: roachtest: Deflake multitenant upgrade test r=rimadeodhar a=rimadeodhar The multitenant upgrade test enforces different test scenarios while upgrading tenants in a mixed version state. The test enforces the following cases: 1. Start storage cluster with binary version: x, cluster version: x 2. Create some tenants with binary version: x and ensure they can connect to the cluster and run a workload. 3. Using the mixed version test framework, upgrade the storage cluster with binary version: x+1, cluster version: x. In this mixed version state, create remaining tenants with binary version: x and run a workload. 4. Finalize the storage cluster. At this point, the storage cluster has binary version: x+1 and cluster version: x+1 5. Upgrade tenants with binary version: x+1 and confirm tenants can connect to the storage cluster and run a workload. In #131847, the test was rewritten using the new mixed version test framework. However, this change exposed this test to a scenario that can cause this test to fail at step 3 above. The MVT framework also runs the mixed version test (i.e. with the tenant at the older binary version) when the cluster is in the finalizing stage. This scenario is run with a [prefixed probability](https://github.com/cockroachdb/cockroach/blob/b29d6f670b94492730eec233051cbe38a57b1ae9/pkg/cmd/roachtest/roachtestutil/mixedversion/mixedversion.go#L633). However, if we attempt to start the tenants with the previous version (i.e. the version the cluster is being upgraded from) when the cluster is being finalized, the tenants rightfully fail to connect which the test incorrectly interprets as a failure. As a result, we would see this test fail occassionally since the test was updated to use the new MVT. This PR modifies the test to ensure that in the finalizing state, we start the tenants with the right version. Epic: none Fixes: #136447 Release note: None 138977: roachtest: fix WaitForReady bugs r=kyle-a-wong a=kyle-a-wong WaitForReady was not properly handling when the deadline of a timeout context was reached while waiting for the cluster to be ready. Previously, hitting the timeout would result in the roachtest running for 10 minutes until the whole test timed out. Now, WaitForReady respects when the timeutil.RunWithTimeout context times out. Updated WaitForReady's checkReady func to use RoachtestHTTPClient's http client when making the call to GET /health?ready=1 instead of RoachtestHTTPClient's wrapper method. This wrapper method attempts to authenticate the user if no session id exists, but this endpoint doesn't require authentication so it doesnt need to use the wrapper method. Fixes: #138143 Resolves: #136128 Epic: None Release note: None Co-authored-by: rimadeodhar <rima@cockroachlabs.com> Co-authored-by: Kyle Wong <37189875+kyle-a-wong@users.noreply.github.com>
blathers-crl bot
pushed a commit
that referenced
this pull request
Jan 13, 2025
The multitenant upgrade test enforces different test scenarios while upgrading tenants in a mixed version state. The test enforces the following cases: 1. Start storage cluster with binary version: x, cluster version: x 2. Create some tenants with binary version: x and ensure they can connect to the cluster and run a workload. 3. Using the mixed version test framework, upgrade the storage cluster with binary version: x+1, cluster version: x. In this mixed version state, create remaining tenants with binary version: x and run a workload. 4. Finalize the storage cluster. At this point, the storage cluster has binary version: x+1 and cluster version: x+1 5. Upgrade tenants with binary version: x+1 and confirm tenants can connect to the storage cluster and run a workload. In #131847, the test was rewritten using the new mixed version test framework. However, this change exposed this test to a scenario that can cause this test to fail at step 3 above. The MVT framework also runs the mixed version test (i.e. with the tenant at the older binary version) when the cluster is in the finalizing stage. This scenario is run with a prefixed probability. However, if we attempt to start the tenants with the previous version (i.e. the version the cluster is being upgraded from) when the cluster is being finalized, the tenants rightfully fail to connect which the test incorrectly interprets as a failure. As a result, we would see this test fail occassionally since the test was updated to use the new MVT. This PR modifies the test to ensure that in the finalizing state, we start the tenants with the right version. Epic: none Fixes: #136447 Release note: None
mohini-crl
pushed a commit
to mohini-crl/cockroach
that referenced
this pull request
Jan 17, 2025
The multitenant upgrade test enforces different test scenarios while upgrading tenants in a mixed version state. The test enforces the following cases: 1. Start storage cluster with binary version: x, cluster version: x 2. Create some tenants with binary version: x and ensure they can connect to the cluster and run a workload. 3. Using the mixed version test framework, upgrade the storage cluster with binary version: x+1, cluster version: x. In this mixed version state, create remaining tenants with binary version: x and run a workload. 4. Finalize the storage cluster. At this point, the storage cluster has binary version: x+1 and cluster version: x+1 5. Upgrade tenants with binary version: x+1 and confirm tenants can connect to the storage cluster and run a workload. In cockroachdb#131847, the test was rewritten using the new mixed version test framework. However, this change exposed this test to a scenario that can cause this test to fail at step 3 above. The MVT framework also runs the mixed version test (i.e. with the tenant at the older binary version) when the cluster is in the finalizing stage. This scenario is run with a prefixed probability. However, if we attempt to start the tenants with the previous version (i.e. the version the cluster is being upgraded from) when the cluster is being finalized, the tenants rightfully fail to connect which the test incorrectly interprets as a failure. As a result, we would see this test fail occassionally since the test was updated to use the new MVT. This PR modifies the test to ensure that in the finalizing state, we start the tenants with the right version. Epic: none Fixes: cockroachdb#136447 Release note: None
rimadeodhar
added a commit
to rimadeodhar/cockroach
that referenced
this pull request
May 2, 2025
The multitenant upgrade test enforces different test scenarios while upgrading tenants in a mixed version state. The test enforces the following cases: 1. Start storage cluster with binary version: x, cluster version: x 2. Create some tenants with binary version: x and ensure they can connect to the cluster and run a workload. 3. Using the mixed version test framework, upgrade the storage cluster with binary version: x+1, cluster version: x. In this mixed version state, create remaining tenants with binary version: x and run a workload. 4. Finalize the storage cluster. At this point, the storage cluster has binary version: x+1 and cluster version: x+1 5. Upgrade tenants with binary version: x+1 and confirm tenants can connect to the storage cluster and run a workload. In cockroachdb#131847, the test was rewritten using the new mixed version test framework. However, this change exposed this test to a scenario that can cause this test to fail at step 3 above. The MVT framework also runs the mixed version test (i.e. with the tenant at the older binary version) when the cluster is in the finalizing stage. This scenario is run with a prefixed probability. However, if we attempt to start the tenants with the previous version (i.e. the version the cluster is being upgraded from) when the cluster is being finalized, the tenants rightfully fail to connect which the test incorrectly interprets as a failure. As a result, we would see this test fail occassionally since the test was updated to use the new MVT. This PR modifies the test to ensure that in the finalizing state, we start the tenants with the right version. Epic: none Fixes: cockroachdb#136447 Release note: None
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit rewrites the
multitenant-upgraderoachtest to use themixedversionframework. Previously, this test would create a fixed set of tenants and run basic queries on them.With this change, a lot of aspects of the test are randomized: the number of tenants created, the amount of SQL server processes for each, the timing of tenant creation. We also run a
tpccworkload on the tenants instead of the previous basic sanity checks.Fixes: #117673
Release note: None