roachtest: upgrade GCE to n2 and AWS to m6id and c6id#104419
Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom Jun 11, 2023
Merged
roachtest: upgrade GCE to n2 and AWS to m6id and c6id#104419craig[bot] merged 1 commit intocockroachdb:masterfrom
craig[bot] merged 1 commit intocockroachdb:masterfrom
Conversation
Member
|
This change is |
erikgrinaker
approved these changes
Jun 7, 2023
Contributor
erikgrinaker
left a comment
There was a problem hiding this comment.
Nice. Probably goes without saying that we'll want an annotation for this, as it's going to throw our benchmarks out of whack.
herkolategan
approved these changes
Jun 7, 2023
tbg
approved these changes
Jun 7, 2023
Member
tbg
left a comment
There was a problem hiding this comment.
Thanks! Also worth doing a canary run of 1-2 roachtests for both GCE and AWS, disregard if you've done it already.
Member
Author
Absolutely! Canary runs are queued up; won't merge until we have the signal. AWS: https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/10440661?buildTab=tests |
Member
Author
|
AWS run succeeded, but GCE run had a couple of issues, |
0ee6b20 to
ac232a7
Compare
Member
Author
|
After further tweaks, it looks like everything passed,
In the GCE run, there were two cluster creation errors, one of which is a known issue (resolved by [1]), the other one is a small quota in |
Previously, roachtest used n1 in GCE, m5d and c6d in AWS. CockroachDB Cloud hardware now uses n2 in GCE, m6i in AWS [1]. This change brings roachtest hardware into parity with CockroachDB Cloud; it is based on the draft PR in [2]. [1] https://cockroachlabs.atlassian.net/wiki/spaces/MC/pages/2799501550/CockroachDB+Cloud+Hardware [2] cockroachdb#99991 Epic: none Release note: None Co-authored-by: Nick Travers <travers@cockroachlabs.com>
ac232a7 to
5ad20cc
Compare
Member
Author
|
Quota in |
Member
Author
|
Both runs succeeded, modulo the know issue with
N.B. variance between runs is much higher on master. Further study is queued up once we have more data points for this change. TL;DR: shaving ~3 hours from a GCE run is a quick win, considering we were approaching 24h for some GCE runs. |
Member
Author
|
TFTR! bors r=erikgrinaker,tbg,herkolategan |
Contributor
|
Build succeeded: |
|
Encountered an error creating backports. Some common things that can go wrong:
You might need to create your backport manually using the backport tool. error creating merge commit from 5ad20cc to blathers/backport-release-22.2-104419: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict [] you may need to manually resolve merge conflicts with the backport tool. Backport to branch 22.2.x failed. See errors above. error creating merge commit from 5ad20cc to blathers/backport-release-23.1-104419: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict [] you may need to manually resolve merge conflicts with the backport tool. Backport to branch 23.1.x failed. See errors above. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
srosenberg
added a commit
to srosenberg/cockroach
that referenced
this pull request
Jun 20, 2023
Since the bump to new instance types in GCE and AWS [1], we are still experiencing occasional cluster creation issues owing to "insufficient capacity". GCE quota has already been bumped, with `asia-northeast1` being the latest, and hopefully last. The most recent cluster creation in AWS is owing to "insufficient capacity" of `c6id.24xlarge` in us-east-2a. As a workaround, we extend the existing zone override to place `c6id.24xlarge` into us-east-2b, which allegedly has sufficient capacity. Note, the long-term fix is to rework how cluster creation retry currently operates, by effectively trying other AZs. [1] cockroachdb#104419 Epic: none Fixes: cockroachdb#78601 (comment) Release note: None
craig bot
pushed a commit
that referenced
this pull request
Jun 21, 2023
105234: roachprod: add aws AZ override for c6id.24xlarge r=renatolabs a=srosenberg Since the bump to new instance types in GCE and AWS [1], we are still experiencing occasional cluster creation issues owing to "insufficient capacity". GCE quota has already been bumped, with `asia-northeast1` being the latest, and hopefully last. The most recent cluster creation in AWS is owing to "insufficient capacity" of `c6id.24xlarge` in us-east-2a. As a workaround, we extend the existing zone override to place `c6id.24xlarge` into us-east-2b, which allegedly has sufficient capacity. Note, the long-term fix is to rework how cluster creation retry currently operates, by effectively trying other AZs. [1] #104419 Epic: none Fixes: #78601 (comment) Release note: None Co-authored-by: Stan Rosenberg <stan.rosenberg@gmail.com>
blathers-crl bot
pushed a commit
that referenced
this pull request
Jun 21, 2023
Since the bump to new instance types in GCE and AWS [1], we are still experiencing occasional cluster creation issues owing to "insufficient capacity". GCE quota has already been bumped, with `asia-northeast1` being the latest, and hopefully last. The most recent cluster creation in AWS is owing to "insufficient capacity" of `c6id.24xlarge` in us-east-2a. As a workaround, we extend the existing zone override to place `c6id.24xlarge` into us-east-2b, which allegedly has sufficient capacity. Note, the long-term fix is to rework how cluster creation retry currently operates, by effectively trying other AZs. [1] #104419 Epic: none Fixes: #78601 (comment) Release note: None
srosenberg
added a commit
to srosenberg/cockroach
that referenced
this pull request
Jun 22, 2023
Since the bump to new instance types in GCE and AWS [1], we are still experiencing occasional cluster creation issues owing to "insufficient capacity". GCE quota has already been bumped, with `asia-northeast1` being the latest, and hopefully last. The most recent cluster creation in AWS is owing to "insufficient capacity" of `c6id.24xlarge` in us-east-2a. As a workaround, we extend the existing zone override to place `c6id.24xlarge` into us-east-2b, which allegedly has sufficient capacity. Note, the long-term fix is to rework how cluster creation retry currently operates, by effectively trying other AZs. [1] cockroachdb#104419 Epic: none Fixes: cockroachdb#78601 (comment) Release note: None
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously, roachtest used n1 in GCE, m5d and c6d in AWS. CockroachDB Cloud hardware now uses n2 in GCE, m6i in AWS [1]. This change brings roachtest hardware into parity with CockroachDB Cloud; it is based on the draft PR in [2].
[1] https://cockroachlabs.atlassian.net/wiki/spaces/MC/pages/2799501550/CockroachDB+Cloud+Hardware
[2] #99991
Epic: none
Release note: None