roachtest: limit FingerprintValidator's memory usage internally#89782
Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom Oct 13, 2022
Merged
Conversation
Member
|
This change is |
Author
|
A potential way to deal with flakes like #87574 (comment). |
miretskiy
approved these changes
Oct 11, 2022
In cockroachdb#89332, we started optionally validating changefeed semantics in the `FingerprintValidator` by making sure that unseen updates are never observed if a `resolved` message is received. In order to keep bounds on the validator's memory usage, the `cdc/mixed-versions` test was setting a maximum number of operations at the workload level. However, a consequence of that change is that it creates the possibility for the workload to finish before the roachtest has received all the `resolved` events it expects (for instance, if draining takes longer than usual for some reason; a variety of other non-determinism is also at play). To deal with this possibility, we instead enforce a maximum number of previously seen events at the validator's level. For the `cdc/mixed-versions` test, we set a maximum of 100,000 previously seen events being stored (20MB memory footprint). This should be equivalent to roughly 50,000 bank transfers, and give enough time for the test to finish successfully. Epic: None. Release note: None.
d2d8edc to
ac7758d
Compare
srosenberg
approved these changes
Oct 12, 2022
Member
srosenberg
left a comment
There was a problem hiding this comment.
Reviewable status:
complete! 1 of 0 LGTMs obtained (waiting on @smg260)
Author
|
bors r=miretskiy,srosenberg TFTRs! |
Contributor
|
Build failed (retrying...): |
Contributor
|
Build succeeded: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In #89332, we started optionally validating changefeed semantics in the
FingerprintValidatorby making sure that unseen updates are never observed if aresolvedmessage is received. In order to keep bounds on the validator's memory usage, thecdc/mixed-versionstest was setting a maximum number of operations at the workload level. However, a consequence of that change is that it creates the possibility for the workload to finish before the roachtest has received all theresolvedevents it expects (for instance, if draining takes longer than usual for some reason; a variety of other non-determinism is also at play).To deal with this possibility, we instead enforce a maximum number of previously seen events at the validator's level. For the
cdc/mixed-versionstest, we set a maximum of 100,000 previously seen events being stored (20MB memory footprint). This should be equivalent to roughly 50,000 bank transfers, and give enough time for the test to finish successfully.Release note: None
Epic: None.