fix(e2e): reproduction an fixing of missing evidence in e2e nightlies #3234
Merged
sergio-mena merged 7 commits intomainfrom Jun 11, 2024
Merged
fix(e2e): reproduction an fixing of missing evidence in e2e nightlies #3234sergio-mena merged 7 commits intomainfrom
e2e nightlies #3234sergio-mena merged 7 commits intomainfrom
Conversation
e2enightlies e2enightlies
e2enightlies e2e nightlies
8 tasks
hvanz
approved these changes
Jun 11, 2024
mergify bot
pushed a commit
that referenced
this pull request
Jun 11, 2024
…es (#3234) Closes #3233 This PR is is to be interpreted commit by commit. * Commit 1: Changes in `e2e runner to repro the problem (these changes will stay). * Commit 2. Making the problem repro appear in this PR (look at the ❌ next to commit 2 below, and check the `e2e` logs) * Commit 3: Fix. (look at the ✅ next to commit 3 below) * Commit 4: Revert commit 2 * Commit 5: Adapt nightlies to start running all regression manifests in (newly introduced) `test/e2e/networks_regressions` directory. Also, adapt manifest generator to create testnets with more evidences. * Further commits: fix e2e infra The reproduction consists in coming up with a testnet where some evidences can be delayed in their transit to the next validator, long enough to become invalid (too old). The delay is achieved by having a validator with skewed clock, in such a way that every time it proposes (untimely proposal) other peers shut down their p2p connection with that validator. The validator having the skewed clock is the one receiving all evidences. The root cause is that, in `e2e`, the max age of evidence is too restrictive (500ms in time, and 7 heights in terms blocks). So, if the gossiping of transactions is delayed, some of them are too old when they arrive to the first validator that can propose them. The fix consists in relaxing the max age of evidences in `e2e`: 1500ms, or 14 heights. Without the fix, the e2e run fails almost every time. With the fix, I haven't been able to repro it after multiple attempts. --- #### PR checklist - [x] Tests written/updated - ~~[ ] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog)~~ - ~~[ ] Updated relevant documentation (`docs/` or `spec/`) and code comments~~ - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec (cherry picked from commit bb6c702) # Conflicts: # .github/workflows/e2e-nightly-1x.yml
2 tasks
mergify bot
pushed a commit
that referenced
this pull request
Jun 11, 2024
…es (#3234) Closes #3233 This PR is is to be interpreted commit by commit. * Commit 1: Changes in `e2e runner to repro the problem (these changes will stay). * Commit 2. Making the problem repro appear in this PR (look at the ❌ next to commit 2 below, and check the `e2e` logs) * Commit 3: Fix. (look at the ✅ next to commit 3 below) * Commit 4: Revert commit 2 * Commit 5: Adapt nightlies to start running all regression manifests in (newly introduced) `test/e2e/networks_regressions` directory. Also, adapt manifest generator to create testnets with more evidences. * Further commits: fix e2e infra The reproduction consists in coming up with a testnet where some evidences can be delayed in their transit to the next validator, long enough to become invalid (too old). The delay is achieved by having a validator with skewed clock, in such a way that every time it proposes (untimely proposal) other peers shut down their p2p connection with that validator. The validator having the skewed clock is the one receiving all evidences. The root cause is that, in `e2e`, the max age of evidence is too restrictive (500ms in time, and 7 heights in terms blocks). So, if the gossiping of transactions is delayed, some of them are too old when they arrive to the first validator that can propose them. The fix consists in relaxing the max age of evidences in `e2e`: 1500ms, or 14 heights. Without the fix, the e2e run fails almost every time. With the fix, I haven't been able to repro it after multiple attempts. --- #### PR checklist - [x] Tests written/updated - ~~[ ] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog)~~ - ~~[ ] Updated relevant documentation (`docs/` or `spec/`) and code comments~~ - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec (cherry picked from commit bb6c702) # Conflicts: # .github/workflows/e2e-nightly-1x.yml # .github/workflows/e2e-nightly-38x.yml # test/e2e/generator/generate.go # test/e2e/networks/ci.toml
2 tasks
sergio-mena
added a commit
that referenced
this pull request
Jun 11, 2024
…es (#3234) Closes #3233 This PR is is to be interpreted commit by commit. * Commit 1: Changes in `e2e runner to repro the problem (these changes will stay). * Commit 2. Making the problem repro appear in this PR (look at the ❌ next to commit 2 below, and check the `e2e` logs) * Commit 3: Fix. (look at the ✅ next to commit 3 below) * Commit 4: Revert commit 2 * Commit 5: Adapt nightlies to start running all regression manifests in (newly introduced) `test/e2e/networks_regressions` directory. Also, adapt manifest generator to create testnets with more evidences. * Further commits: fix e2e infra The reproduction consists in coming up with a testnet where some evidences can be delayed in their transit to the next validator, long enough to become invalid (too old). The delay is achieved by having a validator with skewed clock, in such a way that every time it proposes (untimely proposal) other peers shut down their p2p connection with that validator. The validator having the skewed clock is the one receiving all evidences. The root cause is that, in `e2e`, the max age of evidence is too restrictive (500ms in time, and 7 heights in terms blocks). So, if the gossiping of transactions is delayed, some of them are too old when they arrive to the first validator that can propose them. The fix consists in relaxing the max age of evidences in `e2e`: 1500ms, or 14 heights. Without the fix, the e2e run fails almost every time. With the fix, I haven't been able to repro it after multiple attempts. --- - [x] Tests written/updated - ~~[ ] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog)~~ - ~~[ ] Updated relevant documentation (`docs/` or `spec/`) and code comments~~ - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec
sergio-mena
added a commit
that referenced
this pull request
Jun 11, 2024
…es (#3234) Closes #3233 This PR is is to be interpreted commit by commit. * Commit 1: Changes in `e2e runner to repro the problem (these changes will stay). * Commit 2. Making the problem repro appear in this PR (look at the ❌ next to commit 2 below, and check the `e2e` logs) * Commit 3: Fix. (look at the ✅ next to commit 3 below) * Commit 4: Revert commit 2 * Commit 5: Adapt nightlies to start running all regression manifests in (newly introduced) `test/e2e/networks_regressions` directory. Also, adapt manifest generator to create testnets with more evidences. * Further commits: fix e2e infra The reproduction consists in coming up with a testnet where some evidences can be delayed in their transit to the next validator, long enough to become invalid (too old). The delay is achieved by having a validator with skewed clock, in such a way that every time it proposes (untimely proposal) other peers shut down their p2p connection with that validator. The validator having the skewed clock is the one receiving all evidences. The root cause is that, in `e2e`, the max age of evidence is too restrictive (500ms in time, and 7 heights in terms blocks). So, if the gossiping of transactions is delayed, some of them are too old when they arrive to the first validator that can propose them. The fix consists in relaxing the max age of evidences in `e2e`: 1500ms, or 14 heights. Without the fix, the e2e run fails almost every time. With the fix, I haven't been able to repro it after multiple attempts. --- - [x] Tests written/updated - ~~[ ] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog)~~ - ~~[ ] Updated relevant documentation (`docs/` or `spec/`) and code comments~~ - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec
sergio-mena
added a commit
that referenced
this pull request
Jun 11, 2024
…es (backport #3234) (#3239) Closes #3233 This PR is is to be interpreted commit by commit. * Commit 1: Changes in `e2e runner to repro the problem (these changes will stay). * Commit 2. Making the problem repro appear in this PR (look at the ❌ next to commit 2 below, and check the `e2e` logs) * Commit 3: Fix. (look at the ✅ next to commit 3 below) * Commit 4: Revert commit 2 * Commit 5: Adapt nightlies to start running all regression manifests in (newly introduced) `test/e2e/networks_regressions` directory. Also, adapt manifest generator to create testnets with more evidences. * Further commits: fix e2e infra The reproduction consists in coming up with a testnet where some evidences can be delayed in their transit to the next validator, long enough to become invalid (too old). The delay is achieved by having a validator with skewed clock, in such a way that every time it proposes (untimely proposal) other peers shut down their p2p connection with that validator. The validator having the skewed clock is the one receiving all evidences. The root cause is that, in `e2e`, the max age of evidence is too restrictive (500ms in time, and 7 heights in terms blocks). So, if the gossiping of transactions is delayed, some of them are too old when they arrive to the first validator that can propose them. The fix consists in relaxing the max age of evidences in `e2e`: 1500ms, or 14 heights. Without the fix, the e2e run fails almost every time. With the fix, I haven't been able to repro it after multiple attempts. --- #### PR checklist - [x] Tests written/updated - ~~[ ] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog)~~ - ~~[ ] Updated relevant documentation (`docs/` or `spec/`) and code comments~~ - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec <hr>This is an automatic backport of pull request #3234 done by [Mergify](https://mergify.com). --------- Co-authored-by: Sergio Mena <sergio@informal.systems>
sergio-mena
added a commit
that referenced
this pull request
Jun 11, 2024
…es (backport #3234) (#3240) Closes #3233 This PR is is to be interpreted commit by commit. * Commit 1: Changes in `e2e runner to repro the problem (these changes will stay). * Commit 2. Making the problem repro appear in this PR (look at the ❌ next to commit 2 below, and check the `e2e` logs) * Commit 3: Fix. (look at the ✅ next to commit 3 below) * Commit 4: Revert commit 2 * Commit 5: Adapt nightlies to start running all regression manifests in (newly introduced) `test/e2e/networks_regressions` directory. Also, adapt manifest generator to create testnets with more evidences. * Further commits: fix e2e infra The reproduction consists in coming up with a testnet where some evidences can be delayed in their transit to the next validator, long enough to become invalid (too old). The delay is achieved by having a validator with skewed clock, in such a way that every time it proposes (untimely proposal) other peers shut down their p2p connection with that validator. The validator having the skewed clock is the one receiving all evidences. The root cause is that, in `e2e`, the max age of evidence is too restrictive (500ms in time, and 7 heights in terms blocks). So, if the gossiping of transactions is delayed, some of them are too old when they arrive to the first validator that can propose them. The fix consists in relaxing the max age of evidences in `e2e`: 1500ms, or 14 heights. Without the fix, the e2e run fails almost every time. With the fix, I haven't been able to repro it after multiple attempts. --- #### PR checklist - [x] Tests written/updated - ~~[ ] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog)~~ - ~~[ ] Updated relevant documentation (`docs/` or `spec/`) and code comments~~ - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec <hr>This is an automatic backport of pull request #3234 done by [Mergify](https://mergify.com). --------- Co-authored-by: Sergio Mena <sergio@informal.systems>
33 tasks
hvanz
pushed a commit
that referenced
this pull request
Jun 20, 2024
…es (backport #3234) (#3239) Closes #3233 This PR is is to be interpreted commit by commit. * Commit 1: Changes in `e2e runner to repro the problem (these changes will stay). * Commit 2. Making the problem repro appear in this PR (look at the ❌ next to commit 2 below, and check the `e2e` logs) * Commit 3: Fix. (look at the ✅ next to commit 3 below) * Commit 4: Revert commit 2 * Commit 5: Adapt nightlies to start running all regression manifests in (newly introduced) `test/e2e/networks_regressions` directory. Also, adapt manifest generator to create testnets with more evidences. * Further commits: fix e2e infra The reproduction consists in coming up with a testnet where some evidences can be delayed in their transit to the next validator, long enough to become invalid (too old). The delay is achieved by having a validator with skewed clock, in such a way that every time it proposes (untimely proposal) other peers shut down their p2p connection with that validator. The validator having the skewed clock is the one receiving all evidences. The root cause is that, in `e2e`, the max age of evidence is too restrictive (500ms in time, and 7 heights in terms blocks). So, if the gossiping of transactions is delayed, some of them are too old when they arrive to the first validator that can propose them. The fix consists in relaxing the max age of evidences in `e2e`: 1500ms, or 14 heights. Without the fix, the e2e run fails almost every time. With the fix, I haven't been able to repro it after multiple attempts. --- #### PR checklist - [x] Tests written/updated - ~~[ ] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog)~~ - ~~[ ] Updated relevant documentation (`docs/` or `spec/`) and code comments~~ - [x] Title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec <hr>This is an automatic backport of pull request #3234 done by [Mergify](https://mergify.com). --------- Co-authored-by: Sergio Mena <sergio@informal.systems>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3233
This PR is is to be interpreted commit by commit.
e2elogs)test/e2e/networks_regressionsdirectory. Also, adapt manifest generator to create testnets with more evidences.The reproduction consists in coming up with a testnet where some evidences can be delayed in their transit to the next validator, long enough to become invalid (too old). The delay is achieved by having a validator with skewed clock, in such a way that every time it proposes (untimely proposal) other peers shut down their p2p connection with that validator. The validator having the skewed clock is the one receiving all evidences.
The root cause is that, in
e2e, the max age of evidence is too restrictive (500ms in time, and 7 heights in terms blocks). So, if the gossiping of transactions is delayed, some of them are too old when they arrive to the first validator that can propose them.The fix consists in relaxing the max age of evidences in
e2e: 1500ms, or 14 heights.Without the fix, the e2e run fails almost every time. With the fix, I haven't been able to repro it after multiple attempts.
PR checklist
[ ] Changelog entry added in.changelog(we use unclog to manage our changelog)[ ] Updated relevant documentation (docs/orspec/) and code comments