experimental mempool: Backport #1558 and #1584 to 0.34.x#1630
Merged
Conversation
Backports #1558 and #1584 to 0.38.x (#1592) * mempool: Limit gossip connections to persistent and non-persistent peers (experimental) (#1584) * Experimental - Reduce # of connections effectively used to gossip transactions out (#1558) * maxpeers for mempool * mempool: fix max_peers bcast routine active flag * Use semaphore to limit concurrency * Rename MaxPeers to MaxOutboundPeers * Add max_outbound_peers to config toml template * Rename in error message * Renams the parameter to highlight its experimental nature. Extend the AddPeer method to return an error. Moves the semaphone to outside the broadcast routine * reverting the addition of error to AddPeer. It fails if the context is done and handling this case will be done some other time, when an actual context is passed into acquire. * reverting the addition of error to AddPeer. It fails if the context is done and handling this case will be done some other time, when an actual context is passed into acquire. * Fixing lint issue * renaming semaphore to something more meaningful * make default value 0, which is the same as the current behavior. 10 is the recommended value. * adding new flag to manifest.go * Adding changelog * Improve the description of the parameter in the generated config file. * Add metric to track the current number of active connections. * Change metric to gauge type and rename it. * e2e: Allow disabling the PEX reactor on all nodes in the testnet * Apply suggestions from code review * Update config/config.go comment * fix lint error * Improve config description * Rename metric (remove experimental prefix) * Add unit test * Improve unit test * Update mempool/reactor.go comment --------- * Updating test file, leaving it broken for now * mempool: Limit gossip connections to persistent and non-persistent peers (experimental) (#1584) * Ignore persistent peers from limiting of outbound connections * Update 1558-experimental-gossip-limiting.md Update changeling * Fix typo in mempool/metrics.go * Use two independent configs and semaphores for persistent and non-persistent peers * Forgot to rename in test * Update metric description * Rename semaphores * Add comment to unit test --------- * Reverting to old way of reporting errors * Reverting change that shouldn't have been included in cherry-pick * Reverting tests to use older functions * fix rebase merge --------- Co-authored-by: Adi Seredinschi <adizere@gmail.com> Co-authored-by: Ethan Buchman <ethan@coinculture.info> Co-authored-by: Daniel Cason <daniel.cason@informal.systems> Co-authored-by: hvanz <hernan.vanzetto@gmail.com> Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Sergio Mena <sergio@informal.systems>
Contributor
|
Pls also cherry-pick #1629 |
melekes
approved these changes
Nov 16, 2023
…s the order of transactions in the receiving reactor. (#1629)
thanethomson
approved these changes
Nov 16, 2023
.changelog/unreleased/improvements/1558-experimental-gossip-limiting.md
Outdated
Show resolved
Hide resolved
…miting.md Co-authored-by: Thane Thomson <connect@thanethomson.com>
Galadrin
pushed a commit
to Galadrin/cometbft
that referenced
this pull request
Dec 2, 2023
…4.x (cometbft#1630) * Backports cometbft#1558 and cometbft#1584 to 0.37.x (cometbft#1611) Backports cometbft#1558 and cometbft#1584 to 0.38.x (cometbft#1592) * mempool: Limit gossip connections to persistent and non-persistent peers (experimental) (cometbft#1584) * Experimental - Reduce # of connections effectively used to gossip transactions out (cometbft#1558) * maxpeers for mempool * mempool: fix max_peers bcast routine active flag * Use semaphore to limit concurrency * Rename MaxPeers to MaxOutboundPeers * Add max_outbound_peers to config toml template * Rename in error message * Renams the parameter to highlight its experimental nature. Extend the AddPeer method to return an error. Moves the semaphone to outside the broadcast routine * reverting the addition of error to AddPeer. It fails if the context is done and handling this case will be done some other time, when an actual context is passed into acquire. * reverting the addition of error to AddPeer. It fails if the context is done and handling this case will be done some other time, when an actual context is passed into acquire. * Fixing lint issue * renaming semaphore to something more meaningful * make default value 0, which is the same as the current behavior. 10 is the recommended value. * adding new flag to manifest.go * Adding changelog * Improve the description of the parameter in the generated config file. * Add metric to track the current number of active connections. * Change metric to gauge type and rename it. * e2e: Allow disabling the PEX reactor on all nodes in the testnet * Apply suggestions from code review * Update config/config.go comment * fix lint error * Improve config description * Rename metric (remove experimental prefix) * Add unit test * Improve unit test * Update mempool/reactor.go comment --------- * Updating test file, leaving it broken for now * mempool: Limit gossip connections to persistent and non-persistent peers (experimental) (cometbft#1584) * Ignore persistent peers from limiting of outbound connections * Update 1558-experimental-gossip-limiting.md Update changeling * Fix typo in mempool/metrics.go * Use two independent configs and semaphores for persistent and non-persistent peers * Forgot to rename in test * Update metric description * Rename semaphores * Add comment to unit test --------- * Reverting to old way of reporting errors * Reverting change that shouldn't have been included in cherry-pick * Reverting tests to use older functions * fix rebase merge --------- Co-authored-by: Adi Seredinschi <adizere@gmail.com> Co-authored-by: Ethan Buchman <ethan@coinculture.info> Co-authored-by: Daniel Cason <daniel.cason@informal.systems> Co-authored-by: hvanz <hernan.vanzetto@gmail.com> Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Sergio Mena <sergio@informal.systems> * Comment that feature only applies to v0 mempool * Fix new metric * This commit makes the test be the same as in main, that is, it ignores the order of transactions in the receiving reactor. (cometbft#1629) * Update .changelog/unreleased/improvements/1558-experimental-gossip-limiting.md Co-authored-by: Thane Thomson <connect@thanethomson.com> --------- Co-authored-by: Adi Seredinschi <adizere@gmail.com> Co-authored-by: Ethan Buchman <ethan@coinculture.info> Co-authored-by: Daniel Cason <daniel.cason@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Sergio Mena <sergio@informal.systems> Co-authored-by: lasaro <lasaro@informal.systems> Co-authored-by: Thane Thomson <connect@thanethomson.com>
Galadrin
pushed a commit
to Galadrin/cometbft
that referenced
this pull request
Dec 7, 2023
…4.x (cometbft#1630) * Backports cometbft#1558 and cometbft#1584 to 0.37.x (cometbft#1611) Backports cometbft#1558 and cometbft#1584 to 0.38.x (cometbft#1592) * mempool: Limit gossip connections to persistent and non-persistent peers (experimental) (cometbft#1584) * Experimental - Reduce # of connections effectively used to gossip transactions out (cometbft#1558) * maxpeers for mempool * mempool: fix max_peers bcast routine active flag * Use semaphore to limit concurrency * Rename MaxPeers to MaxOutboundPeers * Add max_outbound_peers to config toml template * Rename in error message * Renams the parameter to highlight its experimental nature. Extend the AddPeer method to return an error. Moves the semaphone to outside the broadcast routine * reverting the addition of error to AddPeer. It fails if the context is done and handling this case will be done some other time, when an actual context is passed into acquire. * reverting the addition of error to AddPeer. It fails if the context is done and handling this case will be done some other time, when an actual context is passed into acquire. * Fixing lint issue * renaming semaphore to something more meaningful * make default value 0, which is the same as the current behavior. 10 is the recommended value. * adding new flag to manifest.go * Adding changelog * Improve the description of the parameter in the generated config file. * Add metric to track the current number of active connections. * Change metric to gauge type and rename it. * e2e: Allow disabling the PEX reactor on all nodes in the testnet * Apply suggestions from code review * Update config/config.go comment * fix lint error * Improve config description * Rename metric (remove experimental prefix) * Add unit test * Improve unit test * Update mempool/reactor.go comment --------- * Updating test file, leaving it broken for now * mempool: Limit gossip connections to persistent and non-persistent peers (experimental) (cometbft#1584) * Ignore persistent peers from limiting of outbound connections * Update 1558-experimental-gossip-limiting.md Update changeling * Fix typo in mempool/metrics.go * Use two independent configs and semaphores for persistent and non-persistent peers * Forgot to rename in test * Update metric description * Rename semaphores * Add comment to unit test --------- * Reverting to old way of reporting errors * Reverting change that shouldn't have been included in cherry-pick * Reverting tests to use older functions * fix rebase merge --------- Co-authored-by: Adi Seredinschi <adizere@gmail.com> Co-authored-by: Ethan Buchman <ethan@coinculture.info> Co-authored-by: Daniel Cason <daniel.cason@informal.systems> Co-authored-by: hvanz <hernan.vanzetto@gmail.com> Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Sergio Mena <sergio@informal.systems> * Comment that feature only applies to v0 mempool * Fix new metric * This commit makes the test be the same as in main, that is, it ignores the order of transactions in the receiving reactor. (cometbft#1629) * Update .changelog/unreleased/improvements/1558-experimental-gossip-limiting.md Co-authored-by: Thane Thomson <connect@thanethomson.com> --------- Co-authored-by: Adi Seredinschi <adizere@gmail.com> Co-authored-by: Ethan Buchman <ethan@coinculture.info> Co-authored-by: Daniel Cason <daniel.cason@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Sergio Mena <sergio@informal.systems> Co-authored-by: lasaro <lasaro@informal.systems> Co-authored-by: Thane Thomson <connect@thanethomson.com>
Galadrin
pushed a commit
to Galadrin/cometbft
that referenced
this pull request
Dec 7, 2023
…4.x (cometbft#1630) * Backports cometbft#1558 and cometbft#1584 to 0.37.x (cometbft#1611) Backports cometbft#1558 and cometbft#1584 to 0.38.x (cometbft#1592) * mempool: Limit gossip connections to persistent and non-persistent peers (experimental) (cometbft#1584) * Experimental - Reduce # of connections effectively used to gossip transactions out (cometbft#1558) * maxpeers for mempool * mempool: fix max_peers bcast routine active flag * Use semaphore to limit concurrency * Rename MaxPeers to MaxOutboundPeers * Add max_outbound_peers to config toml template * Rename in error message * Renams the parameter to highlight its experimental nature. Extend the AddPeer method to return an error. Moves the semaphone to outside the broadcast routine * reverting the addition of error to AddPeer. It fails if the context is done and handling this case will be done some other time, when an actual context is passed into acquire. * reverting the addition of error to AddPeer. It fails if the context is done and handling this case will be done some other time, when an actual context is passed into acquire. * Fixing lint issue * renaming semaphore to something more meaningful * make default value 0, which is the same as the current behavior. 10 is the recommended value. * adding new flag to manifest.go * Adding changelog * Improve the description of the parameter in the generated config file. * Add metric to track the current number of active connections. * Change metric to gauge type and rename it. * e2e: Allow disabling the PEX reactor on all nodes in the testnet * Apply suggestions from code review * Update config/config.go comment * fix lint error * Improve config description * Rename metric (remove experimental prefix) * Add unit test * Improve unit test * Update mempool/reactor.go comment --------- * Updating test file, leaving it broken for now * mempool: Limit gossip connections to persistent and non-persistent peers (experimental) (cometbft#1584) * Ignore persistent peers from limiting of outbound connections * Update 1558-experimental-gossip-limiting.md Update changeling * Fix typo in mempool/metrics.go * Use two independent configs and semaphores for persistent and non-persistent peers * Forgot to rename in test * Update metric description * Rename semaphores * Add comment to unit test --------- * Reverting to old way of reporting errors * Reverting change that shouldn't have been included in cherry-pick * Reverting tests to use older functions * fix rebase merge --------- Co-authored-by: Adi Seredinschi <adizere@gmail.com> Co-authored-by: Ethan Buchman <ethan@coinculture.info> Co-authored-by: Daniel Cason <daniel.cason@informal.systems> Co-authored-by: hvanz <hernan.vanzetto@gmail.com> Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Sergio Mena <sergio@informal.systems> * Comment that feature only applies to v0 mempool * Fix new metric * This commit makes the test be the same as in main, that is, it ignores the order of transactions in the receiving reactor. (cometbft#1629) * Update .changelog/unreleased/improvements/1558-experimental-gossip-limiting.md Co-authored-by: Thane Thomson <connect@thanethomson.com> --------- Co-authored-by: Adi Seredinschi <adizere@gmail.com> Co-authored-by: Ethan Buchman <ethan@coinculture.info> Co-authored-by: Daniel Cason <daniel.cason@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Sergio Mena <sergio@informal.systems> Co-authored-by: lasaro <lasaro@informal.systems> Co-authored-by: Thane Thomson <connect@thanethomson.com>
Galadrin
pushed a commit
to Galadrin/cometbft
that referenced
this pull request
Dec 7, 2023
…4.x (cometbft#1630) * Backports cometbft#1558 and cometbft#1584 to 0.37.x (cometbft#1611) Backports cometbft#1558 and cometbft#1584 to 0.38.x (cometbft#1592) * mempool: Limit gossip connections to persistent and non-persistent peers (experimental) (cometbft#1584) * Experimental - Reduce # of connections effectively used to gossip transactions out (cometbft#1558) * maxpeers for mempool * mempool: fix max_peers bcast routine active flag * Use semaphore to limit concurrency * Rename MaxPeers to MaxOutboundPeers * Add max_outbound_peers to config toml template * Rename in error message * Renams the parameter to highlight its experimental nature. Extend the AddPeer method to return an error. Moves the semaphone to outside the broadcast routine * reverting the addition of error to AddPeer. It fails if the context is done and handling this case will be done some other time, when an actual context is passed into acquire. * reverting the addition of error to AddPeer. It fails if the context is done and handling this case will be done some other time, when an actual context is passed into acquire. * Fixing lint issue * renaming semaphore to something more meaningful * make default value 0, which is the same as the current behavior. 10 is the recommended value. * adding new flag to manifest.go * Adding changelog * Improve the description of the parameter in the generated config file. * Add metric to track the current number of active connections. * Change metric to gauge type and rename it. * e2e: Allow disabling the PEX reactor on all nodes in the testnet * Apply suggestions from code review * Update config/config.go comment * fix lint error * Improve config description * Rename metric (remove experimental prefix) * Add unit test * Improve unit test * Update mempool/reactor.go comment --------- * Updating test file, leaving it broken for now * mempool: Limit gossip connections to persistent and non-persistent peers (experimental) (cometbft#1584) * Ignore persistent peers from limiting of outbound connections * Update 1558-experimental-gossip-limiting.md Update changeling * Fix typo in mempool/metrics.go * Use two independent configs and semaphores for persistent and non-persistent peers * Forgot to rename in test * Update metric description * Rename semaphores * Add comment to unit test --------- * Reverting to old way of reporting errors * Reverting change that shouldn't have been included in cherry-pick * Reverting tests to use older functions * fix rebase merge --------- Co-authored-by: Adi Seredinschi <adizere@gmail.com> Co-authored-by: Ethan Buchman <ethan@coinculture.info> Co-authored-by: Daniel Cason <daniel.cason@informal.systems> Co-authored-by: hvanz <hernan.vanzetto@gmail.com> Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Sergio Mena <sergio@informal.systems> * Comment that feature only applies to v0 mempool * Fix new metric * This commit makes the test be the same as in main, that is, it ignores the order of transactions in the receiving reactor. (cometbft#1629) * Update .changelog/unreleased/improvements/1558-experimental-gossip-limiting.md Co-authored-by: Thane Thomson <connect@thanethomson.com> --------- Co-authored-by: Adi Seredinschi <adizere@gmail.com> Co-authored-by: Ethan Buchman <ethan@coinculture.info> Co-authored-by: Daniel Cason <daniel.cason@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Sergio Mena <sergio@informal.systems> Co-authored-by: lasaro <lasaro@informal.systems> Co-authored-by: Thane Thomson <connect@thanethomson.com>
Galadrin
pushed a commit
to Galadrin/cometbft
that referenced
this pull request
Dec 7, 2023
…4.x (cometbft#1630) * Backports cometbft#1558 and cometbft#1584 to 0.37.x (cometbft#1611) Backports cometbft#1558 and cometbft#1584 to 0.38.x (cometbft#1592) * mempool: Limit gossip connections to persistent and non-persistent peers (experimental) (cometbft#1584) * Experimental - Reduce # of connections effectively used to gossip transactions out (cometbft#1558) * maxpeers for mempool * mempool: fix max_peers bcast routine active flag * Use semaphore to limit concurrency * Rename MaxPeers to MaxOutboundPeers * Add max_outbound_peers to config toml template * Rename in error message * Renams the parameter to highlight its experimental nature. Extend the AddPeer method to return an error. Moves the semaphone to outside the broadcast routine * reverting the addition of error to AddPeer. It fails if the context is done and handling this case will be done some other time, when an actual context is passed into acquire. * reverting the addition of error to AddPeer. It fails if the context is done and handling this case will be done some other time, when an actual context is passed into acquire. * Fixing lint issue * renaming semaphore to something more meaningful * make default value 0, which is the same as the current behavior. 10 is the recommended value. * adding new flag to manifest.go * Adding changelog * Improve the description of the parameter in the generated config file. * Add metric to track the current number of active connections. * Change metric to gauge type and rename it. * e2e: Allow disabling the PEX reactor on all nodes in the testnet * Apply suggestions from code review * Update config/config.go comment * fix lint error * Improve config description * Rename metric (remove experimental prefix) * Add unit test * Improve unit test * Update mempool/reactor.go comment --------- * Updating test file, leaving it broken for now * mempool: Limit gossip connections to persistent and non-persistent peers (experimental) (cometbft#1584) * Ignore persistent peers from limiting of outbound connections * Update 1558-experimental-gossip-limiting.md Update changeling * Fix typo in mempool/metrics.go * Use two independent configs and semaphores for persistent and non-persistent peers * Forgot to rename in test * Update metric description * Rename semaphores * Add comment to unit test --------- * Reverting to old way of reporting errors * Reverting change that shouldn't have been included in cherry-pick * Reverting tests to use older functions * fix rebase merge --------- Co-authored-by: Adi Seredinschi <adizere@gmail.com> Co-authored-by: Ethan Buchman <ethan@coinculture.info> Co-authored-by: Daniel Cason <daniel.cason@informal.systems> Co-authored-by: hvanz <hernan.vanzetto@gmail.com> Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Sergio Mena <sergio@informal.systems> * Comment that feature only applies to v0 mempool * Fix new metric * This commit makes the test be the same as in main, that is, it ignores the order of transactions in the receiving reactor. (cometbft#1629) * Update .changelog/unreleased/improvements/1558-experimental-gossip-limiting.md Co-authored-by: Thane Thomson <connect@thanethomson.com> --------- Co-authored-by: Adi Seredinschi <adizere@gmail.com> Co-authored-by: Ethan Buchman <ethan@coinculture.info> Co-authored-by: Daniel Cason <daniel.cason@informal.systems> Co-authored-by: Andy Nogueira <me@andynogueira.dev> Co-authored-by: Sergio Mena <sergio@informal.systems> Co-authored-by: lasaro <lasaro@informal.systems> Co-authored-by: Thane Thomson <connect@thanethomson.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport #1558 and #1584 to v0.34.x
Cheery-picked from the backport to v0.37.x (#1611)
PR checklist
.changelog(we use unclog to manage our changelog)docs/orspec/) and code comments