Description

The transaction pool in bor (derived from geth) follows a very strict consistency model as there's only a single lock which is being used for all internal operations. This means the reader of the pool will get the most updated results but this takes a hit in performance. This doesn't prioritise the miner which needs the next set of transactions to process immediately. The waiting time when the pool is filled with ~100k transactions can go >1s leaving very less time for execution.

This PR does multiple changes to decouple the internal price heap (which is a costly operation when there are more elements) from the run reorg operation. This frees up the lock faster allowing miner to pull transactions without waiting much. It still aims to achieve eventual consistency by updating everything in background.

Some more technical details on the changes

• Move operations needing lock in run reorg function together, release the lock as early as possible, and then call the reheap (which is costly). We're okay if this takes more time now as worker can immediately call the pending function and get whatever needed for building the next block.
• Protect the functions of priced heap by it's internal lock (which is also used for reheap)
• Convert some functions in add transactions to run asynchronously i.e. run them in a go routine unblocking the parent function. This mostly involves all calls to the heap (adding / removing).
• There are some blocking heap calls in add transactions which needs the lock. Unblock the global lock while heap operations are going on. Once the heap operations are done, we can acquire the lock again. This is safe to do as this will delay the addition of a new transaction in the worst case. On the other hand, it reduces the wait time of miner.

On implementing these changes, pretty good performance improvements can be seen in the waiting time of worker to get pending transactions on a shadow forked mainnet node. While the experiment was conducted only a couple of times, the pending time has reduced to <200-300ms which is way better than ~1s earlier. This means we have way more time for execution and are not blocked by internal pool operations.

Changes

• Bugfix (non-breaking change that solves an issue)
• Hotfix (change that solves an urgent issue, and requires immediate attention)
• New feature (non-breaking change that adds functionality)
• Breaking change (change that is not backwards-compatible and/or changes current functionality)
• Changes only for a subset of nodes

Breaking changes

Please complete this section if any breaking changes have been made, otherwise delete it

Nodes audience

In case this PR includes changes that must be applied only to a subset of nodes, please specify how you handled it (e.g. by adding a flag with a default value...)

Checklist

• I have added at least 2 reviewer or the whole pos-v1 team
• I have added sufficient documentation in code
• I will be resolving comments - if any - by pushing each fix in a separate commit and linking the commit hash in the comment reply
• Created a task in Jira and informed the team for implementation in Erigon client (if applicable)
• Includes RPC methods changes, and the Notion documentation has been updated

Cross repository changes

• This PR requires changes to heimdall
  • In case link the PR here:
• This PR requires changes to matic-cli
  • In case link the PR here:

Testing

• I have added unit tests
• I have added tests to CI
• I have tested this code manually on local environment
• I have tested this code manually on remote devnet using express-cli
• I have tested this code manually on amoy
• I have created new e2e tests into express-cli

Manual tests

Please complete this section with the steps you performed if you ran manual tests for this functionality, otherwise delete it

Additional comments

Please post additional comments in this section if you have them, otherwise delete it