test: add functional test for outbound connection management#33954
test: add functional test for outbound connection management#33954mzumsande wants to merge 3 commits intobitcoin:masterfrom
Conversation
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code Coverage & BenchmarksFor details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33954. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please copy-paste ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
8005a48 to
1158564
Compare
|
Concept ACK, thanks! |
|
there are still some intermittent timeouts, will put in draft until I've fixed them. |
1158564 to
b961ad4
Compare
53457d8 to
0cb0857
Compare
| if self.auto_outbound_factory: | ||
| actual_to_addr, actual_to_port, listener = self.auto_outbound_factory(i) | ||
| else: | ||
| # Create listener with default P2PInterface | ||
| listener = P2PInterface() | ||
| actual_to_addr, actual_to_port = self._create_auto_outbound_listener(listener) |
There was a problem hiding this comment.
Maybe get auto_outbound_factory() and _create_auto_outbound_listener() to have the same interface? That is, change _create_auto_outbound_listener() to also return the listener? Then this will be more consistent:
if self.auto_outbound_factory:
actual_to_addr, actual_to_port, listener = self.auto_outbound_factory(i)
else:
actual_to_addr, actual_to_port, listener = self._create_auto_outbound_listener(listener)and then we can have one function (pointer) that either points to the default one or to the user-provided one and here just:
actual_to_addr, actual_to_port, listener = self.auto_outbound_factory(i)without an if condition. That would mean to also rename _create_auto_outbound_listener() to auto_outbound_factory() and remove the line self.auto_outbound_factory = None from BitcoinTestFramework::__init__().
Feel free to ignore if you do not see this as an improvement.
There was a problem hiding this comment.
I like that suggestion - took it!
| socks5_config = self._setup_auto_outbound_mode() | ||
| for i in range(num_nodes): | ||
| extra_args[i] = extra_args[i] + [ | ||
| f"-proxy={socks5_config.addr[0]}:{socks5_config.addr[1]}" |
There was a problem hiding this comment.
Maybe this needs format_addr_port() as well, in case the address happens to be IPv6 and needs to be enclosed in []?
There was a problem hiding this comment.
The config address is hardcoded in _setup_auto_outbound_mode to 127.0.0.1, so IPv6 shouldn't be a concern I think?
There was a problem hiding this comment.
Right, only if changed in the future. Or if somebody is playing with this locally and changes it to fir their environment. An address like ::1 would also upset this because it would result in -proxy=::1:12345 which I guess will result in "syntax" error.
| def disconnect_last_block_relay_peer(self): | ||
| """Find and disconnect the most recently connected block-relay-only peer. | ||
| Done manually instead of waiting for the node to disconnect automatically to | ||
| avoid rare interactions with the scheduling of the next extra block-relay-only peer""" | ||
| node = self.nodes[0] |
There was a problem hiding this comment.
The other helper functions
has_regular_outbounds(),
has_extra_block_relay_peer() and
has_extra_full_outbound_peer()
take a node argument. Maybe add such argument also to disconnect_last_block_relay_peer() for consistency?
PS1 same for wait_for_network_specific_peer()
PS2 all of those end up using self.nodes[0], so another option is to drop the node argument from all.
There was a problem hiding this comment.
I decided to drop the node arg everywhere.
| self.log.info("Advance time") | ||
| self.cur_time += self.MOCKTIME_BUMP | ||
| node.setmocktime(self.cur_time) | ||
| self.generate(node, 1) # Disable stale-tip extra-outbound connections that could interfer |
There was a problem hiding this comment.
Is there an unlikely scenario that a stale-tip extra-outbound connection is initiated after advancing the mocktime and before self.generate(node, 1)? And eventually causing the interference that this is trying to disable?
There was a problem hiding this comment.
I think probably not in practice - CheckForStaleTipAndEvictPeers fires every 45 seconds (on a steady clock, not influenced by the mocktime bump). So I think this means it would only fire if 45 real seconds passed since the last mockscheduler call, which seems very unlikely.
| def wait_for_network_specific_peer(self): | ||
| """Advance time, wait for a network-specific full outbound peer, and trigger eviction. | ||
|
|
||
| Extra block-relay-only connections are prioritized over network-specific ones, but defensively: | ||
| if an already connected tor or cjdns address is picked, there is no retry. This can cause either | ||
| type of peer to arrive first. | ||
| """ |
There was a problem hiding this comment.
Is it possible that the node already has at least one connection to all reachable networks when this function starts? Would that result in timeouts due to has_extra_full_outbound_peer(node) never returning True?
There was a problem hiding this comment.
To avoid this, the test first adds 100 IPv4 addresses to addrman, then waits for the regular otubound connections (setup_peers), and only then the onion and cjdns addresses are added to addrman. So I think it shouldn't be possible.
There was a problem hiding this comment.
What about the tor and cjdns connections being established after adding to addrman and before wait_for_network_specific_peer() is called?
0cb0857 to
1562496
Compare
This new option allows tests to make use of automatic outbound connections (made from addrman). A SOCKS5 proxy is used to redirect the connection, by default to a P2PInterface For advanced customization, tests can provide auto_outbound_factory() to overwrite where each outbound connection made by the node should be redirected to (for example to another node) Co-authored-by: Vasil Dimov <vd@FreeBSD.org>
1562496 to
b5e9e38
Compare
Using the test framework avoids some duplicated code.
The test covers - Extra outbound connections when the tip is stale - Regularly scheduled extra block-relay-only connections - Management of full outbound connection slots with respect to network-specific peers
b5e9e38 to
c733d11
Compare
mzumsande
left a comment
There was a problem hiding this comment.
Thanks for the review @vasild!
Rebased, addressed feedback, and also added a commit that uses the test-framework auto-outbound mode for p2p_private_broadcast.py, removing some duplicate code.
I just had a rare (?) local intermittent failure of p2p_private_broadcast.py that I need to look into, so I will keep it in draft a little bit longer.
| def destinations_factory(requested_to_addr, requested_to_port): | ||
| with self.auto_outbound_destinations_lock: | ||
| i = len(self.auto_outbound_destinations) | ||
|
|
||
| actual_to_addr, actual_to_port, listener = self.auto_outbound_factory(i) | ||
|
|
There was a problem hiding this comment.
The order in which bitcoind makes connections is a bit undeterministic and also which connection is it going to make is not 100% deterministic either (e.g. is the next connection going to be full-outbound or feeler or net-specific?). #34410 addresses that by not relying only on the connection index but looking up the requested-to address and port in the debug log in order to figure out what the connection is.
That said, auto_outbound_factory() might benefit from getting more information about the connection, in addition to its index, like the requested address and port:
- actual_to_addr, actual_to_port, listener = self.auto_outbound_factory(i)
+ actual_to_addr, actual_to_port, listener = self.auto_outbound_factory(i, requested_to_addr, requested_to_port)|
🐙 This pull request conflicts with the target branch and needs rebase. |
For #29415, vasild added the possibility to have outbound connections made "naturally" by the node using addrman entries, by redirecting them via a SOCKS5 proxy to python p2ps or other full nodes, so that the node under test behaves as if it is connected to normal non-local peers from arbitrary networks, while we have full control over the peers.
I think this feature is really cool and want to suggest to integrate it more prominently into the test framework ("
auto_outbound_mode") in the first commit because it allows to test the code in which the node decides to make connections - in contrast to the existingadd_outbound_p2p_connectionwhere the functional test intiates the connections.The second commit uses it to add functional test coverage for the following features of outbound connection management:
This should work well with the test from #29415 (which would keep its own
destinations_factorybut can call_create_auto_outbound_listenerto avoid much of code duplication).Other possible uses for
auto_outbound_modeI could think of are more detailed testing of anchors (feature_anchors.py), feeler connections, or any other p2p logic that uses the network of peers.