-
Notifications
You must be signed in to change notification settings - Fork 38.7k
init: Split file path handling out of -asmap option #33631
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
This adds a new option -asmapfile which allows to set the asmap file, while -asmap now is simply a bool option.
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code Coverage & BenchmarksFor details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33631. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
IMO #33631 and #33632 are both reasonable ways of patching up a confusing UI and making it slightly better, and either would be an improvement over current behavior. But neither of these PRs seem to solve the fundamental problem here: that if a user goes through the trouble of saving diff
--- a/src/init.cpp
+++ b/src/init.cpp
@@ -1547,15 +1547,20 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
// Read asmap file if configured
std::vector<bool> asmap;
- if (args.IsArgSet("-asmap") && !args.IsArgNegated("-asmap")) {
- fs::path asmap_path = args.GetPathArg("-asmap", DEFAULT_ASMAP_FILENAME);
+ fs::path asmap_path = args.GetPathArg("-asmap", DEFAULT_ASMAP_FILENAME);
+ if (!asmap_path.empty()) {
if (!asmap_path.is_absolute()) {
asmap_path = args.GetDataDirNet() / asmap_path;
}
- if (!fs::exists(asmap_path)) {
+ if (!args.IsArgSet("-asmap") && !fs::exists(asmap_path)) {
+ // It is ok for asmap file not to exist if not specified.
+ asmap_path.clear();
+ } else if (!fs::exists(asmap_path)) {
InitError(strprintf(_("Could not find asmap file %s"), fs::quoted(fs::PathToString(asmap_path))));
return false;
}
+ }
+ if (!asmap_path.empty()) {
asmap = DecodeAsmap(asmap_path);
if (asmap.size() == 0) {
InitError(strprintf(_("Could not parse asmap file %s"), fs::quoted(fs::PathToString(asmap_path))));
--- a/test/functional/feature_asmap.py
+++ b/test/functional/feature_asmap.py
@@ -49,7 +49,14 @@ class AsmapTest(BitcoinTestFramework):
self.nodes[node_id].addpeeraddress(address=f"101.{addr}.0.0", tried=tried, port=8333)
def test_without_asmap_arg(self):
- self.log.info('Test bitcoind with no -asmap arg passed')
+ self.log.info(f'Test bitcoind with no -asmap arg passed, with {DEFAULT_ASMAP_FILENAME} present')
+ shutil.copyfile(self.asmap_raw, self.default_asmap)
+ self.stop_node(0)
+ with self.node.assert_debug_log(expected_messages(self.default_asmap)):
+ self.start_node(0)
+ os.remove(self.default_asmap)
+
+ self.log.info(f'Test bitcoind with no -asmap arg passed, without {DEFAULT_ASMAP_FILENAME} present')
self.stop_node(0)
with self.node.assert_debug_log(['Using /16 prefix for IP bucketing']):
self.start_node(0)Alternately, if it is unsafe to load this data by default unless the node is explicitly configured to load it, then it is unnecessarily confusing to define a default filename that is not used by default. It would make more sense to make the default value empty: diff
--- a/src/init.cpp
+++ b/src/init.cpp
@@ -158,7 +158,6 @@ static constexpr bool DEFAULT_STOPAFTERBLOCKIMPORT{false};
#endif
static constexpr int MIN_CORE_FDS = MIN_LEVELDB_FDS + NUM_FDS_MESSAGE_CAPTURE;
-static const char* DEFAULT_ASMAP_FILENAME="ip_asn.map";
/**
* The PID file facilities.
@@ -532,7 +531,7 @@ void SetupServerArgs(ArgsManager& argsman, bool can_listen_ipc)
ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
argsman.AddArg("-addnode=<ip>", strprintf("Add a node to connect to and attempt to keep the connection open (see the addnode RPC help for more info). This option can be specified multiple times to add multiple nodes; connections are limited to %u at a time and are counted separately from the -maxconnections limit.", MAX_ADDNODE_CONNECTIONS), ArgsManager::ALLOW_ANY | ArgsManager::NETWORK_ONLY, OptionsCategory::CONNECTION);
- argsman.AddArg("-asmap=<file>", strprintf("Specify asn mapping used for bucketing of the peers (default: %s). Relative paths will be prefixed by the net-specific datadir location.", DEFAULT_ASMAP_FILENAME), ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION);
+ argsman.AddArg("-asmap=<file>", strprintf("Specify asn mapping used for bucketing of the peers (default: none). Relative paths will be prefixed by the net-specific datadir location."), ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION);
argsman.AddArg("-bantime=<n>", strprintf("Default duration (in seconds) of manually configured bans (default: %u)", DEFAULT_MISBEHAVING_BANTIME), ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION);
argsman.AddArg("-bind=<addr>[:<port>][=onion]", strprintf("Bind to given address and always listen on it (default: 0.0.0.0). Use [host]:port notation for IPv6. Append =onion to tag any incoming connections to that address and port as incoming Tor connections (default: 127.0.0.1:%u=onion, testnet3: 127.0.0.1:%u=onion, testnet4: 127.0.0.1:%u=onion, signet: 127.0.0.1:%u=onion, regtest: 127.0.0.1:%u=onion)", defaultChainParams->GetDefaultPort() + 1, testnetChainParams->GetDefaultPort() + 1, testnet4ChainParams->GetDefaultPort() + 1, signetChainParams->GetDefaultPort() + 1, regtestChainParams->GetDefaultPort() + 1), ArgsManager::ALLOW_ANY | ArgsManager::NETWORK_ONLY, OptionsCategory::CONNECTION);
argsman.AddArg("-cjdnsreachable", "If set, then this host is configured for CJDNS (connecting to fc00::/8 addresses would lead us to the CJDNS network, see doc/cjdns.md) (default: 0)", ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION);
@@ -1548,11 +1547,11 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
// Read asmap file if configured
std::vector<bool> asmap;
if (args.IsArgSet("-asmap") && !args.IsArgNegated("-asmap")) {
- fs::path asmap_path = args.GetPathArg("-asmap", DEFAULT_ASMAP_FILENAME);
+ fs::path asmap_path = args.GetPathArg("-asmap");
if (!asmap_path.is_absolute()) {
asmap_path = args.GetDataDirNet() / asmap_path;
}
- if (!fs::exists(asmap_path)) {
+ if (!fs::is_regular_file(asmap_path)) {
InitError(strprintf(_("Could not find asmap file %s"), fs::quoted(fs::PathToString(asmap_path))));
return false;
}
--- a/test/functional/feature_asmap.py
+++ b/test/functional/feature_asmap.py
@@ -29,7 +29,6 @@ import shutil
from test_framework.test_framework import BitcoinTestFramework
from test_framework.util import assert_equal
-DEFAULT_ASMAP_FILENAME = 'ip_asn.map' # defined in src/init.cpp
ASMAP = 'src/test/data/asmap.raw' # path to unit test skeleton asmap
VERSION = 'fec61fa21a9f46f3b17bdcd660d7f4cd90b966aad3aec593c99b35f0aca15853'
@@ -80,21 +79,18 @@ class AsmapTest(BitcoinTestFramework):
os.remove(filename)
def test_default_asmap(self):
- shutil.copyfile(self.asmap_raw, self.default_asmap)
+ msg = f"Error: Could not find asmap file \"{self.datadir}{os.sep}\""
for arg in ['-asmap', '-asmap=']:
self.log.info(f'Test bitcoind {arg} (using default map file)')
self.stop_node(0)
- with self.node.assert_debug_log(expected_messages(self.default_asmap)):
- self.start_node(0, [arg])
- os.remove(self.default_asmap)
+ self.node.assert_start_raises_init_error(extra_args=[arg], expected_msg=msg)
def test_asmap_interaction_with_addrman_containing_entries(self):
self.log.info("Test bitcoind -asmap restart with addrman containing new and tried entries")
self.stop_node(0)
- shutil.copyfile(self.asmap_raw, self.default_asmap)
- self.start_node(0, ["-asmap", "-checkaddrman=1", "-test=addrman"])
+ self.start_node(0, [f"-asmap={self.asmap_raw}", "-checkaddrman=1", "-test=addrman"])
self.fill_addrman(node_id=0)
- self.restart_node(0, ["-asmap", "-checkaddrman=1", "-test=addrman"])
+ self.restart_node(0, [f"-asmap={self.asmap_raw}", "-checkaddrman=1", "-test=addrman"])
with self.node.assert_debug_log(
expected_msgs=[
"CheckAddrman: new 2, tried 2, total 4 started",
@@ -102,29 +98,29 @@ class AsmapTest(BitcoinTestFramework):
]
):
self.node.getnodeaddresses() # getnodeaddresses re-runs the addrman checks
- os.remove(self.default_asmap)
def test_default_asmap_with_missing_file(self):
self.log.info('Test bitcoind -asmap with missing default map file')
self.stop_node(0)
- msg = f"Error: Could not find asmap file \"{self.default_asmap}\""
- self.node.assert_start_raises_init_error(extra_args=['-asmap'], expected_msg=msg)
+ msg = f"Error: Could not find asmap file \"{self.datadir}{os.sep}missing\""
+ self.node.assert_start_raises_init_error(extra_args=[f'-asmap=missing'], expected_msg=msg)
def test_empty_asmap(self):
self.log.info('Test bitcoind -asmap with empty map file')
self.stop_node(0)
- with open(self.default_asmap, "w", encoding="utf-8") as f:
+
+ empty_asmap = os.path.join(self.datadir, "ip_asn.map")
+ with open(empty_asmap, "w", encoding="utf-8") as f:
f.write("")
- msg = f"Error: Could not parse asmap file \"{self.default_asmap}\""
- self.node.assert_start_raises_init_error(extra_args=['-asmap'], expected_msg=msg)
- os.remove(self.default_asmap)
+ msg = f"Error: Could not parse asmap file \"{empty_asmap}\""
+ self.node.assert_start_raises_init_error(extra_args=[f'-asmap={empty_asmap}'], expected_msg=msg)
+ os.remove(empty_asmap)
def test_asmap_health_check(self):
self.log.info('Test bitcoind -asmap logs ASMap Health Check with basic stats')
- shutil.copyfile(self.asmap_raw, self.default_asmap)
msg = "ASMap Health Check: 4 clearnet peers are mapped to 3 ASNs with 0 peers being unmapped"
with self.node.assert_debug_log(expected_msgs=[msg]):
- self.start_node(0, extra_args=['-asmap'])
+ self.start_node(0, extra_args=[f'-asmap={self.asmap_raw}'])
raw_addrman = self.node.getrawaddrman()
asns = []
for _, entries in raw_addrman.items():
@@ -133,12 +129,10 @@ class AsmapTest(BitcoinTestFramework):
if asn not in asns:
asns.append(asn)
assert_equal(len(asns), 3)
- os.remove(self.default_asmap)
def run_test(self):
self.node = self.nodes[0]
self.datadir = self.node.chain_path
- self.default_asmap = os.path.join(self.datadir, DEFAULT_ASMAP_FILENAME)
base_dir = self.config["environment"]["SRCDIR"]
self.asmap_raw = os.path.join(base_dir, ASMAP)
|
| argsman.AddArg("-addnode=<ip>", strprintf("Add a node to connect to and attempt to keep the connection open (see the addnode RPC help for more info). This option can be specified multiple times to add multiple nodes; connections are limited to %u at a time and are counted separately from the -maxconnections limit.", MAX_ADDNODE_CONNECTIONS), ArgsManager::ALLOW_ANY | ArgsManager::NETWORK_ONLY, OptionsCategory::CONNECTION); | ||
| argsman.AddArg("-asmap=<file>", strprintf("Specify asn mapping used for bucketing of the peers (default: %s). Relative paths will be prefixed by the net-specific datadir location.", DEFAULT_ASMAP_FILENAME), ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION); | ||
| argsman.AddArg("-asmap", "Use AS mapping for IP bucketing (default: 0)", ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION); | ||
| argsman.AddArg("-asmapfile=<file>", strprintf("Specify asn mapping file used for bucketing of the peers (default: %s). Relative paths will be prefixed by the net-specific datadir location.", DEFAULT_ASMAP_FILENAME), ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: AS mapping versus "asn mapping", let's settle on one spelling.
|
Concept ACK. Splitting it to a separate filename option makes sense. imo it's better to avoid overloading the type (boolean or path) of options and introducing ambiguity. This is conceptually simpler for the user and simplifies the code. |
|
@laanwj do you agree this PR would be unnecessary if the Or if it is not safe to load asmap data be default, it would seem cleaner not to have a default'filename that is not actually used by default. |
The reason i like this PR is that i dislike how the current code interprets an argument as either a boolean or a string. This is not compatible with stricter option parsing. If reading a file by default or embedding it into the binary gets rid of that, that's fine with me too. But i guess even in that case, one might want to either disable the functionality, or override the path? It seems partly orthogonal to these options.
Right. |
Yes, these things are done with
I do think the issues are pretty directly related, not orthogonal. The basic problem here is that this option's default value ( If the default behavior is changed to match the default value (as in the first diff), or if the default value is changed to match the default behavior (as in the second diff) no issues should remain. By contrast, this PR fixes the mismatched defaults by splitting them into two options, making the interface more complicated and requiring multiple parameters just to specify a single file. It also makes the way asmap files are specified different than the way other files are specified ( |
|
@ryanofsky @laanwj Thank you the in-depth discussion here. I previously thought it might be best to leave the default file behavior untouched but if people are unhappy with it then now, in this PR, is certainly the right time to fix it and I am not married to the idea of keeping it the same. I have thought about asmap more than most in the last couple of years so I've simply gotten used to how things were.
I do think it's a bit unsafe to simply introduce loading and using the default file just because it's present. I don't know how well people manage their datadirs but it could be that someone has a very old default file there from when the feature was first developed or some limited example file from a workshop or so. So the two options I would feel comfortable with right now would be: 1. Use the default file when it's present but rename the default file once in the same step. For example I have a slight tendency to the first option because some people seem to like the convenience of this feature. What do you think? |
I think I like both equally, and it feels like whichever default you pick shouldn't matter too much because it will change again soon when asmap data is embedded? (I'm assuming when asmap data is embedded, the default will be to use embedded data unless you specify -noasmap or a filename.) |
The long term plan is to have the embedded data used by default, however first the idea was to have a release or two with the data embedded but the usage still off by default. This is a cautious intermediate step similar to how assumeutxo was merged without a mainnet param and how v2 transport was off-by-default in the beginning. While I am personally very confident in these matters there are still regularly the same set of questions raised about the data that is embedded and how it degrades after 6 or 12 months. Also concerns about nodes from crowded ASs having trouble finding all the connections they want. The intermediary step helps to build confidence further over that time that on-by-default is truly effective and safe. Just giving some background but to discuss this in particular I think #28792 is the right place for that. That said, I think it makes sense to aim the design primarily for the future when embedded data is used by default. So for that future version we would have data from an explicit file path, the default file path or the embedded data and with any of those available the feature is used unless For the intermediate time period we have the options of the current behavior of this PR, or the options you mentioned above with my modification suggestion. Now that I have laid it out like this I am again leaning towards keeping the current behavior for the intermediary step: Data can be provided with explicit path, the default path or (with #28792) from the embedded data but no matter where the data is from, the feature isn't used unless And of course removing the default file just gets rid of this problem entirely. |
Currently, if `-asmap` is specified without a filename this tries to load `ip_asn.map` data file. This commit requires `-asmap=ip_asn.map` or another filename to be specified explicitly. The change is intended to make behavior of the option explicit avoid confusion reported bitcoin#33386 where documentation specifies a default file that is not actually loaded by default. This change was originally implemented in bitcoin#33631 (comment) and various alternatives were discussed there.
This makes sense, but I think if it is the plan, it should rule out option "1. Use the default file when it's present" so So I opened PR #33770 implementing option "2. Remove the default file behavior." I think it's a straightforward improvement with no real downsides that's less confusing than current behavior and more compatible with your short & long term plans. But if you do prefer current behavior over #33770 that's also fine and I'm happy to close it. I agree current behavior seems ok as an intermediate state. I was also thinking more about the problem you mentioned of "old default files laying in the datadir" being loaded. This seems like a general problem that will always exist as long as external asmap files can be loaded. A good solution could be have a timestamp in asmap files reflecting when the data was collected, and have bitcoind refuse to load external asmap files with timestamps less than the timestamp in their embedded asmap file. |
Currently, if `-asmap` is specified without a filename bitcoind tries to load `ip_asn.map` data file. This change now requires `-asmap=ip_asn.map` or another filename to be specified explicitly. The change is intended to make behavior of the option explicit avoid confusion reported bitcoin#33386 where documentation specifies a default file which is not actually loaded by default. It was originally implemented in bitcoin#33631 (comment) and various alternatives are discussed there.
Currently, if `-asmap` is specified without a filename bitcoind tries to load `ip_asn.map` data file. This change now requires `-asmap=ip_asn.map` or another filename to be specified explicitly. The change is intended to make behavior of the option explicit avoid confusion reported bitcoin#33386 where documentation specifies a default file which is not actually loaded by default. It was originally implemented in bitcoin#33631 (comment) and various alternatives are discussed there.
Currently, if `-asmap` is specified without a filename, bitcoind tries to load `ip_asn.map` data file. This change now requires `-asmap=ip_asn.map` or another filename to be specified explicitly. The change is intended to make behavior of the option explicit avoid confusion reported bitcoin#33386 where documentation specifies a default file which is not actually loaded by default. It was originally implemented in bitcoin#33631 (comment) and various alternatives are discussed there. Co-authored-by: Fabian Jahr <fjahr@protonmail.com>
|
Is this still relevant, given you've ACK'd #33770? |
Yes, these are solving two separate reasons people were unhappy with the Even though @ryanofsky didn't explicitly review this PR here yet my understanding is that he didn't intend #33770 as a replacement of this either, but please correct me if I am wrong. |
I did intend to #33770 to be an alternative this PR and as far I I know there wouldn't be any benefits to this PR if #33770 is merged, unless I'm missing something. Both this PR and #33770 are addressing the problem of |
Currently, if `-asmap` is specified without a filename, bitcoind tries to load `ip_asn.map` data file. This change now requires `-asmap=ip_asn.map` or another filename to be specified explicitly. The change is intended to make behavior of the option explicit avoid confusion reported bitcoin#33386 where documentation specifies a default file which is not actually loaded by default. It was originally implemented in bitcoin#33631 (comment) and various alternatives are discussed there. Co-authored-by: Fabian Jahr <fjahr@protonmail.com>
Ok, then I misunderstood this but I still think that these two topics are orthogonal and at CoreDev and here people have voiced support for having the bool option split from the file path. Especially in the context of having embedded data this is much clearer since #33770 doesn't fix the messy ambiguity between bool and path usage of the parameter. Also, we now could be in a temporary state with where using |
|
Thanks for explaining. Now I see why you still want this PR. In the short term with #33770 there are no problems, and in the long terms with embedded asmap turned on by default there are no problems. But in the medium turn with embedded asmap data present but not turned on by default, you dislike the fact that To clarify:
|
Yes, this is what has bothered me most about the arg for a long time and why I had included my minimal fix for this in the original embedding PR already, just implemented a bit differently than you suggest but with the same effect. I think it's just not acceptable that you couldn't even use the embedded asmap data by setting the option from the conf file because of this. This was also noted in #33386.
Right, but we don't know yet how long the long term means here. I am happy to see new reviewers of asmap PRs now but I think there may still be some additional convincing necessary to enable it by default. This is different from BIP324 where, as far as I remember, it was accepted pretty early that one release would be enough before it could be enabled by default. Given how scarce review has been I don't want to open this discussion for asmap before the embedding has actually been merged, to not create more distractions for those interested in the topic.
Sure, this is basically what I had suggested with #33632 just implemented differently, you had already commented on this there in earlier. I was asked to make a decision for which way to go between these different approaches to not drag out the discussion and I chose this one because the approach is cleaner even though it's more verbose and because I sensed slightly more support for it. I understand that you would prefer #33632 to this approach here and if more reviewers agree with you I am open to switch the approach. But for now I will keep going with this because of the rationale above and @laanwj had already expressed preference for this approach earlier: #33631 (comment) Lastly I want to add this for anyone else who reads this and wonders, why not stack this change on top of #28792 because |
|
I'm slightly confused as to what the plan is between this and #33770. This PR both has code conflicts and logical conflicts with #33770. 33770 redefines Since 33770 seems more likely to be merged soon, how would this PR change to resolve those conflicts. |
It is not the opposite way, it's an orthogonal issue. #33770 removes the usage of the default file and this PR splits the |
|
I think having two options to control such a minor feature is unintuive and overkill. If that's what people like, no objection as it really isn't important. But my preference would be to put everything in the
If #33770 goes in, and that's behavior we want to keep, |
|
I guess there are too many walls of text here and elsewhere so I will try to make it more concise, this PR here (and #33632 with a different approach) fixes two things:
#33770 doesn't fix this, it just gets rid of the current reason anyone would use |
288b8c3 doc: Drop (default: none) from -i2psam description (Ryan Ofsky) f6ec351 init: Require explicit -asmap filename (Ryan Ofsky) Pull request description: Currently, if `-asmap` is specified without a filename bitcoind tries to load `ip_asn.map` data file. This change now requires `-asmap=ip_asn.map` or another filename to be specified explicitly. The change is intended to make behavior of the option explicit and avoid confusion reported #33386 where documentation specifies a default file which is not actually loaded by default. It was originally implemented in #33631 (comment) and various alternatives are discussed there. ACKs for top commit: brunoerg: reACK 288b8c3 fjahr: re-ACK 288b8c3 vostrnad: utACK 288b8c3 achow101: ACK 288b8c3 Tree-SHA512: 11a38a03892a58d6ccc1505cfbf915f58a86df9891761d89dc54b92d40593ee3cbb2d7c7bdbb922b871b3529072ef7f34cc98393aff6e8f0633b56352315b27c
|
Yes conflict with #33770 is superficial, and both could be merged. They both solved #33386 in different ways, but are otherwise independent. I don't think this PR is a good idea, but don't oppose it if others like it. I don't think separating Right now since #33770 is merged there's no issue with the |
|
🐙 This pull request conflicts with the target branch and needs rebase. |
I agree. |
|
Seems like there is now more opposition to this approach, so I will close this and consider an alternative as a follow-up when the embedding is merged. |
Note: There is an alternative change that leaves the behavior of
-asmapas is but makes slight improvements and adds better documentation: #33632 Please indicate with your conceptual review which option you prefer, thanks!This change adds a new option
-asmapfilewhich allows to set the asmap file, while-asmapnow is simply a bool option. Also fixes the asmap functional test docs, since then had become out of sync with the actual content of the test.The current handling of
-asmapis a bit confusing and atypical of our usual options behavior. The-asmapoption can be used as a boolean which turns on the feature and looks for the file in the default location. But it can be alternatively be used to pass a path to a file which is not in the default location, which also turns on the feature implicitly.See #33386 for further discussion. Closes #33386