Skip to content

Conversation

@pinheadmz
Copy link
Member

@pinheadmz pinheadmz commented Nov 27, 2022

Closes #26579

The -netinfo dashboard did not list peers that were connected via "unroutable" networks. This included local peers including local-network peers. Personally, I run one bitcoind instance on my network that is used by other services like Wasabi Wallet and LND running on other machines.

This PR adds an "npr" (not publicly routable) column to the table of networks (ipv4, ipv6, onion, etc) so that every connection to the node is listed, and the totals are accurate as they relate to max inbound and max outbound limits.

Example connecting in regtest mode to one local and one remote peer:

Bitcoin Core client v24.99.0-151ce099ea8f-dirty regtest - server 70016/Satoshi:24.99.0/

<->   type   net  mping   ping send recv  txn  blk  hb addrp addrl  age id address         version
 in          npr      0      0   90   90                              1  1 127.0.0.1:59180 70016/Satoshi:24.99.0/
out manual  ipv4     63     63   84   84         3                    3  0 143.244.175.41  70016/Satoshi:24.0.1/
                     ms     ms  sec  sec  min  min                  min

         ipv4    ipv6     npr   total   block  manual
in          0       0       1       1
out         1       0       0       1       0       1
total       1       0       1       2

Local addresses: n/a


@DrahtBot
Copy link
Contributor

DrahtBot commented Nov 27, 2022

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process.

Type Reviewers
ACK jonatack

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #26988 (rpc: Add test-only RPC addrmaninfo for new/tried table address count by stratospher)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@jonatack
Copy link
Member

Thanks! I'll have a look today.

Copy link
Member

@jonatack jonatack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK

Copy link
Member

@jonatack jonatack Nov 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing NETWORKS here also affects -addrinfo, so it would need to be updated as well. Otherwise, with this change -addrinfo will print an additional line that always returns "other": 0,.

Would it make any sense to handle Network::NET_UNROUTABLE and Network::INTERNAL separately? Say, print npr for "not publicly routable" and int for "internal". In theory both could potentially be returned by getpeerinfo, which calls GetNetworkName() for the string value, and the GUI handles both of them in NetworkToQString(). It doesn't seem to make sense that we would ever see an "internal" returned by getpeerinfo (called by -netinfo) or by getnodeaddresses (called by -addrinfo, and getnodeaddresses also raises if called with "not publicly routable" passed as a network), but as a sanity check perhaps handle them like the GUI does.

Also, the -netinfo help would need to be updated for the net column description.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated and force-pushed, i think the logic is cleaner now anyway, by just including the extra categories from GetNetworkName() but only displaying them if peers are actually connected there. Thanks!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just as we only print columns for networks we are using, I think it would be better to only print this column if we have any such peers.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@maflcko maflcko changed the title cli: include local ("unroutable") peers in -netinfo table cli: include local ("unroutable") peers in -netinfo table Nov 28, 2022
@maflcko maflcko changed the title cli: include local ("unroutable") peers in -netinfo table cli: include local ("unroutable") peers in -netinfo table Nov 28, 2022
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NETWORK_SHORTNAMES?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code looks like it should probably get some refactoring. No reason it can't work with network strings (not ids) and figure out what "unlisted" networks have peers on its own, and doing so would be more future-proof as an added bonus.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea -- would it make sense to #include <netbase.h> in bitcoin-cli.cpp? Then we can share one single list enum Network as well as GetNetworkName() and ParseNetwork() without re-writing those in bitcoin-cli. I could also add a new function GetNetworkShortName() in netbase.cpp so everything is in one place. If new networks are added in the future, bitcoin-cli.cpp shouldn't need any changes...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@luke-jr gave this refactor a try in 7dd8b095ada2711487057788206deb7e310b8092

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I'm saying there shouldn't be an enum Network or network_id at all - just use the strings.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok something like std::map<std::string, int> to keep track of count of peers by network string?

@luke-jr
Copy link
Member

luke-jr commented Dec 3, 2022

## Commit message ##
-    cli: include local ("unroutable") peers in -netinfo table
+    cli: include local ("unreachable") peers in -netinfo table

Both of these are kinda wrong :|

@pinheadmz
Copy link
Member Author

@luke-jr I think I implemented the cleaner refactor you were thinking of in 0d0994a2e2e7b10ee428fde0f6c9235faf2a8e6a

I also applied that to -addrinfo which does have one drawback, we will now only list networks that come up in the getnodeaddresses RPC call, and not "all" networks, which were previously defined in a hard-coded list. Dunno if @jonatack has any thoughts on that, or if this is even considered a breaking API change ?

@fanquake
Copy link
Member

fanquake commented Dec 9, 2022

or if this is even considered a breaking API change ?

These interfaces have no API or guarantees. This is called out in -netinfo, and the same should hold for -addrinfo:

This human-readable interface will change regularly and is not intended to be a stable API.

Comment on lines 278 to 281
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (m_counts.find(network) == m_counts.end()) {
m_counts.insert({network, 0});
}
++m_counts.find(network)->second;
++m_counts[network];

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, fixed in bf69a13677680dbf2065e18e3e5c7882ec3aab33

Comment on lines 463 to 464
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (!network["reachable"].get_bool())
continue;
if (!network["reachable"].get_bool()) {
continue;
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, fixed in bf69a13677680dbf2065e18e3e5c7882ec3aab33

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Total is just in+out, and turning the std::array into a std::pair gets default zero initialisation, which would be useful later. We only need total one place, so IMO we should just sum it there rather than counting one by one.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in bf69a13677680dbf2065e18e3e5c7882ec3aab33 -- the only messy part was that std::pair values can't be retrieved by variable like i in the for loop, so I put a switch in there so that 0 = first and 1 = second, I dunno if that's best.

Comment on lines 514 to 515
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs braces at least, but it would be nice to avoid finding twice too.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks fixed in bf69a13677680dbf2065e18e3e5c7882ec3aab33

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to sort the networks in some particular order?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the networks will be sorted by reachable networks first (ipv4, ipv6) followed by any non-reachable networks we still have peers for (npr, i.e. local). The order of reachable networks will be provided by rpc getnetworkinfo which I think ultimately comes form the order of network names in netbase.cpp

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

std::map sorts alphabetically, IIRC?

@jonatack
Copy link
Member

Not sure I'm keen on the refactorings; are they necessary? It looks like they are trading an efficient data structure for a less-efficient one and moving additional logic/processing work into the loop.

@luke-jr
Copy link
Member

luke-jr commented Dec 14, 2022

I'm not sure a std::map is less efficient than mapping to arbitrary [hash-like] values, the latter of which breaks/has to be kept updated any time the software changes. The refactor just makes the interface generic so it works without a hard-coded network list.

Comment on lines 475 to 481
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// If peer network is not reachable, insert into map (only display if used)
m_counts[network];
if (is_outbound) {
++m_counts.find(network)->second.second;
} else {
++m_counts.find(network)->second.first;
}
++std::get<is_outbound ? 1 : 0>(m_counts[network]);

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't think you can use a variable in std::get< ? > ?

bitcoin-cli.cpp:476:15: error: no matching function for call to 'get'
 ++std::get<is_outbound ? 1 : 0>(m_counts[network]);

https://stackoverflow.com/questions/43621098/c-stdgetvariable-fails

Comment on lines 504 to 508
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
std::string name = network;
const std::map<std::string, std::string>::const_iterator it = NETWORK_SHORTNAMES.find(name);
if (it != NETWORK_SHORTNAMES.end()) {
name = it->second;
}
const std::map<std::string, std::string>::const_iterator it = NETWORK_SHORTNAMES.find(network);
const std::string& network_display_name = (it == NETWORK_SHORTNAMES.end()) ? network : it->second;

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or maybe add a nice utility function like:

template <typename M>
const M::mapped_type map_at_or(const M& map, const M::key_type& key, const M::mapped_type& default) {
    const auto it = map.find(key);
    if (it == map.end()) return default;
    return it->second;
}

Then we can do:

                const std::string& network_display_name = map_at_or(NETWORK_SHORTNAMES, network, network);

Comment on lines 579 to 591
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for (const auto& type : m_counts) {
int count = 0;
switch (i) {
case 0:
count = std::get<0>(type.second); // inbound peers by network
break;
case 1:
count = std::get<1>(type.second); // outbound peers by network
break;
case 2:
count = std::get<0>(type.second) + std::get<1>(type.second); // total peers by network
break;
}
for (const auto& [network, node_counts] : m_counts) {
const int count = (i == 2) ? (node_counts.first + node_counts.second) : std::get<i>(node_counts);

@pinheadmz
Copy link
Member Author

I reset this PR back to the original commit (with Luke's nit addressed). I'd like to try and just get the minimal code changes merged to improve the feature. Then if you guys like we can talk about ways to refactor if necessary

@jonatack
Copy link
Member

jonatack commented Feb 1, 2023

Tested ACK with suggestions for clarity or coherence with the existing code.

tested by adding the following 3 lines

@@ -562,6 +562,10 @@ public:
             result += strprintf("                     ms     ms  sec  sec  min  min                %*s\n\n", m_max_age_length, "min");
         }
 
+        m_counts.at(0).at(0) += 1;
+        m_counts.at(1).at(0) += 2;
+        m_counts.at(2).at(0) += 3;
+
         // Report peer connection totals by type.

suggestions diff

@@ -58,7 +58,7 @@ static constexpr int8_t UNKNOWN_NETWORK{-1};
 // See GetNetworkName() in netbase.cpp
 static constexpr std::array NETWORKS{"not_publicly_routable", "ipv4", "ipv6", "onion", "i2p", "cjdns", "internal"};
 static constexpr std::array NETWORK_SHORT_NAMES{"npr", "ipv4", "ipv6", "onion", "i2p", "cjdns", "int"};
-static constexpr std::array NETWORK_UNREACHABLE{0 /* "not_publicly_routable" */, 6 /* "internal" */};
+static constexpr std::array UNREACHABLE_NETWORK_IDS{/*not_publicly_routable*/0, /*internal*/6};
 
 /** Default number of blocks to generate for RPC generatetoaddress. */
 static const std::string DEFAULT_NBLOCKS = "1";
@@ -575,10 +575,10 @@ public:
         for (const UniValue& network : networkinfo["networks"].getValues()) {
             if (network["reachable"].get_bool()) {
                 const std::string& network_name{network["name"].get_str()};
                 const int8_t network_id{NetworkStringToId(network_name)};
                 if (network_id == UNKNOWN_NETWORK) continue;
                 result += strprintf("%8s", network_name); // column header
                 reachable_networks.push_back(network_id);
             }
         };
 
-        for (const size_t index : NETWORK_UNREACHABLE) {
-            if (m_counts.at(2).at(index)) {
-                reachable_networks.push_back(index);
-                result += strprintf("%8s", NETWORK_SHORT_NAMES[index]);
+        for (size_t network_id : UNREACHABLE_NETWORK_IDS) {
+            if (m_counts.at(2).at(network_id)) {
+                result += strprintf("%8s", NETWORK_SHORT_NAMES.at(network_id)); // column header
+                reachable_networks.push_back(network_id);
             }
         }

@pinheadmz
Copy link
Member Author

@jonatack thank you! applied suggestions and f-pushed

Copy link
Member

@jonatack jonatack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK ea4fbc5277c6fdf43cf675c8cc612f4821f55868, modulo the following change so that -addrinfo prints only the relevant networks:

@@ -292,7 +292,7 @@ public:
         // Prepare result to return to user.
         UniValue result{UniValue::VOBJ}, addresses{UniValue::VOBJ};
         uint64_t total{0}; // Total address count
-        for (size_t i = 0; i < NETWORKS.size(); ++i) {
+        for (size_t i = 1; i < NETWORKS.size() - 1; ++i) {

Tested this further with the following change on a running node:

--- a/src/bitcoin-cli.cpp
+++ b/src/bitcoin-cli.cpp
@@ -478,7 +478,7 @@ public:
         // Count peer connection totals, and if DetailsRequested(), store peer data in a vector of structs.
         for (const UniValue& peer : batch[ID_PEERINFO]["result"].getValues()) {
             const std::string network{peer["network"].get_str()};
-            const int8_t network_id{NetworkStringToId(network)};
+            const int8_t network_id = NetworkStringToId(network) % 2 ? 0 : 6;

Screenshot 2023-02-01 at 13 36 12

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can ignore, the following version might be more idiomatic C++.

-        for (const size_t network_id : UNREACHABLE_NETWORK_IDS) {
-            if (m_counts.at(2).at(network_id)) {
-                result += strprintf("%8s", NETWORK_SHORT_NAMES.at(network_id)); // column header
-                reachable_networks.push_back(network_id);
-            }
+        for (size_t network_id : UNREACHABLE_NETWORK_IDS) {
+            if (m_counts.at(2).at(network_id) == 0) continue;
+            result += strprintf("%8s", NETWORK_SHORT_NAMES.at(network_id)); // column header
+            reachable_networks.push_back(network_id);
         }

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! both suggestions applied.

@jonatack
Copy link
Member

jonatack commented Feb 2, 2023

Re-tested ACK 77192c9

@jonatack
Copy link
Member

jonatack commented Feb 13, 2023

This is a small, contained, non-risky change and seems fine to merge.

@maflcko maflcko merged commit 68e484a into bitcoin:master Feb 15, 2023
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Feb 15, 2023
…nfo table

77192c9 cli: include local ("unreachable") peers in -netinfo table (Matthew Zipkin)

Pull request description:

  Closes bitcoin#26579

  The `-netinfo` dashboard did not list peers that were connected via "unroutable" networks. This included local peers including local-network peers. Personally, I run one bitcoind instance on my network that is used by other services like Wasabi Wallet and LND running on other machines.

  This PR adds an "npr" (not publicly routable) column to the table of networks (ipv4, ipv6, onion, etc) so that every connection to the node is listed, and the totals are accurate as they relate to max inbound and max outbound limits.

  Example connecting in regtest mode to one local and one remote peer:

  ```
  Bitcoin Core client v24.99.0-151ce099ea8f-dirty regtest - server 70016/Satoshi:24.99.0/

  <->   type   net  mping   ping send recv  txn  blk  hb addrp addrl  age id address         version
   in          npr      0      0   90   90                              1  1 127.0.0.1:59180 70016/Satoshi:24.99.0/
  out manual  ipv4     63     63   84   84         3                    3  0 143.244.175.41  70016/Satoshi:24.0.1/
                       ms     ms  sec  sec  min  min                  min

           ipv4    ipv6     npr   total   block  manual
  in          0       0       1       1
  out         1       0       0       1       0       1
  total       1       0       1       2

  Local addresses: n/a

  ```

ACKs for top commit:
  jonatack:
    Re-tested ACK 77192c9

Tree-SHA512: 78aa68bcff0dbaadb5f0604bf023fe8fd921313bd8276d12581f7655c089466a48765f9e123cb31d7f1d294d5ca45fdefdf8aa220466ff738f32414f41099c06
@bitcoin bitcoin locked and limited conversation to collaborators Feb 15, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bitcoin-cli -netinfo ignores inbound peer from local network

6 participants