Slow discovery loop/node learning causes Alice.grant to timeout for large network of nodes #1444
Description
In the current network, Alice.grant randomly works or fails. Initially, it seemed that the sampling algorithm was including inaccessible Ursulas
File "/Users/derek/Documents/Github/repos/forks/david/nucypher/nucypher/policy/policies.py", line 386, in sample
handpicked_ursulas=handpicked_ursulas)
File "/Users/derek/Documents/Github/repos/forks/david/nucypher/nucypher/policy/policies.py", line 604, in sample_essential
timeout=30)
File "/Users/derek/Documents/Github/repos/forks/david/nucypher/nucypher/policy/policies.py", line 565, in __find_ursulas
raise RuntimeError(f"Timed out after {timeout} seconds; Cannot find {missing_nodes}.")
RuntimeError: Timed out after 30 seconds; Cannot find 0xFC1324835DE22d78b8E5aF783a962904D0460355.
but @cygnusv pointed out that it is probably related to the discovery loop and the fact that node certificates are downloaded and stores 2 or 3 times for all nodes, so the timeout is exceeded.
Discussion from Discord on November 5th
dnunezToday at 12:28 PM
It's not a problem of unavailable Ursulas
It's a problem of the discovery loop
For some reason, when granting the loop tries to learn again about all nodes
kpraschToday at 12:29 PM
All nodes, or blocking for specific ones?
dnunezToday at 12:29 PM
Downloads and writes certificates like 2 or 3 times per node
All nodes
When the network had 30 nodes, a timeout of 30 seconds was enough
I think if you manually rise that timeout to the size of the network, you will be able to Grant
There's something extremely inefficient in the intersection of granting and the learning loop
But I haven't spent enough time to debug it
kpraschToday at 12:31 PM
What evidence do you have that the cert is being written 2 or 3 times?
(just getting my head around the issue)
dnunezToday at 12:32 PM
@kprasch logs and code inspection