Rerunning CI to see if the failure is intermittent.

(https://app.circleci.com/pipelines/github/nucypher/nucypher/7861/workflows/f1ae5cba-538c-4a1a-906d-f334553fc39f/jobs/133175)

Some results from the failure investigation.

test_one_node_stores_a_bunch_of_others: started at commit 37929b3

In Learner.load_seednodes(), a sprout corresponding to the seed node from the test was created twice: once as maybe_sage_node, and the second time as seed_node (in the iteration over self._seed_nodes). Since NodeSprout did not have __eq__, these objects were treated as different ones by FleetSensor (record_node() puts new nodes in a set _nodes_to_add). Then, since sets are unordered, either of the two objects could end up in the actual fleet state during record_fleet_state().

On the other hand, node_storage.store_node_metadata() call had a different logic - since it did not accumulate changes and instead just saved the given one in the dictionary, the newer object always overwrote the older one.

Therefore, during the assertion in the test, where known_nodes is compared with the node_storage, two things could happen:

• The seednode sprout never had mature() called on it. In that case the lists would differ, since two different sprout objects with the same checksum addresses were not considered equal.
• The seednode sprout (the newer one) had mature() called on it. In that case, one would see an Ursula object in one list, and a NodeSprout in the other, and the test would fail just the same.

Solution to this may vary depending on what semantics exactly we want. Currently, I committed the following solution:

• add NodeSprout.__eq__() (two sprouts are equal if their checksum addresses are equal)
• force the replacement behavior of the older object with the newer one in FleetSensor.record_node(), to match that of the node storage.

This seems to fix the issue. Although perhaps it is not the best solution, since it does not fix the similarly looking failure in test_learner_uses_both_nodes_from_storage_and_fallback_nodes.

Why it appeared in this PR? node.mature() in FleetSensor which was removed in this PR converted the sprout into an Ursula, and two Ursulas with the same stamps are considered equal, so only the former possibility from the list above caused a test failure, making it much more rare.

To be continued.

(Edit) Damn, this change opened the hell's gate.

More investigation.

The introduction of NodeSprout.__eq__ in this PR broke logic in remember_node(). At the start there, we're comparing node == self, and if True, we don't remember the node. This worked because if node was a NodeSprout, it didn't have __eq__, and the Character.__eq__ (of self) was called, which only needed other.stamp. The initial version of NodeSprout.__eq__ I added only compared sprouts to sprouts, so it allowed node to remember itself in FleetSensor, causing test failures.

Additionally, sets of mixed Ursula and NodeSprout objects could not be compared in tests because even when a sprout and a full Ursula were equal, their hash() was different - sprouts used public_address, and Ursulas used stamp. Since the new NodeSprout.__eq__ uses stamp, it is logical to change NodeSprout.__hash__ to use it as well, because the assumption is that if two objects are equal, their hashes should be equal too.

The cli failure seems to be present in main as well

The cli failure seems to be present in main as well

Ah, nevermind, it's the probationary period ending again.