Skip to content

Treasure Island...#1741

Merged
KPrasch merged 443 commits intomainfrom
†reasure-ïsland-@ss-pi®ate-ß̆çh
Sep 8, 2020

Hidden character warning

The head ref may contain hidden characters: "\u2020reasure-\u00efsland-@ss-pi\u00aeate-\u00df\u00cc\u2020\u00e7h"
Merged

Treasure Island...#1741
KPrasch merged 443 commits intomainfrom
†reasure-ïsland-@ss-pi®ate-ß̆çh

Conversation

@jMyles
Copy link
Contributor

@jMyles jMyles commented Feb 24, 2020

A number of old, nagging issues have persisted since the early days of this codebase. A coherent collection of them represents a set of conjoined functionality designed primarily to support development, and specifically the development practices of those days, but create undue complexity for the more mature state of today, particularly in the areas of node discovery, treasure map (or other incidental payload) handling, the logic specific to "firstula" situations, and test logic.

This PR is an attempt to address a large enough subset of these issues to represent a "new normal" for the codebase, without doing so much as to leave readers confused.

What follows is a very high concentration of impact-per-line and issue-days-destroyed.

Faster failing for everyone

It is now much more difficult to accidentally proceed into the logic of any Character's method if that logic will eventually fail because no nodes are available. It is no longer necessary to call start_learning_loop to achieve failfast functionality. Fixes #2110.

Vladimir's comeback

Vladimir can now attempt to store a fraudulent TreasureMap on a target Ursula. Since we now restrict the size of uploads, this attack might be imagined as splitting up an HD movie into a few thousand small pieces and scattering them across the network.

In the course of ensuring that this attack is properly demonstrated and mitigated, all of Vladimir's logic has been refreshed and touched up. This fixes #1075.

Bob's favorite nodes

Where do we store an incidental payload (TreasureMap is the simplest example, but we also want forward-compatibility with others), we do so only on a tiny subset of nodes. These nodes are selected on the basis that the second character in Bob's encrypting trinket matches on of the first few characters of the node's checksum (perhaps soon canonical - see #1995). Fixes #342.

Alice signs maps with her wallet keypair; Ursula verifies against HRAC.

The do_store logic for TreasureMaps now includes a check that the map matches a published policy, and that the author of the policy has signed the map. Fixes #429. Fixes #1736.

Tests now use "hardcoded teachers" as seed node logic.

This was previously not possible because many tests (and in fact the fixtures) left the MOCK_KNOWN_URSULAS_CACHE unclean. Now, each test is responsible for this cleanup, and an error is raised by the major fixtures (federated_ursulas and blockchain_ursulas) if the cache is unclean at setup time.

New asynchrony patterns

Alice now enacts a Policy using a NodeEnagementMutex, which performs network operations asynchronously and has two blocking methods - one which returns after five percent completion and another which returns after conclusion.

Smaller changes

@jMyles jMyles force-pushed the †reasure-ïsland-@ss-pi®ate-ß̆çh branch from 88bef72 to 598f7db Compare February 26, 2020 18:04
@jMyles jMyles force-pushed the †reasure-ïsland-@ss-pi®ate-ß̆çh branch from 8ccfce4 to 65bd36a Compare March 5, 2020 00:10
@KPrasch KPrasch mentioned this pull request Mar 19, 2020
@jMyles jMyles force-pushed the †reasure-ïsland-@ss-pi®ate-ß̆çh branch 2 times, most recently from 9ae1bbf to 0e12b22 Compare April 24, 2020 10:18
@jMyles jMyles force-pushed the †reasure-ïsland-@ss-pi®ate-ß̆çh branch from 400f8cf to 0cb5484 Compare May 8, 2020 05:48
@mswilkison mswilkison added the Epic 📖 Upstream branches with a project focus label May 11, 2020
@jMyles jMyles force-pushed the †reasure-ïsland-@ss-pi®ate-ß̆çh branch 3 times, most recently from 52bc78d to 63cf84d Compare May 22, 2020 20:53
@jMyles jMyles force-pushed the †reasure-ïsland-@ss-pi®ate-ß̆çh branch 4 times, most recently from d7db4ee to 4f99fe6 Compare May 30, 2020 01:20
@jMyles jMyles force-pushed the †reasure-ïsland-@ss-pi®ate-ß̆çh branch from 70fd8db to 66a5f32 Compare June 1, 2020 22:30
@jMyles jMyles force-pushed the †reasure-ïsland-@ss-pi®ate-ß̆çh branch from 66a5f32 to 1a523c5 Compare June 10, 2020 04:31
@jMyles jMyles force-pushed the †reasure-ïsland-@ss-pi®ate-ß̆çh branch from 2b15a20 to f708b77 Compare June 23, 2020 09:34
@KPrasch KPrasch changed the base branch from master to main July 21, 2020 00:45
@jMyles
Copy link
Contributor Author

jMyles commented Jul 27, 2020

@jMyles jMyles force-pushed the †reasure-ïsland-@ss-pi®ate-ß̆çh branch from 2132631 to bde604f Compare July 27, 2020 18:21
Copy link
Member

@cygnusv cygnusv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some initial comments.

@KPrasch KPrasch force-pushed the †reasure-ïsland-@ss-pi®ate-ß̆çh branch 2 times, most recently from bb8cc85 to 4f121c3 Compare July 30, 2020 23:09
@jMyles jMyles force-pushed the †reasure-ïsland-@ss-pi®ate-ß̆çh branch from e2ae201 to 2f3eeac Compare September 2, 2020 03:32
self.is_me = is_me

self._checksum_address = None # Stake Address
# self._checksum_address = None # Stake Address # TODO - wait, why? Why are we setting this to None when it may have already been set in an outer method?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you ever figure this out?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No.

Base class for Nucypher protocol actors.
A participant in the cryptological drama (a screenplay, if you like) of NuCypher.

Characters can represent users, nodes, wallets, offline devices, or other objects of varying levels of abstraction.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I... am incredibly opposed to this line. 😂

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tuxxy What do you suggest?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know, I just don't agree that a character is accurately described in this line lol.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eh? What's inaccurate here?

assert result

yield deferToThread(start_lonely_learning_loop)
# def start_lonely_learning_loop():
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lines can be removed?

@@ -1,6 +1,4 @@
version: 2.1
orbs:
codecov: codecov/codecov@1.0.5
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we getting rid of codecov altogether? Os was this just a temporary measure because it was crazy the other day?

Base class for Nucypher protocol actors.
A participant in the cryptological drama (a screenplay, if you like) of NuCypher.

Characters can represent users, nodes, wallets, offline devices, or other objects of varying levels of abstraction.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tuxxy What do you suggest?

else:
raise InvalidSignature("Signature for message isn't valid: {}".format(signature_to_use))
except (TypeError, AttributeError) as e:
raise InvalidSignature(f"Unable to verify message from stranger: {stranger}")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This block is related to #2203. It's up to you if it requires adding a comment/TODO to signal that.

target_nodes = []
target_hex_match = self.public_keys(DecryptingPower).hex()[1]
while len(target_nodes) < no_less_than:
target_nodes = []
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
target_nodes = []

This line has no effect.

# Not enough matching nodes. Fine, we'll just publish to the first few.
try:
# TODO: This is almost certainly happening in a test. If it does happen in production, it's a bit of a problem. Need to fix #2124 to mitigate.
target_nodes = list(nodes._nodes.values())[0:6]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you want to return target_nodes here, right?

# The good news is that Bob can construct the list easily.
# And - famous last words incoming - there's no cognizable attack surface.
# Sure, Bob can mine encrypting keypairs until he gets the set of target Ursulas on which Alice can
# store a TreasureMap. And then... ???... profit?
Copy link
Member

@cygnusv cygnusv Sep 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This algorithm is kind of quirky and I'm not sure it works as the network grows. As far as I understand, the whole point is that Bob recovers roughly the same list of target nodes that Alice selected to store the TMap, correct?

Let's imagine a scenario where initially we have very few nodes. Alice creates a TMap, and since there are few nodes, the algorithm is forced to expand the search boundary a few times. @jMyles plays the Song of Time and we move 1 year to the future, where we have a bigger network; now Bob tries to retrieve the TMap, and since we have more nodes, there's greater probability that there are matches with small search boundaries. This has the effect that some of the initial target nodes are not discovered by Bob.

I don't think however this is a deal-breaker, but perhaps we should open an issue.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes! This is why I questioned the minimization of #2124 in the call yesterday.

Copy link
Member

@cygnusv cygnusv Sep 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Glad that you already had an issue tracking this :) Let's add a reference here then:

Suggested change
# store a TreasureMap. And then... ???... profit?
# store a TreasureMap. And then... ???... profit?
# TODO: There's some risk that Bob can't get any of the original target Ursulas - See #2124

work_tracker=work_tracker,
start_working_now=start_working_now,
block_until_ready=block_until_ready)
except (Exception, self.WorkerError): # FIXME
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about this Exception here?

Copy link
Member

@cygnusv cygnusv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jMyles! I think we can almost touch the peninsula!


if claim_signing_key:
crypto_power.consume_power_up(SigningPower(pubkey=target_ursula.stamp.as_umbral_pubkey()))
crypto_power.consume_power_up(SigningPower(public_key=target_ursula.stamp.as_umbral_pubkey()))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. This kwarg was renamed to public_key in June 2019 (8021c71), so this means this line's not tested.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're the new codecov, @cygnusv.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess you can add to the PR description that you're removing codecov from CI and offshoring coverage to sunny Spain, instead.

On second thoughts, let's drop the codecov yanking comment...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drop the commit you mean? I'm not sure what to do to fix it. Is codecov serving us? It just gives everything a red x and clogs up the alerts channel.


learner_deferred = self._learning_task.start(interval=self._SHORT_LEARNING_DELAY, now=now)
learner_deferred = self._learning_task.start(interval=self._SHORT_LEARNING_DELAY,
now=now) # TODO: now=now? This block is always False, no?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why your comment? I see several instances where now=False.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, but this is in an else clause of elif now.

from bytestring_splitter import BytestringSplitter, VariableLengthBytestring, BytestringSplittingError, \
BytestringKwargifier
from constant_sorrow.constants import CFRAG_NOT_RETAINED, NO_DECRYPTION_PERFORMED, NOT_SIGNED
from constant_sorrow.constants import CFRAG_NOT_RETAINED
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
from constant_sorrow.constants import CFRAG_NOT_RETAINED

from nucypher.characters.lawful import Bob, Character
from nucypher.crypto.api import keccak_digest, encrypt_and_sign, verify_eip_191
from nucypher.crypto.constants import PUBLIC_ADDRESS_LENGTH, KECCAK_DIGEST_LENGTH
from nucypher.crypto.api import encrypt_and_sign, keccak_digest
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
from nucypher.crypto.api import encrypt_and_sign, keccak_digest

from nucypher.crypto.powers import TransactingPower
from tests.constants import FEE_RATE_RANGE, INSECURE_DEVELOPMENT_PASSWORD
from tests.utils.ursula import make_decentralized_ursulas
from nucypher.crypto.powers import TransactingPower
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
from nucypher.crypto.powers import TransactingPower

@jMyles jMyles force-pushed the †reasure-ïsland-@ss-pi®ate-ß̆çh branch from 7285427 to 546c41d Compare September 7, 2020 06:24
@KPrasch KPrasch requested a review from mswilkison September 7, 2020 20:52
Copy link
Contributor

@tuxxy tuxxy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work @jMyles 🐧

@KPrasch KPrasch merged commit 704a361 into main Sep 8, 2020
@KPrasch KPrasch mentioned this pull request Sep 8, 2020
@KPrasch KPrasch deleted the †reasure-ïsland-@ss-pi®ate-ß̆çh branch January 22, 2021 19:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Epic 📖 Upstream branches with a project focus

Projects

None yet

8 participants