Backup and restore Ursula config by piotr-roslaniec · Pull Request #2682 · nucypher/nucypher

piotr-roslaniec · 2021-05-03T15:17:44Z

Type of PR:

Bugfix
Feature
Documentation
Other

Required reviews:

1
2
3

What this does:

Add CLI commands for Ursula backup, ursula backup and ursula restore.
Backups are encrypted ZIP files protected by a password.
Both commands have the same flags, but the actual behavior depends on the command used. For example, ursula backup --worker-path would denote file sources used to create a backup file. On the other hand, ursula restore --worker-path denotes the destination where unpacked backup files will be stored.

$ nucypher ursula backup \
   --worker-path <path> \ # set to default 
   --keystore-path <path> \   # set to default
   --backup-path <path> \ # required
   --password <str> \ # required
   --overwrite <bool> \ # optional

$ nucypher ursula restore \
   --worker-path <path> \ # set to default 
   --keystore-path <path> \   # set to default
   --backup-path <path> \ # required
   --password <str> \ # required
   --overwrite <bool> \ # optional

Backup archive has a following structure:

$ tree -L 3 restored
restored
├── keystore
│   └── keystore.json
└── worker
    └── nucypher
        ├── cards
        ├── keyring
        ├── known_nodes
        ├── ursula.db
        └── ursula.json

Issues fixed/closed:

Why it's needed:

Explain how this PR fits in the greater context of the NuCypher Network.
E.g., if this PR address a nucypher/productdev issue, let reviewers know!

Notes for reviewers:

The Add functionality to Ursula to backup/restore her data and respective keys #1992 issue mentions another command based on magic-wormhole. I can implement it in another PR.
CI is failing because of a certificate issue

KPrasch · 2021-05-14T19:08:11Z

If there are multiple ursulas on a system, how does the backup/restore functionality determine which database to use? Are we making this distinction at all, or are we inadvertently reusing a single DB for all ursulas on one host?

piotr-roslaniec · 2021-05-17T08:10:00Z

@KPrasch There is no distinction made between databases, backups are made only based on the path to the worker directory. But this is something that can be improved by reading db_filepath from the Ursula config file, and rewriting it accordingly after backup is restored.

cygnusv · 2021-05-24T15:37:24Z

requirements.txt

 pychalk==2.0.1
 pycparser==2.20; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
 pycryptodome==3.9.9
+pycryptodomex==3.10.1


I'm wondering if there can be any implication of having both pycryptodome and pycryptodomex. It seems the x version was released to avoid interference with the pycrypto package, which we don't use anyway (https://www.pycryptodome.org/en/latest/src/introduction.html?highlight=pycryptodomex#pycryptodome). It seems in our case we can potentially use both, but probably it's a dependency issue?

We use both of those packages as indirect dependencies of eth-hash and pyzipper. AFAIK there should be no issues with dependency shadowing, even if we start using them directly, as they have different namespaces (base package names).

nucypher/cli/commands/ursula.py

derekpierre · 2021-05-31T16:11:34Z

We'll also need to add some documentation.

derekpierre

Good stuff - just some minor documentation edits

docs/source/staking/running_a_worker.rst

derekpierre · 2021-06-01T13:54:31Z

nucypher/cli/commands/ursula.py

+    with pyzipper.AESZipFile(backup_options.backup_path) as zf:
+        zf.setpassword(backup_options.password)
+        zf.setencryption(**BACKUP_ENCRYPTION_SETTINGS)
+        _extract_zf(emitter, zf, KEYSTORE_ARCHIVE_ROOT, backup_options.keystore_path, backup_options.overwrite)


👍 - "extract" looks good there.

nucypher/cli/commands/ursula.py

docs/source/staking/running_a_worker.rst

Co-authored-by: Derek Pierre <derek.pierre@gmail.com>

nucypher/cli/commands/ursula.py

derekpierre · 2021-06-02T13:38:24Z

nucypher/cli/commands/ursula.py

+def _prompt_overwrite(path: Path, force: bool):
+    if path.exists():
+        prompt = f"Path already exists: {path.absolute()}"
+        if force:


docs/source/staking/running_a_worker.rst

derekpierre

Having some issues with restore and pre-existing folders.

derek @ Dereks-MacBook-Pro-2 ~/Documents/Github/repos/forks/piotr/nucypher
 [37] → nucypher ursula backup --password llamallama --force
WARNING: Force is enabled
Path already exists: /Users/derek/Documents/Github/repos/forks/piotr/nucypher/ursula-backup.zip - Overwriting.
Wrote /Users/derek/Library/Application Support/nucypher
Wrote /Users/derek/Library/Ethereum/keystore
Backup created at /Users/derek/Documents/Github/repos/forks/piotr/nucypher/ursula-backup.zip


(nucypher) (base)
derek @ Dereks-MacBook-Pro-2 ~/Documents/Github/repos/forks/piotr/nucypher
 [38] → ls -l worker keystore
keystore:
total 0
drwxr-xr-x  3 derek  staff  96  3 Jun 09:29 keystore

worker:
total 0
drwxr-xr-x  3 derek  staff  96  3 Jun 09:23 worker


(nucypher) (base)
derek @ Dereks-MacBook-Pro-2 ~/Documents/Github/repos/forks/piotr/nucypher
 [39] → nucypher ursula restore --password llamallama --worker-path ./worker --keystore-path ./keystore/ --backup-path ./ursula-backup.zip
Path already exists: /Users/derek/Documents/Github/repos/forks/piotr/nucypher/keystore - Overwrite? [y/N]: y
Traceback (most recent call last):
  File "/Users/derek/.local/share/virtualenvs/nucypher-GD4zeyIP/bin/nucypher", line 33, in <module>
    sys.exit(load_entry_point('nucypher', 'console_scripts', 'nucypher')())
  File "/Users/derek/.local/share/virtualenvs/nucypher-GD4zeyIP/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/derek/.local/share/virtualenvs/nucypher-GD4zeyIP/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/derek/.local/share/virtualenvs/nucypher-GD4zeyIP/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/derek/.local/share/virtualenvs/nucypher-GD4zeyIP/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/derek/.local/share/virtualenvs/nucypher-GD4zeyIP/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/derek/.local/share/virtualenvs/nucypher-GD4zeyIP/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/derek/Documents/Github/repos/forks/piotr/nucypher/nucypher/cli/options.py", line 169, in wrapper
    return func(**kwargs)
  File "/Users/derek/Documents/Github/repos/forks/piotr/nucypher/nucypher/cli/options.py", line 169, in wrapper
    return func(**kwargs)
  File "/Users/derek/Documents/Github/repos/forks/piotr/nucypher/nucypher/cli/commands/ursula.py", line 600, in restore
    _extract_zf(emitter, zf, KEYSTORE_ARCHIVE_ROOT, backup_options.keystore_path, backup_options.force)
  File "/Users/derek/Documents/Github/repos/forks/piotr/nucypher/nucypher/cli/commands/ursula.py", line 616, in _extract_zf
    shutil.move(str(archive_path.absolute()), str(destination_path.absolute()))
  File "/Users/derek/.local/share/virtualenvs/nucypher-GD4zeyIP/lib/python3.7/shutil.py", line 564, in move
    raise Error("Destination path '%s' already exists" % real_dst)
shutil.Error: Destination path '/Users/derek/Documents/Github/repos/forks/piotr/nucypher/keystore/keystore' already exists

nucypher/cli/commands/ursula.py

KPrasch

Looking great overall. If we're going to support nucypher ursula restore I think we need to support restoration from mnemonic phrase as well as file-based backups.

piotr-roslaniec added 4 commits May 3, 2021 16:59

Backup and restore Ursula config

0dbeb41

Added a newsfragment

2efa1f9

Test ursula backup

8bc94dd

Test keystore backup

7cd500b

piotr-roslaniec changed the title ~~[WIP] Backup and restore Ursula config~~ Backup and restore Ursula config May 6, 2021

piotr-roslaniec marked this pull request as ready for review May 6, 2021 08:46

piotr-roslaniec requested review from KPrasch, cygnusv, derekpierre and tuxxy May 6, 2021 08:46

cygnusv reviewed May 24, 2021

View reviewed changes

Update default keystore path

02286ad

derekpierre reviewed May 31, 2021

View reviewed changes

nucypher/cli/commands/ursula.py Outdated Show resolved Hide resolved

nucypher/cli/commands/ursula.py Outdated Show resolved Hide resolved

nucypher/cli/commands/ursula.py Outdated Show resolved Hide resolved

piotr-roslaniec added 2 commits June 1, 2021 09:25

Apply PR suggestions

3a131f8

Add backup docs

ee8c6ec

piotr-roslaniec force-pushed the ursula-backup#1992 branch from f4d3ff2 to ee8c6ec Compare June 1, 2021 09:02

derekpierre reviewed Jun 1, 2021

View reviewed changes

nucypher/cli/commands/ursula.py Show resolved Hide resolved

nucypher/cli/commands/ursula.py Show resolved Hide resolved

docs/source/staking/running_a_worker.rst Outdated Show resolved Hide resolved

piotr-roslaniec and others added 2 commits June 1, 2021 16:52

Apply suggestions from code review

e440dd9

Co-authored-by: Derek Pierre <derek.pierre@gmail.com>

Add force flag

df91bdc

piotr-roslaniec force-pushed the ursula-backup#1992 branch from 261ce7b to df91bdc Compare June 2, 2021 09:32

derekpierre reviewed Jun 2, 2021

View reviewed changes

Apply PR suggestions

a60b8d7

derekpierre reviewed Jun 3, 2021

View reviewed changes

nucypher/cli/commands/ursula.py Show resolved Hide resolved

nucypher/cli/commands/ursula.py Show resolved Hide resolved

KPrasch mentioned this pull request Jun 16, 2021

Deterministic Keystore #2701

Merged

KPrasch reviewed Aug 23, 2021

View reviewed changes

piotr-roslaniec closed this Sep 6, 2021

piotr-roslaniec deleted the ursula-backup#1992 branch September 6, 2021 20:32

Conversation

piotr-roslaniec commented May 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KPrasch commented May 14, 2021

Uh oh!

piotr-roslaniec commented May 17, 2021

Uh oh!

cygnusv May 24, 2021

Choose a reason for hiding this comment

Uh oh!

piotr-roslaniec May 27, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

derekpierre commented May 31, 2021

Uh oh!

derekpierre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

derekpierre Jun 1, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

derekpierre Jun 2, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

derekpierre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

KPrasch left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

piotr-roslaniec commented May 3, 2021 •

edited

Loading

KPrasch left a comment •

edited

Loading