-
Notifications
You must be signed in to change notification settings - Fork 38.7k
descriptor: Move filling of keys from DescriptorImpl::MakeScripts to PubkeyProvider::GetPubKey
#31243
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code Coverage & BenchmarksFor details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31243. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
6b91f7d to
f50470e
Compare
theStack
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code-review ACK f50470e33c1bb51531f3fa08f6d29c54df304d6b
To my understanding, the second and fourth commits (db270fa2a33d7c88e58682ab8a219a2e70503c9e and f50470e33c1bb51531f3fa08f6d29c54df304d6b) are pure refactors, while the third one actually changes behaviour for Descriptor::Expand call-sites w.r.t. by returning more public key data than before (stored in the FlatSigningProvider& out parameter), so the first commit (d1be90704559da45e4d76171230ecb9f4a6d9cf8) is needed to avoid breaking the importmulti RPC call behaviour. Without that, the "Imported scripts with pubkeys should not have their pubkeys go into the keypool" sub-test in wallet_importmulti.py fails.
f50470e to
186bc75
Compare
theStack
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re-ACK 186bc75985a0afa96eba3c5bfea09e22fe3ec2c8 🗒️
fjahr
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Apparently I started to review and got pinged now because of the rebase message. Leaving these nits here in case you want to address them in the rebase. Will look at this again soon.
src/wallet/rpc/backup.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Could add a comment here as well, something like
| bool can_keypool = parsed_descs.at(0)->IsSingleKey(); | |
| // Only single key descriptors are allowed to be imported into a legacy wallet's keypool | |
| bool can_keypool = parsed_descs.at(0)->IsSingleKey(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This^ has been resolved now.
src/script/descriptor.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Typo in 7c614b15bc31a7d6f7daaa48c046eb2c7cca697e description “or now”
186bc75 to
1bfabb4
Compare
Sjors
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Concept ACK
Mostly happy with 1bfabb4, but I'm confused about the key origin handling.
| /** Derive a private key, if private data is available in arg. */ | ||
| virtual bool GetPrivKey(int pos, const SigningProvider& arg, CKey& key) const = 0; | ||
| /** Derive a private key, if private data is available in arg and put it into out. */ | ||
| virtual void GetPrivKey(int pos, const SigningProvider& arg, FlatSigningProvider& out) const = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1bfabb4: maybe rename to FillPrivKey?
/** Try to derive the private key at pos, if private data is available in arg, and fill out with it. */
virtual void FillPrivKey(int pos, const SigningProvider& arg, FlatSigningProvider& out) const = 0;That also distinguishes it from ConstPubkeyProvider::GetPrivKey.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The verb "fill" suggests to me that the caller is filling the descriptor with a private key, rather than the descriptor filling something else. I think "Get" is still appropriate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am inclining to agree with @achow101 here. Although I don't like a function that is prefixed with Get returning nothing but at the moment it's a common pattern in the codebase to Get something and update a reference passed in one of the function arguments. The confusion, as highlighted above, that might arise with prefixing with Fill makes me not to want to use that.
| * Caches are not exclusive but this is not tested. Currently we use them exclusively | ||
| */ | ||
| virtual bool GetPubKey(int pos, const SigningProvider& arg, CPubKey& key, KeyOriginInfo& info, const DescriptorCache* read_cache = nullptr, DescriptorCache* write_cache = nullptr) const = 0; | ||
| virtual std::optional<CPubKey> GetPubKey(int pos, const SigningProvider& arg, FlatSigningProvider& out, const DescriptorCache* read_cache = nullptr, DescriptorCache* write_cache = nullptr) const = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
e743d11: maybe rename this to FillPubKey. The filling seems more important, and the return value makes it clear enough it also gets it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the function doc:
-/** Derive a public key.
+/** Derive a public key and put it into `out`There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added the comment.
src/script/descriptor.cpp
Outdated
| std::optional<CPubKey> pub = m_provider->GetPubKey(pos, arg, out, read_cache, write_cache); | ||
| if (!pub) return std::nullopt; | ||
| auto& [pubkey, suborigin] = out.origins[pub->GetID()]; | ||
| Assert(pubkey == *pub); // All subproviders must be inserting a valid origin already |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
e743d11: I'm confused by this comment. I guess by "valid" you mean that a KeyOriginInfo field is present, but it can be a dummy?
It seems more intuitive if we insert an origin entry ourselves here ... if possible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Valid means that the origin is correct. It may not be useful, e.g. the origin of a random key is itself, but it's valid and correct.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Concept ACK
Everything besides my GetPrivKey question looks good to me. Will do a final pass once I get an answer there.
| CPubKey m_pubkey; | ||
| bool m_xonly; | ||
|
|
||
| std::optional<CKey> GetPrivKey(const SigningProvider& arg) const |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm, I am not seeing why having two GetPrivKeys is better than one. The description of the commit only talks about the other one, not this one. For this one the caller still cares about the return value it seems. Can't they be unified under the std::optional returning variant? It seems like the variant using FlatSigningProvider is just used in one place in ExpandPrivate and the whole benefit is saving us one line there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The benefit is in the MuSig2 PRs where GetPrivKey can insert multiple private keys to the FlatSigningProvider, none of which are the private key for the MuSig2 aggregate pubkey itself.
For the ConstPubkeyProvider, it's useful to have another GetPrivKey that returns the single private key since this is behavior is used in a few places within this class's member functions. None of the other pubkey providers need this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Understood. I think I would have chosen a different name for the one in ConstPubkeyProvider, like GetConstPrivKey maybe, but completely optional.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The benefit is in the MuSig2 PRs where GetPrivKey can insert multiple private keys to the FlatSigningProvider, none of which are the private key for the MuSig2 aggregate pubkey itself.
Something around this can be mentioned either in the commit message and/or the PR description.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
like GetConstPrivKey maybe
This is already inside ConstPubkeyProvider class, using Const again could feel repetitive. I think the function overloading (with different argument types) is differentiating enough. Though, some documentation for the new function would be helpful for the reader.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Something around this can be mentioned either in the commit message and/or the PR description.
Added to the commit message.
theStack
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re-ACK 1bfabb4
|
utACK 1bfabb4 |
rkrux
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Few initial comments, I would like to spend more time on this PR to complete the review.
| * Caches are not exclusive but this is not tested. Currently we use them exclusively | ||
| */ | ||
| virtual bool GetPubKey(int pos, const SigningProvider& arg, CPubKey& key, KeyOriginInfo& info, const DescriptorCache* read_cache = nullptr, DescriptorCache* write_cache = nullptr) const = 0; | ||
| virtual std::optional<CPubKey> GetPubKey(int pos, const SigningProvider& arg, FlatSigningProvider& out, const DescriptorCache* read_cache = nullptr, DescriptorCache* write_cache = nullptr) const = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the function doc:
-/** Derive a public key.
+/** Derive a public key and put it into `out`| CPubKey m_pubkey; | ||
| bool m_xonly; | ||
|
|
||
| std::optional<CKey> GetPrivKey(const SigningProvider& arg) const |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The benefit is in the MuSig2 PRs where GetPrivKey can insert multiple private keys to the FlatSigningProvider, none of which are the private key for the MuSig2 aggregate pubkey itself.
Something around this can be mentioned either in the commit message and/or the PR description.
src/script/descriptor.cpp
Outdated
| out.origins.emplace(keyid, std::make_pair(m_pubkey, info)); | ||
| out.pubkeys.emplace(keyid, m_pubkey); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with filling out with pubkeys and origins together in a function.
4e0de38 to
72694b8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Few comments, mostly LGTM.
src/script/descriptor.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This comment is vaguely correct now, technically ok but doesn't gel well with the new object name, causes confusion. IMO best to update it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed it, I don't think there's anything meaningfully useful to say in a comment here.
src/script/descriptor.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall the update count (in this file atleast) of the members of the class FlatSigningProvider& out has decreased now because there are fewer PubkeyProviders and more DescrImpls , which IMO is desirable.
src/script/descriptor.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Coming from previous commit 070e746c2f72db2cc24b1c267b91fafbcc586736 and looking at this function as a whole, there's a lot going on.
Nit: A comment here would be quite helpful for the reader letting them know that origins & pubkeys are being appended.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think a comment is necessary, emplace() is fairly self explanatory.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I realise my suggestion was not clear enough (and confusing). A comment can be present in the documentation of the function declaration instead - I have mentioned in another review comment: #31243 (comment)
src/script/descriptor.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 for removing the usage of entries.back().first, entries.back().second as the output parameters to the GetPubKey(), which was not intuitive to read for me.
72694b8 to
2f28f28
Compare
Legacy wallets should only import keys to the keypool if they came in a single key descriptor. Instead of relying on assumptions about the descriptor based on how many pubkeys show up after expanding the descriptor, explicitly mark descriptors as being single key type and use that for the check.
Instead of having ExpandHelper fill in the origins in the FlatSigningProvider output, have GetPubKey do it by itself. This reduces the extra variables needed in order to track and set origins in ExpandHelper. Also changes GetPubKey to return a std::optional<CPubKey> rather than using a bool and output parameters.
Instead of MakeScripts inconsistently filling the output FlatSigningProvider with the pubkeys involved, just do it in GetPubKey.
Instead of GetPrivKey returning a key and having the caller fill the FlatSigningProvider, have GetPrivKey take the FlatSigningProvider and fill it by itself. This will be necessary for descriptors such as musig() where there are private keys that need to be added to the FlatSigningProvider but do not directly appear in any resulting scripts. GetPrivKey is now changed to void as the caller no longer cares whether it succeeds or fails.
2f28f28 to
acee5c5
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, code review acee5c5
Thank you for splitting the middle commit, makes it easier to review!
Broadly this pull updates the private & public key getters in the PubkeyProviders to also fill the passed SigningProvider out with the said keys; the effect of which is that it cleans the implementation of the call sites such as ExpandHelper in the Descriptors classes. This would result in more (than before) keys being set in the SigningProviders generally and the direct callers of these functions would need to accordingly adjust. I'm inclining to prefer the filling of the SigningProviders with the public key info (origin & key) together instead of having them filled in various functions like it was done previously.
There are few other minor improvements as well like highlighting the non-usage of the arguments in the function signature itself. I've started to look at the #29675 PR to find where the updated functionality of GetPrivKey is used as stated in the commit description.
Before I ack the PR, asked one question related to the first commit - #31243 (comment).
src/wallet/rpc/backup.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This^ has been resolved now.
| return true; | ||
| } | ||
| bool GetPrivKey(int pos, const SigningProvider& arg, CKey& key) const override | ||
| void GetPrivKey(int pos, const SigningProvider& arg, FlatSigningProvider& out) const override |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: Because pos is unused.
- void GetPrivKey(int pos, const SigningProvider& arg, FlatSigningProvider& out) const override
+ void GetPrivKey(int, const SigningProvider& arg, FlatSigningProvider& out) const override
| } | ||
|
|
||
| /** Derive a public key. | ||
| /** Derive a public key and put it into out. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit:
- /** Derive a public key and put it into out.
+ /** Derive a public key and put it into out along with its origin info.
src/script/descriptor.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I realise my suggestion was not clear enough (and confusing). A comment can be present in the documentation of the function declaration instead - I have mentioned in another review comment: #31243 (comment)
| std::copy(scripts_temp.begin(), scripts_temp.end(), std::inserter(script_pub_keys, script_pub_keys.end())); | ||
| for (const auto& key_pair : out_keys.pubkeys) { | ||
| ordered_pubkeys.emplace_back(key_pair.first, desc_internal); | ||
| if (can_keypool) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm looking at it after a month and have lost some context here.
What restriction in the previous code was disallowing the addition of public keys in the keypool for non-single-key type descriptors that made the "Imported scripts with pubkeys should not have their pubkeys go into the keypool" subtest pass? Sorry if this has been answered earlier but I didn't find it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Prior to this PR, the multisig descriptors would not insert any pubkeys int out.pubkeys inside of MakeScripts. The pubkeys that are inserted to that FlatSigningProvider are the ones inserted into the keypool.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see, they were being added for pkh, wpkh, combo, tr, miniscript ones in MakeScript but not for multisig & the like. I debugged this flow in on master while importing a multisig descriptor and found it quite strange that the origins of all the pubkeys involved in the multisig were being added in the out.origins inside ExpandHelper but not all the pubkeys were added in out.pubkeys!
Also, pubkeys being set inside MakeScript but origins being set in ExpandHelper made it difficult to reason about for me, the newer implementation makes it easier to follow.
bitcoinclireg importdescriptors '[{"desc": "wsh(multi(2,tprv8ZgxMBicQKsPdzuc344mDaeUk5zseMcRK9Hst8xodskNu3YbQG5NxLa2X17PUU5yXQhptiBE7F5W5cgEmsfQg4Y21Y18w4DJhLxSb8CurDf,tpubD6NzVbkrYhZ4YiCvExLvH4yh1k3jFGf5irm6TsrArY8GYdEhYVdztQTBtTirmRc6XfSJpH9tayUdnngaJZKDaa2zbqEY29DfcGZW8iRVGUY))#a0tl2jdn", "timestamp": "now"}]'
I used importdescriptors but the Expand flow is the same as in importmulti.
| /** Derive a private key, if private data is available in arg. */ | ||
| virtual bool GetPrivKey(int pos, const SigningProvider& arg, CKey& key) const = 0; | ||
| /** Derive a private key, if private data is available in arg and put it into out. */ | ||
| virtual void GetPrivKey(int pos, const SigningProvider& arg, FlatSigningProvider& out) const = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not a suggestion, just thinking out loud.
For the sake of having a function signature consistent with the corresponding public key function - I suppose returning the private key from this function is not an option because we'd prefer to have fewer instances of private keys moving around within the codebase?
Though I'm wondering can there be an use case later where the caller would need to know whether the private key was gotten in which case the boolean return value is needed; seems a bit odd that the caller would need to find the key in the out every time even if they don't want to use the key.
fjahr
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code review ACK acee5c5
| /** Compare two public keys represented by this provider. | ||
| * Used by the Miniscript descriptors to check for duplicate keys in the script. | ||
| */ | ||
| bool operator<(PubkeyProvider& other) const { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: could make other const if you retouch
Was that an ACK an you just forgot the word or did you just want to say you reviewed? :) |
theStack
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK acee5c5
Was that an ACK an you just forgot the word or did you just want to say you reviewed? :)
I wanted to convey I code reviewed & almost ack. There was one thing that I had not completely reasoned about though it made sense to me intuitively somehow, so was hesitant in acking earlier: #31243 (comment)
| std::copy(scripts_temp.begin(), scripts_temp.end(), std::inserter(script_pub_keys, script_pub_keys.end())); | ||
| for (const auto& key_pair : out_keys.pubkeys) { | ||
| ordered_pubkeys.emplace_back(key_pair.first, desc_internal); | ||
| if (can_keypool) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see, they were being added for pkh, wpkh, combo, tr, miniscript ones in MakeScript but not for multisig & the like. I debugged this flow in on master while importing a multisig descriptor and found it quite strange that the origins of all the pubkeys involved in the multisig were being added in the out.origins inside ExpandHelper but not all the pubkeys were added in out.pubkeys!
Also, pubkeys being set inside MakeScript but origins being set in ExpandHelper made it difficult to reason about for me, the newer implementation makes it easier to follow.
bitcoinclireg importdescriptors '[{"desc": "wsh(multi(2,tprv8ZgxMBicQKsPdzuc344mDaeUk5zseMcRK9Hst8xodskNu3YbQG5NxLa2X17PUU5yXQhptiBE7F5W5cgEmsfQg4Y21Y18w4DJhLxSb8CurDf,tpubD6NzVbkrYhZ4YiCvExLvH4yh1k3jFGf5irm6TsrArY8GYdEhYVdztQTBtTirmRc6XfSJpH9tayUdnngaJZKDaa2zbqEY29DfcGZW8iRVGUY))#a0tl2jdn", "timestamp": "now"}]'
I used importdescriptors but the Expand flow is the same as in importmulti.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably in some follow up: The documentation of the MakeScripts function in DescriptorImpl class can be updated to get rid of the following line because the origin info is added in the GetPubKey functions now and was added in the ExpandHelper function previously.
The origin info of the provided pubkeys is automatically added.
Instead of having
MakeScriptsinfer what pubkeys need to go into the outputFlatSigningProvider, have each of thePubkeyProvidersthat haveGetPubKeyandGetPrivKeycalled fill it directly with relevant keys and origins.This allows for keys and origins to be added that won't directly appear in the output, which is necessary for
musig()descriptors.Split from #29675