Skip to content

Conversation

@achow101
Copy link
Member

@achow101 achow101 commented Nov 7, 2024

Instead of having MakeScripts infer what pubkeys need to go into the output FlatSigningProvider, have each of the PubkeyProviders that have GetPubKey and GetPrivKey called fill it directly with relevant keys and origins.

This allows for keys and origins to be added that won't directly appear in the output, which is necessary for musig() descriptors.

Split from #29675

@DrahtBot
Copy link
Contributor

DrahtBot commented Nov 7, 2024

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31243.

Reviews

See the guideline for information on the review process.

Type Reviewers
ACK fjahr, theStack, rkrux
Concept ACK Sjors

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #31244 (descriptors: MuSig2 by achow101)
  • #29675 (wallet: Be able to receive and spend inputs involving MuSig2 aggregate keys by achow101)
  • #28710 (Remove the legacy wallet and BDB dependency by achow101)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Copy link
Contributor

@theStack theStack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code-review ACK f50470e33c1bb51531f3fa08f6d29c54df304d6b

To my understanding, the second and fourth commits (db270fa2a33d7c88e58682ab8a219a2e70503c9e and f50470e33c1bb51531f3fa08f6d29c54df304d6b) are pure refactors, while the third one actually changes behaviour for Descriptor::Expand call-sites w.r.t. by returning more public key data than before (stored in the FlatSigningProvider& out parameter), so the first commit (d1be90704559da45e4d76171230ecb9f4a6d9cf8) is needed to avoid breaking the importmulti RPC call behaviour. Without that, the "Imported scripts with pubkeys should not have their pubkeys go into the keypool" sub-test in wallet_importmulti.py fails.

@achow101 achow101 force-pushed the descriptor-direct-fill branch from f50470e to 186bc75 Compare November 19, 2024 17:03
Copy link
Contributor

@theStack theStack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-ACK 186bc75985a0afa96eba3c5bfea09e22fe3ec2c8 🗒️

Copy link
Contributor

@fjahr fjahr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently I started to review and got pinged now because of the rebase message. Leaving these nits here in case you want to address them in the rebase. Will look at this again soon.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Could add a comment here as well, something like

Suggested change
bool can_keypool = parsed_descs.at(0)->IsSingleKey();
// Only single key descriptors are allowed to be imported into a legacy wallet's keypool
bool can_keypool = parsed_descs.at(0)->IsSingleKey();

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This^ has been resolved now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Typo in 7c614b15bc31a7d6f7daaa48c046eb2c7cca697e description “or now”

@achow101 achow101 force-pushed the descriptor-direct-fill branch from 186bc75 to 1bfabb4 Compare January 21, 2025 20:15
Copy link
Member

@Sjors Sjors left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK

Mostly happy with 1bfabb4, but I'm confused about the key origin handling.

/** Derive a private key, if private data is available in arg. */
virtual bool GetPrivKey(int pos, const SigningProvider& arg, CKey& key) const = 0;
/** Derive a private key, if private data is available in arg and put it into out. */
virtual void GetPrivKey(int pos, const SigningProvider& arg, FlatSigningProvider& out) const = 0;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1bfabb4: maybe rename to FillPrivKey?

/** Try to derive the private key at pos, if private data is available in arg, and fill out with it. */
virtual void FillPrivKey(int pos, const SigningProvider& arg, FlatSigningProvider& out) const = 0;

That also distinguishes it from ConstPubkeyProvider::GetPrivKey.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The verb "fill" suggests to me that the caller is filling the descriptor with a private key, rather than the descriptor filling something else. I think "Get" is still appropriate.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am inclining to agree with @achow101 here. Although I don't like a function that is prefixed with Get returning nothing but at the moment it's a common pattern in the codebase to Get something and update a reference passed in one of the function arguments. The confusion, as highlighted above, that might arise with prefixing with Fill makes me not to want to use that.

* Caches are not exclusive but this is not tested. Currently we use them exclusively
*/
virtual bool GetPubKey(int pos, const SigningProvider& arg, CPubKey& key, KeyOriginInfo& info, const DescriptorCache* read_cache = nullptr, DescriptorCache* write_cache = nullptr) const = 0;
virtual std::optional<CPubKey> GetPubKey(int pos, const SigningProvider& arg, FlatSigningProvider& out, const DescriptorCache* read_cache = nullptr, DescriptorCache* write_cache = nullptr) const = 0;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

e743d11: maybe rename this to FillPubKey. The filling seems more important, and the return value makes it clear enough it also gets it.

Copy link
Contributor

@rkrux rkrux Mar 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the function doc:

-/** Derive a public key.
+/** Derive a public key and put it into `out`

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the comment.

std::optional<CPubKey> pub = m_provider->GetPubKey(pos, arg, out, read_cache, write_cache);
if (!pub) return std::nullopt;
auto& [pubkey, suborigin] = out.origins[pub->GetID()];
Assert(pubkey == *pub); // All subproviders must be inserting a valid origin already
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

e743d11: I'm confused by this comment. I guess by "valid" you mean that a KeyOriginInfo field is present, but it can be a dummy?

It seems more intuitive if we insert an origin entry ourselves here ... if possible.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid means that the origin is correct. It may not be useful, e.g. the origin of a random key is itself, but it's valid and correct.

Copy link
Contributor

@fjahr fjahr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK

Everything besides my GetPrivKey question looks good to me. Will do a final pass once I get an answer there.

CPubKey m_pubkey;
bool m_xonly;

std::optional<CKey> GetPrivKey(const SigningProvider& arg) const
Copy link
Contributor

@fjahr fjahr Feb 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I am not seeing why having two GetPrivKeys is better than one. The description of the commit only talks about the other one, not this one. For this one the caller still cares about the return value it seems. Can't they be unified under the std::optional returning variant? It seems like the variant using FlatSigningProvider is just used in one place in ExpandPrivate and the whole benefit is saving us one line there.

Copy link
Member Author

@achow101 achow101 Feb 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The benefit is in the MuSig2 PRs where GetPrivKey can insert multiple private keys to the FlatSigningProvider, none of which are the private key for the MuSig2 aggregate pubkey itself.

For the ConstPubkeyProvider, it's useful to have another GetPrivKey that returns the single private key since this is behavior is used in a few places within this class's member functions. None of the other pubkey providers need this.

Copy link
Contributor

@fjahr fjahr Mar 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood. I think I would have chosen a different name for the one in ConstPubkeyProvider, like GetConstPrivKey maybe, but completely optional.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The benefit is in the MuSig2 PRs where GetPrivKey can insert multiple private keys to the FlatSigningProvider, none of which are the private key for the MuSig2 aggregate pubkey itself.

Something around this can be mentioned either in the commit message and/or the PR description.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

like GetConstPrivKey maybe

This is already inside ConstPubkeyProvider class, using Const again could feel repetitive. I think the function overloading (with different argument types) is differentiating enough. Though, some documentation for the new function would be helpful for the reader.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something around this can be mentioned either in the commit message and/or the PR description.

Added to the commit message.

Copy link
Contributor

@theStack theStack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-ACK 1bfabb4

@DrahtBot DrahtBot requested review from Sjors and fjahr February 18, 2025 16:37
@fjahr
Copy link
Contributor

fjahr commented Mar 5, 2025

utACK 1bfabb4

Copy link
Contributor

@rkrux rkrux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few initial comments, I would like to spend more time on this PR to complete the review.

* Caches are not exclusive but this is not tested. Currently we use them exclusively
*/
virtual bool GetPubKey(int pos, const SigningProvider& arg, CPubKey& key, KeyOriginInfo& info, const DescriptorCache* read_cache = nullptr, DescriptorCache* write_cache = nullptr) const = 0;
virtual std::optional<CPubKey> GetPubKey(int pos, const SigningProvider& arg, FlatSigningProvider& out, const DescriptorCache* read_cache = nullptr, DescriptorCache* write_cache = nullptr) const = 0;
Copy link
Contributor

@rkrux rkrux Mar 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the function doc:

-/** Derive a public key.
+/** Derive a public key and put it into `out`

CPubKey m_pubkey;
bool m_xonly;

std::optional<CKey> GetPrivKey(const SigningProvider& arg) const
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The benefit is in the MuSig2 PRs where GetPrivKey can insert multiple private keys to the FlatSigningProvider, none of which are the private key for the MuSig2 aggregate pubkey itself.

Something around this can be mentioned either in the commit message and/or the PR description.

Comment on lines 308 to 317
out.origins.emplace(keyid, std::make_pair(m_pubkey, info));
out.pubkeys.emplace(keyid, m_pubkey);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with filling out with pubkeys and origins together in a function.

@achow101 achow101 force-pushed the descriptor-direct-fill branch 2 times, most recently from 4e0de38 to 72694b8 Compare March 6, 2025 17:28
Copy link
Contributor

@rkrux rkrux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few comments, mostly LGTM.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is vaguely correct now, technically ok but doesn't gel well with the new object name, causes confusion. IMO best to update it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed it, I don't think there's anything meaningfully useful to say in a comment here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall the update count (in this file atleast) of the members of the class FlatSigningProvider& out has decreased now because there are fewer PubkeyProviders and more DescrImpls , which IMO is desirable.

Comment on lines 446 to 453
Copy link
Contributor

@rkrux rkrux Mar 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coming from previous commit 070e746c2f72db2cc24b1c267b91fafbcc586736 and looking at this function as a whole, there's a lot going on.
Nit: A comment here would be quite helpful for the reader letting them know that origins & pubkeys are being appended.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think a comment is necessary, emplace() is fairly self explanatory.

Copy link
Contributor

@rkrux rkrux Apr 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realise my suggestion was not clear enough (and confusing). A comment can be present in the documentation of the function declaration instead - I have mentioned in another review comment: #31243 (comment)

Comment on lines -709 to +716
Copy link
Contributor

@rkrux rkrux Mar 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for removing the usage of entries.back().first, entries.back().second as the output parameters to the GetPubKey(), which was not intuitive to read for me.

Legacy wallets should only import keys to the keypool if they came in a
single key descriptor. Instead of relying on assumptions about the
descriptor based on how many pubkeys show up after expanding the
descriptor, explicitly mark descriptors as being single key type and use
that for the check.
Instead of having ExpandHelper fill in the origins in the
FlatSigningProvider output, have GetPubKey do it by itself. This reduces
the extra variables needed in order to track and set origins in
ExpandHelper.

Also changes GetPubKey to return a std::optional<CPubKey> rather than
using a bool and output parameters.
Instead of MakeScripts inconsistently filling the output
FlatSigningProvider with the pubkeys involved, just do it in GetPubKey.
Instead of GetPrivKey returning a key and having the caller fill the
FlatSigningProvider, have GetPrivKey take the FlatSigningProvider and
fill it by itself. This will be necessary for descriptors such as
musig() where there are private keys that need to be added to the
FlatSigningProvider but do not directly appear in any resulting scripts.

GetPrivKey is now changed to void as the caller no longer cares whether
it succeeds or fails.
@achow101 achow101 force-pushed the descriptor-direct-fill branch from 2f28f28 to acee5c5 Compare April 14, 2025 23:32
Copy link
Contributor

@rkrux rkrux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, code review acee5c5

Thank you for splitting the middle commit, makes it easier to review!

Broadly this pull updates the private & public key getters in the PubkeyProviders to also fill the passed SigningProvider out with the said keys; the effect of which is that it cleans the implementation of the call sites such as ExpandHelper in the Descriptors classes. This would result in more (than before) keys being set in the SigningProviders generally and the direct callers of these functions would need to accordingly adjust. I'm inclining to prefer the filling of the SigningProviders with the public key info (origin & key) together instead of having them filled in various functions like it was done previously.

There are few other minor improvements as well like highlighting the non-usage of the arguments in the function signature itself. I've started to look at the #29675 PR to find where the updated functionality of GetPrivKey is used as stated in the commit description.

Before I ack the PR, asked one question related to the first commit - #31243 (comment).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This^ has been resolved now.

return true;
}
bool GetPrivKey(int pos, const SigningProvider& arg, CKey& key) const override
void GetPrivKey(int pos, const SigningProvider& arg, FlatSigningProvider& out) const override
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Because pos is unused.

- void GetPrivKey(int pos, const SigningProvider& arg, FlatSigningProvider& out) const override
+ void GetPrivKey(int, const SigningProvider& arg, FlatSigningProvider& out) const override

}

/** Derive a public key.
/** Derive a public key and put it into out.
Copy link
Contributor

@rkrux rkrux Apr 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit:

- /** Derive a public key and put it into out.
+ /** Derive a public key and put it into out along with its origin info.

Comment on lines 446 to 453
Copy link
Contributor

@rkrux rkrux Apr 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realise my suggestion was not clear enough (and confusing). A comment can be present in the documentation of the function declaration instead - I have mentioned in another review comment: #31243 (comment)

std::copy(scripts_temp.begin(), scripts_temp.end(), std::inserter(script_pub_keys, script_pub_keys.end()));
for (const auto& key_pair : out_keys.pubkeys) {
ordered_pubkeys.emplace_back(key_pair.first, desc_internal);
if (can_keypool) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm looking at it after a month and have lost some context here.

What restriction in the previous code was disallowing the addition of public keys in the keypool for non-single-key type descriptors that made the "Imported scripts with pubkeys should not have their pubkeys go into the keypool" subtest pass? Sorry if this has been answered earlier but I didn't find it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prior to this PR, the multisig descriptors would not insert any pubkeys int out.pubkeys inside of MakeScripts. The pubkeys that are inserted to that FlatSigningProvider are the ones inserted into the keypool.

Copy link
Contributor

@rkrux rkrux Apr 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, they were being added for pkh, wpkh, combo, tr, miniscript ones in MakeScript but not for multisig & the like. I debugged this flow in on master while importing a multisig descriptor and found it quite strange that the origins of all the pubkeys involved in the multisig were being added in the out.origins inside ExpandHelper but not all the pubkeys were added in out.pubkeys!
Also, pubkeys being set inside MakeScript but origins being set in ExpandHelper made it difficult to reason about for me, the newer implementation makes it easier to follow.

bitcoinclireg importdescriptors '[{"desc": "wsh(multi(2,tprv8ZgxMBicQKsPdzuc344mDaeUk5zseMcRK9Hst8xodskNu3YbQG5NxLa2X17PUU5yXQhptiBE7F5W5cgEmsfQg4Y21Y18w4DJhLxSb8CurDf,tpubD6NzVbkrYhZ4YiCvExLvH4yh1k3jFGf5irm6TsrArY8GYdEhYVdztQTBtTirmRc6XfSJpH9tayUdnngaJZKDaa2zbqEY29DfcGZW8iRVGUY))#a0tl2jdn", "timestamp": "now"}]'

I used importdescriptors but the Expand flow is the same as in importmulti.

Comment on lines -214 to +215
/** Derive a private key, if private data is available in arg. */
virtual bool GetPrivKey(int pos, const SigningProvider& arg, CKey& key) const = 0;
/** Derive a private key, if private data is available in arg and put it into out. */
virtual void GetPrivKey(int pos, const SigningProvider& arg, FlatSigningProvider& out) const = 0;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a suggestion, just thinking out loud.

For the sake of having a function signature consistent with the corresponding public key function - I suppose returning the private key from this function is not an option because we'd prefer to have fewer instances of private keys moving around within the codebase?

Though I'm wondering can there be an use case later where the caller would need to know whether the private key was gotten in which case the boolean return value is needed; seems a bit odd that the caller would need to find the key in the out every time even if they don't want to use the key.

Copy link
Contributor

@fjahr fjahr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review ACK acee5c5

/** Compare two public keys represented by this provider.
* Used by the Miniscript descriptors to check for duplicate keys in the script.
*/
bool operator<(PubkeyProvider& other) const {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: could make other const if you retouch

@DrahtBot DrahtBot requested a review from theStack April 18, 2025 12:01
@fjahr
Copy link
Contributor

fjahr commented Apr 18, 2025

LGTM, code review acee5c5

Was that an ACK an you just forgot the word or did you just want to say you reviewed? :)

Copy link
Contributor

@theStack theStack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-ACK acee5c5

case-sensitivity yocto-nit: in commit subject for 6268bde: s/BIP32PUbKeyProvider/BIP32PubkeyProvider/

Copy link
Contributor

@rkrux rkrux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK acee5c5

Was that an ACK an you just forgot the word or did you just want to say you reviewed? :)

I wanted to convey I code reviewed & almost ack. There was one thing that I had not completely reasoned about though it made sense to me intuitively somehow, so was hesitant in acking earlier: #31243 (comment)

std::copy(scripts_temp.begin(), scripts_temp.end(), std::inserter(script_pub_keys, script_pub_keys.end()));
for (const auto& key_pair : out_keys.pubkeys) {
ordered_pubkeys.emplace_back(key_pair.first, desc_internal);
if (can_keypool) {
Copy link
Contributor

@rkrux rkrux Apr 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, they were being added for pkh, wpkh, combo, tr, miniscript ones in MakeScript but not for multisig & the like. I debugged this flow in on master while importing a multisig descriptor and found it quite strange that the origins of all the pubkeys involved in the multisig were being added in the out.origins inside ExpandHelper but not all the pubkeys were added in out.pubkeys!
Also, pubkeys being set inside MakeScript but origins being set in ExpandHelper made it difficult to reason about for me, the newer implementation makes it easier to follow.

bitcoinclireg importdescriptors '[{"desc": "wsh(multi(2,tprv8ZgxMBicQKsPdzuc344mDaeUk5zseMcRK9Hst8xodskNu3YbQG5NxLa2X17PUU5yXQhptiBE7F5W5cgEmsfQg4Y21Y18w4DJhLxSb8CurDf,tpubD6NzVbkrYhZ4YiCvExLvH4yh1k3jFGf5irm6TsrArY8GYdEhYVdztQTBtTirmRc6XfSJpH9tayUdnngaJZKDaa2zbqEY29DfcGZW8iRVGUY))#a0tl2jdn", "timestamp": "now"}]'

I used importdescriptors but the Expand flow is the same as in importmulti.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably in some follow up: The documentation of the MakeScripts function in DescriptorImpl class can be updated to get rid of the following line because the origin info is added in the GetPubKey functions now and was added in the ExpandHelper function previously.

The origin info of the provided pubkeys is automatically added.

@glozow glozow merged commit 3e78ac6 into bitcoin:master Apr 21, 2025
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants