Skip to content

Conversation

@luke-jr
Copy link
Member

@luke-jr luke-jr commented Apr 25, 2022

walletprocesspsbt has a single logical positional argument and a bunch that only make sense as named options. Convert them to an actual options Object for a better positional API.

Retains backward compatibility for now.

Copy link

@vincenzopalazzo vincenzopalazzo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 31ffd77

@DrahtBot
Copy link
Contributor

DrahtBot commented Apr 25, 2022

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/24963.

Reviews

See the guideline for information on the review process.

Type Reviewers
ACK ryanofsky
Concept ACK achow101
Stale ACK pk-b2, vincenzopalazzo

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #31622 (psbt: add non-default sighash types to PSBTs and unify sighash type match checking by achow101)
  • #29675 (wallet: Be able to receive and spend inputs involving MuSig2 aggregate keys by achow101)
  • #21283 (Implement BIP 370 PSBTv2 by achow101)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Copy link

@pk-b2 pk-b2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 31ffd7782bf241ca09c7e192769ae9506bd2352f

  • Built branch locally and exercised various manual test using both versions of the RPC
  • Verified that passed in arguments get accepted
  • Throws error when passing in options argument + additional invalid arguments

@achow101
Copy link
Member

tbh I prefer #19762 instead of having options objects, but I don't feel strongly either way about this PR.

@luke-jr
Copy link
Member Author

luke-jr commented Nov 9, 2022

tbh I prefer #19762 instead of having options objects, but I don't feel strongly either way about this PR.

#19762 is complimentary to options objects, not an alternative to them.

@achow101
Copy link
Member

achow101 commented Feb 3, 2023

There's a silent merge conflict with master.

wallet/rpc/spend.cpp: In function ‘RPCHelpMan wallet::walletprocesspsbt()’:
wallet/rpc/spend.cpp:1515:75: error: ‘OMITTED_NAMED_ARG’ is not a member of ‘RPCArg::Optional’
 1515 |                     {"options|sign", RPCArg::Type::OBJ, RPCArg::Optional::OMITTED_NAMED_ARG, "",
      |                                                                           ^~~~~~~~~~~~~~~~~
wallet/rpc/spend.cpp: In lambda function:
wallet/rpc/spend.cpp:1574:9: error: ‘RPCTypeCheckArgument’ was not declared in this scope
 1574 |         RPCTypeCheckArgument(options, UniValue::VOBJ);
      |         ^~~~~~~~~~~~~~~~~~~~
wallet/rpc/spend.cpp: In function ‘RPCHelpMan wallet::walletprocesspsbt()’:
wallet/rpc/spend.cpp:1619:5: error: no matching function for call to ‘RPCHelpMan::RPCHelpMan(<brace-enclosed initializer list>)’
 1619 |     };
      |     ^
In file included from wallet/rpc/spend.cpp:10:
./rpc/util.h:364:5: note: candidate: ‘RPCHelpMan::RPCHelpMan(std::string, std::string, std::vector<RPCArg>, RPCResults, RPCExamples, RPCMethodImpl)’
  364 |     RPCHelpMan(std::string name, std::string description, std::vector<RPCArg> args, RPCResults results, RPCExamples examples, RPCMethodImpl fun);
      |     ^~~~~~~~~~
./rpc/util.h:364:79: note:   no known conversion for argument 3 from ‘<brace-enclosed initializer list>’ to ‘std::vector<RPCArg>’
  364 |     RPCHelpMan(std::string name, std::string description, std::vector<RPCArg> args, RPCResults results, RPCExamples examples, RPCMethodImpl fun);
      |                                                           ~~~~~~~~~~~~~~~~~~~~^~~~
./rpc/util.h:362:5: note: candidate: ‘RPCHelpMan::RPCHelpMan(std::string, std::string, std::vector<RPCArg>, RPCResults, RPCExamples)’
  362 |     RPCHelpMan(std::string name, std::string description, std::vector<RPCArg> args, RPCResults results, RPCExamples examples);
      |     ^~~~~~~~~~
./rpc/util.h:362:5: note:   candidate expects 5 arguments, 6 provided
./rpc/util.h:359:7: note: candidate: ‘RPCHelpMan::RPCHelpMan(const RPCHelpMan&)’
  359 | class RPCHelpMan
      |       ^~~~~~~~~~
./rpc/util.h:359:7: note:   candidate expects 1 argument, 6 provided
./rpc/util.h:359:7: note: candidate: ‘RPCHelpMan::RPCHelpMan(RPCHelpMan&&)’
./rpc/util.h:359:7: note:   candidate expects 1 argument, 6 provided

@fanquake fanquake marked this pull request as draft February 28, 2023 10:26
@fanquake
Copy link
Member

Drafted for now. Needs a PR description and rebasing to fix the merge conflict:

wallet/rpc/spend.cpp:1541:75: error: no member named 'OMITTED_NAMED_ARG' in 'RPCArg::Optional'
                    {"options|sign", RPCArg::Type::OBJ, RPCArg::Optional::OMITTED_NAMED_ARG, "",
                                                        ~~~~~~~~~~~~~~~~~~^
wallet/rpc/spend.cpp:1600:9: error: use of undeclared identifier 'RPCTypeCheckArgument'
        RPCTypeCheckArgument(options, UniValue::VOBJ);
        ^
2 errors generated.

@luke-jr
Copy link
Member Author

luke-jr commented Jul 22, 2023

Rebased

@luke-jr luke-jr requested review from pk-b2 and vincenzopalazzo July 22, 2023 22:28
Copy link
Contributor

@ryanofsky ryanofsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Light code review ACK baf99a9c789779e89e7684a550e16cf1fb7ce862

src/rpc/util.h Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "RPC: Support specifying different types for param aliases" (8a11b1525f2fa9f73715db46d228bda1eacdd2b2)

I think just calling this new struct member m_types would be more consistent with the m_names member. Also I think a followup commit should remove the m_type member which is now confusing and redundant.

src/rpc/util.cpp Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "RPC: Support specifying different types for param aliases" (8a11b1525f2fa9f73715db46d228bda1eacdd2b2)

It would be good to have unit test coverage for this new code. Should be easy to add a test creating an RPCArg and calling this method in rpc_tests.cpp.

@DrahtBot DrahtBot requested review from pk-b2 and vincenzopalazzo and removed request for pk-b2 and vincenzopalazzo August 1, 2023 17:13
@achow101 achow101 removed the request for review from pk-b2 April 15, 2024 21:28
@DrahtBot DrahtBot marked this pull request as draft April 23, 2024 06:46
@DrahtBot
Copy link
Contributor

Converted to draft due to failing CI and inactivity

@luke-jr
Copy link
Member Author

luke-jr commented Nov 16, 2024

CI failure appears to be a CI bug unrelated to this PR.

@DrahtBot DrahtBot requested a review from pk-b2 November 16, 2024 23:02
@luke-jr luke-jr marked this pull request as ready for review November 16, 2024 23:21
@luke-jr luke-jr force-pushed the rpc_walletprocesspsbt_options branch from 0d09c94 to 645d041 Compare November 17, 2024 04:37
@luke-jr
Copy link
Member Author

luke-jr commented Nov 17, 2024

Rebased to workaround CI bugs for now (is someone going to fix them?)

@achow101
Copy link
Member

Rebased to workaround CI bugs for now (is someone going to fix them?)

Considering that CI is failing immediately after a walletprocesspsbt call, and that RPC is being changed in this PR, I'm inclined to think that it's probably not a CI issue.

@luke-jr
Copy link
Member Author

luke-jr commented Nov 17, 2024

Considering that CI is failing immediately after a walletprocesspsbt call, and that RPC is being changed in this PR, I'm inclined to think that it's probably not a CI issue.

Yes, there was a new issue introduced by the added commits too. But there was also a CI issue: it failed to merge into master before building.

@maflcko
Copy link
Member

maflcko commented Nov 17, 2024

Considering that CI is failing immediately after a walletprocesspsbt call, and that RPC is being changed in this PR, I'm inclined to think that it's probably not a CI issue.

Yes, there was a new issue introduced by the added commits too. But there was also a CI issue: it failed to merge into master before building.

Not sure what failure you are referring to, but looking at https://github.com/bitcoin/bitcoin/actions/runs/11875971339/job/33093790745, it looks like the problem was that the CI did merge the config into master.

The check-each-commit task is specifically written to compile the exact commits that are provided in the pull request, so that git bisect issues are detected early on.

GHA is certainly buggy and I am not sure if there is a way to ask for some tasks to merge config and code and for others to not merge config nor code.

Doing a rebase seems a simple enough workaround for now. If someone disagrees, pull requests with improvements are always welcome.

@luke-jr luke-jr force-pushed the rpc_walletprocesspsbt_options branch from ebbde4f to 40143ba Compare November 17, 2024 16:33
@luke-jr
Copy link
Member Author

luke-jr commented Nov 17, 2024

Assuming the remaining CI failure is #31307

Copy link
Contributor

@ryanofsky ryanofsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review ACK 40143ba.

This is a pretty simple change to the RPC framework that should make it easy to add new options to RPCs like createwallet that currently accept too many positional parameters, without having to increase the number of positional parameters they already accept.

Without this PR, the only ways to add a new options to an RPC like createwallet without breaking backward compatibility are to add the option as a top-level parameter, where it can be specified as:

bitcoin-cli -named createwallet wallet_name option_name=option_value
bitcoin-cli createwallet wallet_name null null null null null null null option_value

or to add a new OBJ_NAMED_PARAMS parameter where it can be specified as:

bitcoin-cli -named createwallet wallet_name option_name=option_value
bitcoin-cli createwallet wallet_name null null null null null null null '{"option_name": "option_value"}'

The first approach is unsafe because it allows option_value to be specified by position instead of name in an error prone way when there are too many positions. The second approach is safe, but not very appealing because it still increases the number of positional arguments, and doesn't display the old and new arguments in the same way in the documentation, or allow them to be accessed in the same way in the RPC implementation.

By contrast, this PR allows completely modernizing RPCs that accept a lot of positional parameters while retaining backwards compatibility. E.g. it would allow a new createwallet option to be passed as:

bitcoin-cli -named createwallet wallet_name option_name=option_value
bitcoin-cli createwallet wallet_name '{"option_name": "option_value"}'

And it would show all options consistently in the documentation, and give callers ability to pass every option as a named parameter or as a field in an options object.

Notes on how this PR works:

  • The first commit "RPC: Support specifying different types for param aliases" (04d7864) allows multiple types to be specified for parameters if the parameters have aliases, like: "options|sign", {RPCArg::Type::OBJ_NAMED_PARAMS, RPCArg::Type::BOOL}. It enforces that the number of types is the same as the number of aliases. Previously it was only possible to specify a single type for a parameter, even if it had multiple aliases. Additionally:
    • It updates the RPCArg::MatchesType() function used to validate incoming RPC requests to accept any of the types specified in the list, instead of just the first type. If named parameters were passed, this commit does not enforce that the specific type corresponding to the named alias was used, but the third commit fixes this.
    • It fixes RPCHelpMan::GetArgMap(), which is used to generate bitcoin-cli -regtest help dump_all_command_conversions output, to output the correct type for each alias, instead just outputting the first type.
  • The second commit updates walletprocesspsbt to take advantage of the support for alias types added in the first commit. Specifically:
    • It changes the second parameter from "sign" accepting a bool to "options|sign" accepting an options object or a bool.
    • It copies former definitions of the parameters 2-5 ("sign", "sighashtype", "bip32derivs", "finalize") into the options definition so they can be passed as options or named parameters by RPC clients.
    • It keeps parameters 3-5 intact but marks them as hidden. Keeping these parameters allows clients to continue to pass them as positional parameters instead of as named parameters or options fields.
    • It marks all the options {.also_positional = true} to avoid errors from the RPC framework about options and parameters having duplicate names.
    • It updates the implementation of walletprocesspsbt to read options and named arguments out of the options object, but keeps code for reading positional arguments out of the params object if no options object is present.
  • The third commit adds stricter type checking to ensure that if a parameter has multiple aliases and each alias has an assigned type, and the parameter is passed by name instead of position, the parameter must have the expected type for that name. This prevents a bug described #24963 (comment), and the fourth commit just adds a test for this bug.

} else {
// New style options are in an object
UniValue options = request.params[1];
RPCTypeCheckObj(options,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "RPC/Wallet: Convert walletprocesspsbt to use options parameter" (28023ff)

I think it would be good to move this code higher in the function, before the BlockUntilSyncedToCurrentChain and DecodeBase64PSBT calls to give caller more immediate feedback if they are calling this function with invalid parameters or the wrong types.


# After update with wallet, only needs signing
updated = self.nodes[1].walletprocesspsbt(psbt, False, 'ALL', True)['psbt']
assert_equal(updated, self.nodes[1].walletprocesspsbt(psbt, {"sign": False, "sighashtype": 'ALL', "bip32derivs": True})["psbt"])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "RPC/Wallet: Convert walletprocesspsbt to use options parameter" (28023ff)

Might be good to test a combination of named / positional arguments like

assert_equal(updated, self.nodes[1].walletprocesspsbt(psbt, sign=False, sighashtype='ALL', bip32derivs=True)["psbt"])

@DrahtBot
Copy link
Contributor

🐙 This pull request conflicts with the target branch and needs rebase.

@DrahtBot
Copy link
Contributor

⌛ There hasn't been much activity lately and the patch still needs rebase. What is the status here?

  • Is it still relevant? ➡️ Please solve the conflicts to make it ready for review and to ensure the CI passes.
  • Is it no longer relevant? ➡️ Please close.
  • Did the author lose interest or time to work on this? ➡️ Please close it and mark it 'Up for grabs' with the label, so that it can be picked up in the future.

@achow101
Copy link
Member

Closing this as it has not had any activity in a while. If you are interested in continuing work on this, please leave a comment so that it can be reopened.

Marking as up for grabs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants