Skip to content

Conversation

@pablomartin4btc
Copy link
Member

@pablomartin4btc pablomartin4btc commented Oct 4, 2025

This PR simplifies the argument parsing logic and resolves practical issues for bitcoin-cli and similar binaries (e.g. bitcoin-wallet) making their usage more consistent and predictable, enabling valid patterns such as mixing commands with wallet- or RPC-specific options—without relying on argument ordering quirks.

What's changed in ArgsManager that allows the above.
  • Previously, ArgsManager stopped interpreting options once a non-option argument was encountered (non-GNU parsing). For example:

      bitcoind -a -b param -c
    

    would ignore -c because it followed the non-option argument param. In some cases '-c' is interpreted as an argument for 'param', producing unexpected behavior (eg empty arrays for listtransactions command).

  • This PR changes the behavior so that options following non-option argument are recognized and processed (GNU parsing). This is needed for workflows for tools like CLI where command arguments may be followed by wallet- or RPC-specific
    options, e.g.:

      bitcoin-cli -rpcwallet="" listtransactions -rpcwallet=mywallet
    
      bitcoin-cli getaddressinfo <address> -rpcwallet=mywallet
    
  • (other binaries could also benefit from this change, e.g. bitcoin-wallet)

      bitcoin-wallet create -wallet=newname
    
  • Invalid options passed after non-options will be treated as RPC arguments that begin with a - character (so no
    behavioural change against master), e.g.:

      bitcoin-cli -regtest -datadir=/tmp/btc createwallet -mywallet
      {
          "name": "-mywallet"
      }
    
Before and after outputs running RPC commands from CLI/ bitcoin-cli.
  • wallet commands using CLI:
    • getaddressinfo
      • before/ master
        ./build_master/bin/bitcoin-cli -regtest -datadir=/tmp/btc getaddressinfo bcrt1qrnz5xlcwz5d85n3fag6v0vc4lryjc90jhqtpsh -rpcwallet=nonExistent
        error message:
        getaddressinfo "address"
        ...
        
      • after
        ./build/bin/bitcoin-cli -regtest -datadir=/tmp/btc getaddressinfo bcrt1qrnz5xlcwz5d85n3fag6v0vc4lryjc90jhqtpsh -rpcwallet=nonExistent
        error code: -18
        error message:
        Requested wallet does not exist or is not loaded
        
    • listtransactions
      • before/ master
        ./build_master/bin/bitcoin-cli -regtest -datadir=/tmp/btc listtransactions -rpcwallet=nonExistent
        [
        ]
        
      • after
        ./build/bin/bitcoin-cli -regtest -datadir=/tmp/btc listtransactions -rpcwallet=nonExistent
        error code: -18
        error message:
        Requested wallet does not exist or is not loaded
        
    • any command that needs an option to be specifed, it can be added at the end directly instead of before the command
      • before/ master
        ./build_master/bin/bitcoin-cli -regtest -datadir=/tmp/btc listtransactions
        error code: -19
        error message:
        Multiple wallets are loaded. Please select which wallet to use by requesting the RPC through the /wallet/<walletname> URI path. Or for the CLI, specify the "-rpcwallet=<walletname>" option before the command (run "bitcoin-cli -h" for help or "bitcoin-cli listwallets" to see which wallets are currently loaded).
        
        ./build_master/bin/bitcoin-cli -regtest -datadir=/tmp/btc listtransactions -rpcwallet=""
        error code: -19
        error message:
        Multiple wallets are loaded. Please select which wallet to use by requesting the RPC through the /wallet/<walletname> URI path. Or for the CLI, specify the "-rpcwallet=<walletname>" option before the command (run "bitcoin-cli -h" for help or "bitcoin-cli listwallets" to see which wallets are currently loaded).
        
      • after
        /build/bin/bitcoin-cli -regtest -datadir=/tmp/btc listtransactions -rpcwallet=""
        [
          {
            "address": "bcrt1q5pqkrlfrp9rvsg43sxxagrees7r7ul6uyskhs4",
            "parent_descs": [
              "wpkh([76a11ab0/84h/1h/0h]tpubDCDCuyC1Rtm1weZ5kfBYtdchcmqa2t8db7M68wsPHbWhUrYnzFTrHu7ufVVRydt3FdWQy8iNmKbZYbgpWWcEKPAj896x53UbS6i2xRD1qct/0/*)#kuff3m9t"
            ],
            "category": "immature",
            "amount": 50.00000000,
            "label": "",
        ...
        
  • other binaries could benefit from it, e.g. bitcoin-wallet:
    • create command
      • before/ master
        ./build_master/bin/bitcoin-wallet -regtest -datadir=/tmp/btc create
        Wallet name must be provided when creating a new wallet.
        

        (-wallet option has to be passed before the create command in order to work)

        ./build_master/bin/bitcoin-wallet -regtest -datadir=/tmp/btc create -wallet=newWalletName
        Error: Additional arguments provided (-wallet=newWalletName). Methods do not take arguments. Please refer to `-help`.
        
      • after
        ./build/bin/bitcoin-wallet -regtest -datadir=/tmp/btc create -wallet=newWallet
        Topping up keypool...
        Wallet info
        ===========
        Name: newWallet
        Format: sqlite
        Descriptors: yes
        Encrypted: no
        HD (hd seed available): yes
        Keypool Size: 8000
        Transactions: 0
        Address Book: 0
        

-Notes:

  • Backwards compatibility:
    This PR is backward compatible in the sense that it allows new valid invocations that were previously rejected (failed or somehow ignored).
    In master, any valid option following a command is treated as a positional argument. This can lead to unexpected RPC errors or help messages due to mismatched argument types. With this PR, those cases are parsed as proper options when applicable.

  • bitcoin-cli continues to accept RPC arguments that begin with a - character as currently in master (check createwallet example in "What's change in ArgsManager" section above).

  • bitcoin-cli could distinguish between options that begin with single and double dashes, and treat options after the RPC method name that begin with double dashes as RPC named parameters (check example in 4th commit message body).

  • Refactored both ArgsManager::ProcessOptionKey (new function added in this PR) and ArgsManager::ParseParameters in separated commits making the code clearer and the PR much easier to follow and review.

@DrahtBot
Copy link
Contributor

DrahtBot commented Oct 4, 2025

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33540.

Reviews

See the guideline for information on the review process.

Type Reviewers
Concept ACK w0xlt, ryanofsky

If your review is incorrectly listed, please copy-paste <!--meta-tag:bot-skip--> into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #17783 (common: Disallow calling IsArgSet() on ALLOW_LIST options by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

LLM Linter (✨ experimental)

Possible typos and grammar issues:

  • double dash'ed -> double-dashed [apostrophe misuse; "dash'ed" is nonstandard — use "double-dashed" (or "double-dash") for clarity]
  • startig -> starting [typo: "startig" should be "starting"]

2025-12-18

@katesalazar

This comment has been minimized.

Copy link
Contributor

@w0xlt w0xlt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK

@pinheadmz
Copy link
Member

pinheadmz commented Oct 5, 2025

Such good news. Hard to believe this PR doesn't reference an existing user request.

@katesalazar

This comment does not contribute to the technical discussion. Please put in more effort in the future or you may be banned.

@ryanofsky
Copy link
Contributor

Haven't looked at code yet, so forgive if these are dumb questions, but could be nice if PR description addressed these:

  • Is there a way to pass bitcoin-cli RPC arguments that begin with a - character? Can you escape these arguments with -- for example?

  • What's the PR's goal for backwards compatibility? Is the PR only allowing command lines that were disallowed before, and therefore fully backwards compatible? Or are there some command lines which used to be accepted which will now be interpreted differently?

  • What are "numeric command arguments" referenced in the PR description? Maybe give an example. I wasn't aware that numbers were treated differently than other characters for separating options arguments, and I would expect interpretation of numeric values to happen at a later phase than the phase distinguishing options from non-option arguments.

@ryanofsky
Copy link
Contributor

Concept ACK on the idea. I think it probably makes sense to allow options arguments to follow non-option arguments even this change is not strictly backwards compatible, as long as there is some way to escape non-option arguments beginning with -, such as with a -- separator.

@ryanofsky
Copy link
Contributor

Assuming we are ok with losing some backwards compatibility by starting to treating arguments that begin with - as options instead of non-options when they follow arguments that don't begin with -, there could be other interesting ways to extend this PR.

For example, bitcoin-cli could distinguish between options that begin with single and double dashes, and treat options after the RPC method name that begin with double dashes as RPC named parameters. In other words, accept:

bitcoin-cli createwallet mywallet --load_on_startup=1

as an equivalent to:

bitcoin-cli -named createwallet mywallet load_on_startup=1

This seems like it could provide a more convenient way of using named RPC parameters.

@pablomartin4btc
Copy link
Member Author

* Is there a way to pass `bitcoin-cli` RPC arguments that begin with a `-` character?

At the moment, please correct me if I'm wrong, there's no or I haven't seen an RPC that would receive a valid argument that starts with - (but I understand it could be in the future), and there's no test for it otherwise I think the PR would have caught it.

Testing the scenario you mentioned in master would work like for example in:

./build_master/bin/bitcoin-cli -regtest -datadir=/tmp/btc createwallet -xyz
{
  "name": "-xyz"
}

On this PR would fail because it would be an invalid option (from the ArgsMan):

./build/bin/bitcoin-cli -regtest -datadir=/tmp/btc createwallet -xyz
Error parsing command line arguments: Invalid parameter -xyz

I think perhaps the scenario you mention is a valid one since any a string argument for an RPC could start with -, so only the "invalid" options (like -xyz in the previous example) would be leaved or passed to the RPC, which makes sense.

The change would look like this, and we can drop the 2nd commit (bringing back alive ParsePrechecks() and ParseDouble()), which won't make sense anymore (I explain why I brought them back in your last question on this thread since you've asked as well). The change already passes the CIs - CentOS CI failure is unrelated.

So even the following case would be successful:

./build_ryan/bin/bitcoin-cli createwallet -regtest -zyx -datadir=/tmp/btc
{
  "name": "-zyx"
}

Can you escape these arguments with -- for example?

Not on this PR. I see you think we could extend it with the usage of -- in a later comment. I wanted to make a minimum change to allow this feature of GNU-style parsing options so I'd prefer to add that on a separate PR if that's necessary.

* What's the PR's goal for backwards compatibility? Is the PR only allowing command lines that were disallowed before, and therefore fully backwards compatible? Or are there some command lines which used to be accepted which will now be interpreted differently?

The difference now will be that valid options starting with - will be considered or accepted while before or in master these scenarios will be considered that an extra argument is passed to the command and sometimes would be treated as a valid string (when the arg is of such type for the command) returning something empty if it had to match certain condition or would fail because (1) it's a different type like a JSON error, (2) the RPC had received extra arguments resulting in the command help or (3) too many args for CLI commands. So in terms of backwards compatibility, it would consider valid scenarios that were invalid before, and if someone expected an error, if the option is valid it will be taken into consideration and the execution will be successful instead.

* What are "numeric command arguments" referenced in the PR description? Maybe give an example. I wasn't aware that numbers were treated differently than other characters for separating options arguments, and I would expect interpretation of numeric values to happen at a later phase than the phase distinguishing options from non-option arguments.

Yeah the interpretation of numeric values come later on the command execution, not during the parsing in the Argsman, but when I made the change, there was a a test that was failing:

assert_raises_rpc_error(-8, "Invalid start_height", restorewo_wallet.rescanblockchain, -1, 10)

So then I checked that there is another RPC case (getnetworkhashps) which has a default RPCArg::Type::NUM with RPCArg::Default{-1}:

{"height", RPCArg::Type::NUM, RPCArg::Default{-1}, "To estimate at the time of the given height."},

That's the reason of the 2nd commit, reintroducing doubles parsing, but since your first question on this thread, I'm considering this change to be pushed (added a test scenario where commands could receive args starting with -) so that 2nd commit can be dropped as it's no longer needed (that's done in the change mentioned in the previous sentence).

@pablomartin4btc pablomartin4btc force-pushed the argsman-GNU-style-command-line-option-parsing branch 3 times, most recently from 8258dc3 to 8b97f47 Compare October 13, 2025 03:04
@pablomartin4btc
Copy link
Member Author

pablomartin4btc commented Oct 13, 2025

-Updates:

  • Addressed @ryanofsky's feedback:
    • bitcoin-cli accepts RPC arguments that begin with a - character;
    • added a note in description about backwards compatibility;
    • bitcoin-cli treat options after the RPC method name that begin with double dashes as RPC named parameters;
  • Dropped "Reintroduce ParsePrechecks and ParseDouble" commit;
  • Updated description accordingly;
  • CentOS CI failure is unrelated ("no space left on device" -> it seems ci: GHA fallback centos task runs out of space #33293);
  • Pending to fix Tidy CI failure ("recursive call chain").

@pablomartin4btc pablomartin4btc force-pushed the argsman-GNU-style-command-line-option-parsing branch from 8b97f47 to a101bbb Compare October 13, 2025 05:18
@pablomartin4btc
Copy link
Member Author

-Updates:

  • Fixed previous Tidy CI failure ("recursive call chain"), but still working on a refactor of ArgsManager::ProcessOptionKey.

Copy link
Contributor

@w0xlt w0xlt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This command works on master but not in this PR:
./build/bin/bitcoin-cli --datadir=/tmp/btc1 --signet getblockhash 1000

Could it be related to GetCommandArgs()?

Copy link
Contributor

@w0xlt w0xlt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restructuring ProcessOptionKey as below seems to solve the problem of ./build/bin/bitcoin-cli --datadir=/tmp/btc1 --signet getblockhash 1000.

That seems to fix the Method not found regression while preserving the GNU-style options after the command and the new double-dash-named-params feature.

bool ArgsManager::ProcessOptionKey(std::string& key, std::optional<std::string>& val, std::string& error, const bool found_after_non_option) {
    bool double_dash{false};
    const std::optional<std::string> original_val{val};
    std::string original_input{key};
    if (val) original_input += "=" + *val;

    // Normalize leading dashes
    if (key.length() > 1 && key[1] == '-') {
        key.erase(0, 1);   // `--foo` -> `-foo`
        double_dash = true;
    }
    key.erase(0, 1);       // `-foo`  -> `foo`

    KeyInfo keyinfo = InterpretKey(key);
    std::optional<unsigned int> flags = GetArgFlags('-' + keyinfo.name);
    const bool known_option = flags.has_value() && keyinfo.section.empty();

    // Handle the special "named RPC" case early ---
    if (double_dash && found_after_non_option && !known_option) {
        // Try to map this to `-named`, if supported by the binary.
        val.reset();
        KeyInfo named_keyinfo = InterpretKey("named");
        std::optional<unsigned int> named_flags = GetArgFlags('-' + named_keyinfo.name);

        if (named_flags && named_keyinfo.section.empty()) {
            // Binary supports -named: enable it and append the stripped parameter.
            std::optional<common::SettingsValue> named_value =
                InterpretValue(named_keyinfo, /* val */ nullptr, *named_flags, error);
            if (!named_value) return false;

            {
                LOCK(cs_args);
                m_settings.command_line_options[named_keyinfo.name].push_back(*named_value);
                // Strip the leading "--" before passing to the command as a named param
                m_command.emplace_back(original_input.substr(2));
            }
            return true;
        }

        // Binary doesn’t support -named: fall back and treat `--foo` as a
        // normal option (or unknown option handled below).
        keyinfo = InterpretKey(key);
        flags = GetArgFlags('-' + keyinfo.name);
    }

    // --- Normal option handling ---
    if (!flags || !keyinfo.section.empty()) {
        if (!found_after_non_option) {
            // Unknown global option: keep legacy behaviour (error)
            error = strprintf("Invalid parameter %s", original_input);
            return false;
        }

        // Unknown option after command: pass through to the command as an arg
        LOCK(cs_args);
        m_command.emplace_back(original_input);
        return true;
    }

    // Known option (including `--datadir`, `--signet`, etc.)
    std::optional<common::SettingsValue> value =
        InterpretValue(keyinfo, val ? &*val : nullptr, *flags, error);
    if (!value) return false;

    LOCK(cs_args);
    m_settings.command_line_options[keyinfo.name].push_back(*value);
    return true;
}

@pablomartin4btc
Copy link
Member Author

This command works on master but not in this PR: ./build/bin/bitcoin-cli --datadir=/tmp/btc1 --signet getblockhash 1000

Thanks for testing it (not sure if that's valid, no test failed, perhaps we need to add one?), anyways just putting the output's diff here.

master:

/build_master/bin/bitcoin-cli --datadir=/tmp/btc --signet getblockhash 1000
error code: -8
error message:
Block height out of range

this branch:

./build/bin/bitcoin-cli --datadir=/tmp/btc --signet getblockhash 1000
error: timeout on transient error: Could not connect to the server 127.0.0.1:8332

Make sure the bitcoind server is running and that you are connecting to the correct RPC port.
Use "bitcoin-cli -help" for more info.

@pablomartin4btc
Copy link
Member Author

Restructuring ProcessOptionKey as below seems to solve the problem...

Checking it... Thanks for the suggestion!

@pablomartin4btc
Copy link
Member Author

Restructuring ProcessOptionKey as below seems to solve the problem...

Checking it... Thanks for the suggestion!

Ok, all tests pass...

First checked only the affected ones by my changes:

./build/test/functional/feature_config_args.py
./build/test/functional/tool_wallet.py
./build/test/functional/wallet_multiwallet.py
./build/test/functional/wallet_transactiontime_rescan.py

./build/bin/test_bitcoin --log_level=all --run_test=argsman_tests

Then I ran them all including the functional --extended.

Thanks again for working on this! I'm seeing if I can make the code a bit clearer otherwise I'll include your suggestion.

@pablomartin4btc pablomartin4btc force-pushed the argsman-GNU-style-command-line-option-parsing branch from 5089e20 to 0486cba Compare December 8, 2025 15:02
@DrahtBot
Copy link
Contributor

DrahtBot commented Dec 8, 2025

🚧 At least one of the CI tasks failed.
Task lint: https://github.com/bitcoin/bitcoin/actions/runs/20032344935/job/57444563462
LLM reason (✨ experimental): Python linting failed: ruff detected an f-string without placeholders in test code, causing the lint step to fail and CI to exit with error.

Hints

Try to run the tests locally, according to the documentation. However, a CI failure may still
happen due to a number of reasons, for example:

  • Possibly due to a silent merge conflict (the changes in this pull request being
    incompatible with the current code in the target branch). If so, make sure to rebase on the latest
    commit of the target branch.

  • A sanitizer issue, which can only be found by compiling with the sanitizer and running the
    affected test.

  • An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

@pablomartin4btc pablomartin4btc force-pushed the argsman-GNU-style-command-line-option-parsing branch 3 times, most recently from 5f0ff21 to f6f44df Compare December 8, 2025 15:58
@pablomartin4btc pablomartin4btc force-pushed the argsman-GNU-style-command-line-option-parsing branch from f6f44df to 4f96ced Compare December 8, 2025 16:24
@pablomartin4btc
Copy link
Member Author

-Updates:

  • Added @w0xlt's fix on long (double dash --) options that worked on master, adding a new test for it (the test would pass in master but fail without @w0xlt's changes).
  • Refactored both ArgsManager::ProcessOptionKey (new function added in this PR) and ArgsManager::ParseParameters in separated commits making the code clearer and the PR much easier to follow and review.

pablomartin4btc and others added 7 commits December 18, 2025 13:56
- Centralizes normalization of option keys in `ProcessOptionKey`.
  This will help in next commit to reuse this function avoiding code
  duplication.
- Improves readability and makes behavior easier to reason about.

Functional and unit tests confirm no behavior change.
Previously, ArgsManager stopped interpreting options once a
non-option argument was encountered (non-GNU parsing).

For example:

    bitcoind -a -b param -c

would ignore `-c` because it followed the non-option argument
`param`. In cases where some 'param' command receives parameters,
'-c' is interpreted as one of them producing unexpected behavior
(e.g. empty arrays for listtransactions command).

This commit changes the behavior so that options following
non-option argument are recognized and processed (GNU parsing).
This is needed for workflows for tools like CLI where command
arguments may be followed by wallet- or RPC-specific options, e.g.:

    bitcoin-cli listtransactions -rpcwallet=mywallet

    bitcoin-cli getaddressinfo <address> -rpcwallet=mywallet

    bitcoin-wallet create -wallet=newname

Also, invalid options passed after non-options will be treated
as RPC arguments that begin with a '-' (dash) character (so no
behavioural change against master), e.g.:

    bitcoin-cli -regtest -datadir=/tmp/btc createwallet -mywallet
    {
        "name": "-mywallet"
    }
Refactor bitcoin-cli to use command arguments provided by ArgsManager
instead of manually slicing argv.

- Introduce GetCommandArgs() helper.
- Remove local argv/argc skipping logic.
- Simplify CommandLineRPC argument collection.

This improves consistency in bitcoin-cli, avoiding duplicating parsing
logic.

Added a new functional test passing an option after a non-option
argument (GNU-style).

No behavior change intended.
Treat options that begin wih double dashes as RPC named
params:

bitcoin-cli createwallet mywallet --load_on_startup=true

as an equivalent to:

bitcoin-cli -named createwallet mywallet load_on_startup=true
Improve handling of long (`--`) options passed to bitcoin-cli, such as
`--datadir` and `--signet`. Previously, some code paths in CLI didn't
properly recognize or forward double-dash options, causing unexpected
argument parsing behavior or timeouts.

This change ensures that long options are processed consistently,
matching ArgsManager behavior and CLI semantics.

Example fixed case:
    /build/bin/bitcoin-cli --datadir=/tmp/btc1 --signet getblockhash 1000

Co-authored-by: w0xlt <w0xlt@users.noreply.github.com>
@pablomartin4btc pablomartin4btc force-pushed the argsman-GNU-style-command-line-option-parsing branch from 4f96ced to 34460c9 Compare December 18, 2025 17:09
@pablomartin4btc
Copy link
Member Author

Rebased.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants