Skip to content

pow: add the API to calculate next pow#2090

Closed
vincenzopalazzo wants to merge 1 commit intorust-bitcoin:masterfrom
vincenzopalazzo:macros/calculate-next-work-requried
Closed

pow: add the API to calculate next pow#2090
vincenzopalazzo wants to merge 1 commit intorust-bitcoin:masterfrom
vincenzopalazzo:macros/calculate-next-work-requried

Conversation

@vincenzopalazzo
Copy link
Copy Markdown
Contributor

@vincenzopalazzo vincenzopalazzo commented Sep 24, 2023

This is an initial work to support a helper function to calculate the next pow inside rust-bitcoin itself.

Fixes: #1997

This is out for early feedback on API design and also possible improvement that I did not see in the algorithm implementation

@vincenzopalazzo vincenzopalazzo marked this pull request as ready for review September 29, 2023 11:19
@vincenzopalazzo vincenzopalazzo changed the title WIP: pow: add the API to calculate next pow pow: add the API to calculate next pow Sep 29, 2023
Copy link
Copy Markdown
Contributor

@junderw junderw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The calculations in this function are unrelated to the calculation of a new target.

Please fix the calculations, and if possible the spelling nits too.

@vincenzopalazzo vincenzopalazzo force-pushed the macros/calculate-next-work-requried branch from c9976c8 to ad69ac5 Compare October 4, 2023 16:56
@vincenzopalazzo
Copy link
Copy Markdown
Contributor Author

I've rebased and pushed the changes that I understood from the review. I've also added the testnet work as future work, as discussed with @apoelstra.


Regarding some of the feedback that @junderw provided, I couldn't find a solution. I was only told that my approach was incorrect, but I'm unsure how to modify my code to address the identified issues.

Could you please provide a clearer review to guide me in fulfilling your requirements?

Thank you.

@junderw
Copy link
Copy Markdown
Contributor

junderw commented Oct 4, 2023

Could you please provide a clearer review to guide me in fulfilling your requirements?

The requirements aren't special. Just "perform correct calculations."

I don't understand how much clearer I can be without re-writing your code completely.

Adding tests might help prove your calculations are right. You can even take some values from mainnet blocks and plug them in and make sure you get the same results.

@junderw
Copy link
Copy Markdown
Contributor

junderw commented Oct 4, 2023

Here's a single test to help you.

Currently it fails

running 1 test
thread 'pow::tests::test_calculate_next_work_required' panicked at 'assertion failed: `(left == right)`
  left: `179`,
 right: `386197775`',
#[test]
fn test_calculate_next_work_required() {
    use std::str::FromStr;
    let mainnet_params = Params::new(crate::Network::Bitcoin);
    // Difficulty 808416 -> 810432 on mainnet

    // Expected result bits = 0x1704e90f (bits of 810432)
    let expected = 0x1704e90f;
    // We take the timestamp of 810431 (the block before)
    let height_810431 = Height::from_consensus(810431).unwrap();
    let timestamp_810431 = 0x651bc919;

    // This closure should return the header for 808416 since 810431 - 2015 is 808416
    let fetch_header_808416 = |_height| {
        // Header for 808416
        Ok(Header {
            version: crate::block::Version::from_consensus(0x29322000),
            prev_blockhash: BlockHash::from_str(
                "000000000000000000027ecc78c2da1cc5c0b0496706baa7e4d7c80812c10bf3",
            )
            .unwrap(),
            merkle_root: crate::TxMerkleNode::from_str(
                "b920d5b5ebef4e9d106072944e0729cea8bf6defc583a7d87063041a316a757b",
            )
            .unwrap(),
            time: 0x650964b5,
            bits: CompactTarget(0x1704ed7f),
            nonce: 0xc637a163,
        })
    };
    let result = calculate_next_work_required(
        mainnet_params,
        height_810431,
        timestamp_810431,
        fetch_header_808416,
    )
    .unwrap();
    assert_eq!(result, expected);
}

@junderw
Copy link
Copy Markdown
Contributor

junderw commented Oct 4, 2023

Please see vincenzopalazzo#1 for fixes.

@junderw
Copy link
Copy Markdown
Contributor

junderw commented Oct 4, 2023

Found a bug in the Work constants #2103

@vincenzopalazzo
Copy link
Copy Markdown
Contributor Author

This calculation is ok, but the variable names should change between mid-state and final result.

You are right, I get completely confused! now review your code I get where I was going. Thanks for doing that

@vincenzopalazzo vincenzopalazzo force-pushed the macros/calculate-next-work-requried branch from 0a5e854 to a6941ae Compare October 4, 2023 21:09
@vincenzopalazzo vincenzopalazzo force-pushed the macros/calculate-next-work-requried branch from a6941ae to 754e481 Compare October 4, 2023 21:11
Copy link
Copy Markdown
Member

@tcharding tcharding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not review the logic in the calculate_next_work_required function, to do so requires me to understand the Core code since this mirrors it. It would be really nice if that was not the case, eg. what do the div by 4 and mul by 4s mean?

You can ignore me if you like, but we will need to get 2 reviewers to understand the logic.

@vincenzopalazzo vincenzopalazzo force-pushed the macros/calculate-next-work-requried branch from 0471c86 to adba1ed Compare November 1, 2023 10:16
@tcharding
Copy link
Copy Markdown
Member

tcharding commented Nov 3, 2023

Hey mate, I spent the afternoon playing with this PR. The only way I understand hard things is to code them myself so I re-wrote a new buggy version. I can write a review here but it is probably easier for you to just take a look and grab anything you like and leave out anything you don't. Then I'll re-review again.

https://github.com/tcharding/rust-bitcoin/blob/606bfe93d0268f7f29c3234ee4072b8c39ae79ee/bitcoin/src/pow.rs#L955

EDIT: Oh sick, while checking the link I saw the bug!

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think many projects will desire async support here, can we get rid of the closure or something? IIUC this is just pulling all the blocks from the retarget period so we could just accept IntoIterator<Item=I>, I: Borrow<Header>. People would then just fetch all the blocks first and compute the difficulty afterwards. (I know it's a bit wasteful but I don't have a better idea that doesn't involve confusing API workarounds.) This also makes tests much easier.

Also if we do end up with closure we should use Height in the parameter.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need all the blocks; we only need the starting and ending blocks.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I guess there's an even easier soluton where we just take those headers directly as arguments :).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we only take the first and last block [header] we put the burden of doing this loop on the user

https://github.com/bitcoin/bitcoin/blob/82ea4e787c791acbc85fd3043dd6bae038cba4f2/src/pow.cpp#L32

FTR I don't actually understand what the checks are for in this loop.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(And they have to understand that this loop is only required for testnet, so no one is going to do it.)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think many projects will desire async support here, can we get rid of the closure or something?

Does the async support require also an async caller? otherwise, it is just a blocking call, right?

But I also like the idea of passing it as an input parameter, and the @tcharding is a good point.

This is the era of choice, we must support async or we do not want it at the API level right now? I leave you the fun of choose 😄 because I am bias

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd put this inside if to more obviously avoid costly computation.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Crap, this is confusing, looks like duplicated line! :D (there's no solution, I'm just ranting)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ha, I tried twice to make this nicer and couldn't come up with anything.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OT: we really should have all_zeros as an inherent method. I think I did that in split hash triat PR.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in #2496

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're converting everything to consensus and then comparing. You could (and most likely should) just not convert and compare the strong types instead. In fact, if there's any reason not to something is probably horribly wrong.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we have comparison opcodes on CompactTarget. If we do we are planning to get rid of them. See #2110. (I see that you commented there that we should have explict named methods for the different comparison methods. Ok, but they don't exist right now.)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, the pow stuff is on the top of my list at the moment, I started on it yesterday.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, crap, so we have an actual use case for comparison, kinda undermining the point of #2110, but it's still obscure. It's also super weird that Core compares those. Makes me quite intrigued to figure out WTF is going on.

Using Eq now could be a nice way to not forget to change the code (presumably to use explicit method) when #2110 is resolved.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, yes, we have a use case within PoW computation. But there's lots of stuff we do in that computation that we don't necessarily want to expose by itself.

@tcharding
Copy link
Copy Markdown
Member

@Kixunil suggested new functions min, clamp help a lot, IMO. The code I wrote yesterday still looks a lot like C++.

@tcharding
Copy link
Copy Markdown
Member

copied from lightningdevkit/rust-lightning#2124 (comment)

So you guys need #2090 merged and backported to 0.30.1 as well as 0.31.1, right?

Is ready for review, this PR is blocking me on nakamoto, but I think I will have a followup regarding the API.

I don't see any force pushes in the last 5 days, is the github UI glitching?

@vincenzopalazzo
Copy link
Copy Markdown
Contributor Author

suggested new functions min, clamp help a lot, IMO. The code I wrote yesterday still looks a lot like C++.

What is wrong with being like C++? I do not see anything wrong with C++ style. However, the bad thing about the C++ version is that there are more than 4 people trying to understand the core code, so the only thing that would be good to improve is the documentation and the code, other black magic rust is just making the core difficult to read when the time to say "I wrote yesterday still looks a lot like Rust." will come.

I will go through the review over the weekend

@Kixunil
Copy link
Copy Markdown
Collaborator

Kixunil commented Nov 10, 2023

What is wrong with being like C++? I do not see anything wrong with C++ style.

There's a reason we use quite different style in Rust. Aside from many bugs, Rust is much more readable.

Min, max a and clamp are not black magic.

@vincenzopalazzo
Copy link
Copy Markdown
Contributor Author

Min, max a and clamp are not black magic.

This is also possible in C++ with std::min and std::max but other than that (I did not write the code that you reviewed, the commit is Co-Developed if you noted) I do not see why we should change the structure of the code that we do not understand well enough to change and simplify from the core version.

Other than using the min, man and clamp simplification I do not expect to see this function structure different from the core version, except if there is someone who will step up and take ownership of the code by also fulfilling the needs of the user libraries like nakamoto, ldk and friends

@vincenzopalazzo
Copy link
Copy Markdown
Contributor Author

I don't see any force pushes in the last 5 days, is the github UI glitching?

Sorry @tcharding I got overwhelmed by the rust-bitcoin notification and I lost it inside my notification view. You should see the force push now

@vincenzopalazzo vincenzopalazzo force-pushed the macros/calculate-next-work-requried branch 3 times, most recently from 176b249 to a0cb47b Compare November 11, 2023 09:14
@Kixunil
Copy link
Copy Markdown
Collaborator

Kixunil commented Nov 11, 2023

I, and likely most Rustceans, find those methods more readable. That's a good enough reason already. But it's not something I would block a PR for. But it's possible we'll have to change it anyway if a future version of clippy has a lint for this.

@vincenzopalazzo
Copy link
Copy Markdown
Contributor Author

    Checking secp256k1 v0.28.0
error[E0658]: use of unstable library feature 'clamp'
    --> bitcoin/src/pow.rs:1018:45
     |
1018 |     let adjusted_timespan = actual_timespan.clamp(pow_target_timespan / 4, pow_target_timespan * 4);
     |                                             ^^^^^
     |
     = note: see issue #44095 <https://github.com/rust-lang/rust/issues/44095> for more information

error: aborting due to previous error

For more information about this error, try `rustc --explain E0658`.
error: could not compile `bitcoin`

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error[E0658]: use of unstable library feature 'clamp'
    --> bitcoin/src/pow.rs:1018:45
     |
1018 |     let adjusted_timespan = actual_timespan.clamp(pow_target_timespan / 4, pow_target_timespan * 4);
     |                                             ^^^^^
     |
     = note: see issue #44095 <https://github.com/rust-lang/rust/issues/44095> for more information

error: aborting due to previous error

cc @tcharding @Kixunil

@Kixunil
Copy link
Copy Markdown
Collaborator

Kixunil commented Nov 11, 2023

Oh, it has MSRV 1.50, sorry about that. Skip that one and consider adding // TODO: update to clamp in MSRV >= 1.50

@vincenzopalazzo vincenzopalazzo force-pushed the macros/calculate-next-work-requried branch 2 times, most recently from 55828ad to c0785a2 Compare November 11, 2023 10:00
This is a initial work to support a helper function
to calculate a next pow inside rust-bitcoin itself.

Link: rust-bitcoin#1997
Co-Developed-by: @junderw
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
@vincenzopalazzo vincenzopalazzo force-pushed the macros/calculate-next-work-requried branch from c0785a2 to 3f05d67 Compare November 11, 2023 10:05
@tcharding
Copy link
Copy Markdown
Member

I do not see why we should change the structure of the code that we do not understand well enough to change and simplify from the core version.

Who is the second "we" referring to? After spending time on this last week I am more than comfortable to say I understand the core code and can re-write it in Rust style.

except if there is someone who will step up and take ownership of the code by also fulfilling the needs of the user libraries like nakamoto, ldk and friends

I can take ownership of this, I was intentionally not doing so because I didn't want to steam roll over you since you got here first. If you don't mind how this gets done and just want to see the functionality in rust-bitcoin then lets close this PR and I'll push it in through #2180. I'm super keen to get this in because LDK is waiting on it so they can upgrade.

If there is any other functionality missing that nakamoto needs please do say so, I'm working me hardest to support folk these days. I have not personally looked at the nakamoto codebase though.

To clarify, if you leave this open I'll assume you want to do the work and will review and wait for you. If you close this PR I'll push forward with #2180.

@junderw
Copy link
Copy Markdown
Contributor

junderw commented Nov 12, 2023

I think we should close in favor of 2180. I am willing to review this PR again though, if it needs reviewing.

@Kixunil
Copy link
Copy Markdown
Collaborator

Kixunil commented Nov 13, 2023

I think the sync API is much more concerning than any code style around min/max.

@vincenzopalazzo
Copy link
Copy Markdown
Contributor Author

@tcharding up to you, I really do not care who is the author of the PR or how the code will be written more rusty or not Cpp style. I care just that if I have some bugs the code is clear that I can read it without cry

The code that will be merged will have no API user, so we do not know if with this API I will be able to sync nakamoto with the network and fully validate the chain or if the user of the API needs some different UX. So my priority is to have this functionality so I can continue to do my work and provide more tests about the code.

When we have this code merged then I need to way the wallet user to update to the new version of rust bitcoin and then I can test nakamoto. So this sounds a work that will take months.

I think the sync API is much more concerning than any code style around min/max.

We can also think of moving the function in an async style due to that in the rust compiler we are doing some work to make the async programming usable. However, this is not good for people who are using not standard async IO (e.g nakamoto) or low-level API with mio and I do not like forcing this async style without having some concrete user. e.g: If nakamoto is going to use this API we do not need an async function

@Kixunil
Copy link
Copy Markdown
Collaborator

Kixunil commented Nov 13, 2023

Maybe we can modify the API to be completion based? I know it can have performance issues but if it's done right it might be fine. Like what I did with push_decode.

@tcharding
Copy link
Copy Markdown
Member

@tcharding up to you, I really do not care who is the author of the PR

Can you close this if you are ok for me to work on #2180, I don't want to close your PR.

or how the code will be written more rusty or not Cpp style. I care just that if I have some bugs the code is clear that I can read it without cry

Don't worry, writing code that is brain-dead simple to read is my thing :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: Add implementation of CalculateNextWorkRequired

5 participants