Skip to content

Add difficulty adjustment calculation#2740

Merged
apoelstra merged 1 commit intorust-bitcoin:masterfrom
rustaceanrob:difficulty-adjustment
May 15, 2024
Merged

Add difficulty adjustment calculation#2740
apoelstra merged 1 commit intorust-bitcoin:masterfrom
rustaceanrob:difficulty-adjustment

Conversation

@rustaceanrob
Copy link
Copy Markdown
Contributor

@rustaceanrob rustaceanrob commented May 3, 2024

Hi, I hit a roadblock with the current pow API. As far as I can tell, the only workaround to calculate the next work required similar to bitcoin/src/pow.cpp is to use a general big integer library, convert the Target to bytes, do the math, and convert back to Target from bytes. I have also been working with Floresta and their solution was to fork off and exposed the U256 struct publicly on their branch. I think these home brewed difficulty adjustment solutions will continually pop up, so I created a from_next_work_required method to return a Target. My work veers significantly from #2180, as I only provided a single method to do so, without further guidance on when exactly this retarget occurs.

I am happy to add tests once I get further direction from maintainers if this as a likelihood of being accepted or not. Thanks.

@github-actions github-actions bot added the C-bitcoin PRs modifying the bitcoin crate label May 3, 2024
@coveralls
Copy link
Copy Markdown

coveralls commented May 3, 2024

Pull Request Test Coverage Report for Build 9089264878

Details

  • 126 of 127 (99.21%) changed or added relevant lines in 1 file are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.1%) to 83.318%

Changes Missing Coverage Covered Lines Changed/Added Lines %
bitcoin/src/pow.rs 126 127 99.21%
Totals Coverage Status
Change from base Build 9088761779: 0.1%
Covered Lines: 19303
Relevant Lines: 23168

💛 - Coveralls

@apoelstra
Copy link
Copy Markdown
Member

cc #1820 which added some different-but-related missing functionality

@apoelstra
Copy link
Copy Markdown
Member

In 206bc621d34f0ef25b5d91a512d2e67e635f77e4:

There is trailing whitespace on several lines that I'd like you to remove. You can find them with git grep ' $'.

And could you add a couple unit tests, maybe by taking headers from the actual blockchain, that demonstrate that the adjustment does the right thing under normal circumstances, when it hits the "minimum 1 difficulty" rule, and when it hits one of the factor-of-4 rails? (The latter has never happened on Bitcoin but it should be easy to gin up a test vector by editing block timestamps, since we're not validating the PoW here.)

@apoelstra
Copy link
Copy Markdown
Member

But concept ACK -- this functionality is missing AFAICT. Thanks for your contribution!

@rustaceanrob rustaceanrob force-pushed the difficulty-adjustment branch from 206bc62 to ed5574f Compare May 8, 2024 05:37
@rustaceanrob
Copy link
Copy Markdown
Contributor Author

I approach NACKed myself over the weekend. After thinking about it I decided that the Bitcoin Core approach is less susceptible to user misinterpretation. Also when considering the difficulty adjustment it is more appropriate to be dealing with CompactTarget since it is used in the headers and has lower precision. I updated the documentation with reference to the Bitcoin Core implementation and an example. Also added 5 test cases for each scenario.

Copy link
Copy Markdown
Member

@tcharding tcharding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this man! I was no where near getting back to #2180 and I really wanted to deliver this functionality in the next release, so thank you.

My review as mostly styleistic but includes a question about the function argument choice - that is the most important thing in the review I believe.

Thanks again

@tcharding
Copy link
Copy Markdown
Member

I couldn't manage to get a link to a line number for some reason so I did this instead:
https://github.com/rust-bitcoin/rust-bitcoin/pull/2180/files#r1594975932

This is what I meant by my question about the timespan parameter pushing the knowledge of the off-by-one error onto users.

@apoelstra
Copy link
Copy Markdown
Member

@tcharding I don't understand exactly what you're suggesting. Whether the user passes a header or header.timestamp does not affect whether they will be burned by the off-by-one error.

It seems like our choices are either (a) take the whole blockchain, or at least the most recent few thousand blocks, and compute the block intervals ourselves, protecting the user, or (b) take just the two relevant blocks, trusting the user to choose the right ones.

This PR currently has both (though maybe they could be named more clearly to distinguish them), and I think we need both, since we don't want to require the user to have tons of blockheaders available when they're not even relevant.

@tcharding
Copy link
Copy Markdown
Member

tcharding commented May 9, 2024

@tcharding I don't understand exactly what you're suggesting. Whether the user passes a header or header.timestamp does not affect whether they will be burned by the off-by-one error.

Mine is just a question on API design and whether we want/need the additional functionality next_target (#2180) provides. I think this PR is good.

In #2180 the user passes current header to next_target, that function then either returns the header.bits if the next block is not a retarget block or does the calculation by calling calculate_next_target, the caller does not need to be aware of the the off-by-one stuff.

FTR #2180 suffers the same issue I'm describing because both CompactTarget::from_next_work_required (this PR) and calculate_next_target (#2180) are basically the same. @rustaceanrob has done better than me already by documenting it, I just cargo-cult-programmed the -1.

@apoelstra
Copy link
Copy Markdown
Member

Ok, yes, you can encapsulate this by requiring the user provide an accessor for every single block in the chain. (Though you can cheat by knowing exactly which blocks it is going to access, of course.)

This PR does this, but also provides a method in which you only need to have the two relevant headers. So I think it's strictly better.

@apoelstra
Copy link
Copy Markdown
Member

ACK ed5574f5b511ca803a9fb041d820631e7ae57158 except that we should restore the "just pass the whole chain" functionality that was present in the original PR.

@rustaceanrob
Copy link
Copy Markdown
Contributor Author

rustaceanrob commented May 10, 2024

restore the "just pass the whole chain" functionality

Okay, back to two Header? I was struggling to find good names for them where the ordering is unambiguous. How is current, previous?

@apoelstra
Copy link
Copy Markdown
Member

I think you want something like current and last_epoch_boundary.

@apoelstra
Copy link
Copy Markdown
Member

Okay, back to two Header?

Yeah, I think we should have both.

@rustaceanrob rustaceanrob force-pushed the difficulty-adjustment branch from ed5574f to 4114394 Compare May 11, 2024 03:19
@apoelstra
Copy link
Copy Markdown
Member

In 0aa279aa33791dcc462f90e1574bd8d443871fbb:

  • If you are going to cite Core line numbers, please specify which commit of Core you are using
  • Sorry, I meant that we should have both a "compute from two headers" method (and I don't care if it takes timestamps or whole headers or what) and also a "compute from all the headers" method (which chooses the right headers to use).

@rustaceanrob
Copy link
Copy Markdown
Contributor Author

With the compute from all the headers approach, I've only come up with two solutions, both concerned with chain height. The only height-implicit approach I know of is to pass a &[Header] representing the current header chain. For low resource devices that might be a challenge, but then again the other methods of this PR would have them covered if they take 5 minutes to research how the adjustment works. Otherwise, the explicit approach is to pass a current chain height and some function to index into the chain at a specific height as in #2180. Implied in that approach would be pinning down the Height structure. I am compelled to just go with &[Header], is that alright?

@tcharding
Copy link
Copy Markdown
Member

I'm leaning towards thinking this is mergable as is and we do the additional functionality in a separate PR. This PR is already a big improvement.

@apoelstra
Copy link
Copy Markdown
Member

The only height-implicit approach I know of is to pass a &[Header] representing the current header chain.

You can take a closure which takes a usize and returns the header at that index. We have an existing PoW-related function that does this, I forget which one.

@tcharding

I'm leaning towards thinking this is mergable as is and we do the additional functionality in a separate PR. This PR is already a big improvement.

Sure, I'm happy to do this -- but we're a little ways from release and we're close to having this PR having the API we want. It's up to you @rustaceanrob. If you want me to just merge this I'll do it.

@rustaceanrob rustaceanrob force-pushed the difficulty-adjustment branch 2 times, most recently from 6536b5e to 54b5456 Compare May 14, 2024 07:10
@rustaceanrob
Copy link
Copy Markdown
Contributor Author

If you are going to cite Core line numbers, please specify which commit of Core you are using

Updated to included the specific commit

Sure, I'm happy to do this -- but we're a little ways from release and we're close to having this PR having the API we want. It's up to you @rustaceanrob. If you want me to just merge this I'll do it

I think this is an appropriate stopping point for now. Users are able to perform the adjustment and any additions would be additive. I am unsure how much time I will have to work on this in the next couple days. I am envisioning something similar to #2180 where there is a nice FnMut that takes a Height. In a follow up PR we can discuss this further.

apoelstra
apoelstra previously approved these changes May 14, 2024
Copy link
Copy Markdown
Member

@apoelstra apoelstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 54b5456785568f29bc71971bf3cbd7fee7663edb

tcharding
tcharding previously approved these changes May 14, 2024
Copy link
Copy Markdown
Member

@tcharding tcharding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excluding the docs typo

ACK 54b5456785568f29bc71971bf3cbd7fee7663edb

@rustaceanrob rustaceanrob dismissed stale reviews from tcharding and apoelstra via 47dc4a3 May 15, 2024 02:50
@rustaceanrob rustaceanrob force-pushed the difficulty-adjustment branch from 54b5456 to 47dc4a3 Compare May 15, 2024 02:50
Copy link
Copy Markdown
Member

@tcharding tcharding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 47dc4a3

@tcharding
Copy link
Copy Markdown
Member

git range-diff didn't work, I had to use the Compare link above.

Copy link
Copy Markdown
Member

@apoelstra apoelstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 47dc4a3 used range-diff

@apoelstra apoelstra merged commit 44a6402 into rust-bitcoin:master May 15, 2024
@rustaceanrob rustaceanrob deleted the difficulty-adjustment branch May 15, 2024 17:50
@tcharding
Copy link
Copy Markdown
Member

tcharding commented May 15, 2024

used range-diff

Bollocks :)

@Davidson-Souza
Copy link
Copy Markdown

@apoelstra @tcharding how hard would it be to backport this? I want to get rid of my out-of-tree version of bitcoin in floresta and this would be very useful to have on 0.32

@apoelstra
Copy link
Copy Markdown
Member

I think it'd be super straightforward. @tcharding can you open a PR? I suspect you can just cherry-pick this commit.

@Davidson-Souza
Copy link
Copy Markdown

Thanks @apoelstra . If @tcharding can't do it I can give it a shot.

@apoelstra
Copy link
Copy Markdown
Member

@Davidson-Souza that would be great actually, since then both of us can ACK it and merge it quickly.

@tcharding
Copy link
Copy Markdown
Member

tcharding commented Oct 18, 2024

Hey @apoelstra I see you added a new label, we have three backport labels already that are version specific. When I added those (eg port-0.32.x) I thought we would add all three if needed and remove them as we did the backports.

@apoelstra
Copy link
Copy Markdown
Member

@tcharding oh, good to know, thanks. I will rename them so they have "backport" in the name and drop mine.

@apoelstra apoelstra added port-0.32.x Needs porting to 0.32.x branch and removed Needs Backport labels Oct 18, 2024
apoelstra added a commit that referenced this pull request Oct 21, 2024
6624b94 feat(pow): add difficulty adjustment calculation (Rob N)

Pull request description:

  As discussed in #2740 (comment) this is a backport of #2740 to 0.32.x. I've simply cherry-picked the commit and all tests are passing locally.

ACKs for top commit:
  tcharding:
    ACK 6624b94
  apoelstra:
    ACK 6624b94; successfully ran local tests

Tree-SHA512: 27c122a051b590cdae66557044fcd72c417cf5d4ffc2bbf2d334b06f8f773038990168c116de3a026b08de6eb833d049eb510b58bb179dfd5add2245b37450a3
@tcharding
Copy link
Copy Markdown
Member

Backported in #3494

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

C-bitcoin PRs modifying the bitcoin crate port-0.32.x Needs porting to 0.32.x branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants