I just realized there's an alternative approach to this that bypasses the async issue and seems more useful. This whole algorithm is just a part of SPV verification, so why not just implement SPV?

pub struct BlockHeaderValidator {
   // ...
}

impl BlockHeaderValidator {
    /// Creates the validator using genesis block of the network
    pub fn new(network: Network) -> Self { /* ... */ }

    /// Creates the validator for specific network starting from given block header
    pub fn from_params(first_header: &Header, params: Params) -> Self { /* ... */ }

    /// Tries to advance the validation by attaching the provided header.
    ///
    /// On success the new header is attached, on error the state is unchanged.
    pub fn next_header(&mut self, header: &Header) -> Result<(), ValidationError> { /* ... */ }
}

Then not only would downstream users not have to implement retargeting but also avoid most of the validation work. The only thing I'm unsure about is how to handle reorgs. Maybe clone the validator and verify each branch in parallel?

I just realized there's an alternative approach to this that bypasses the async issue and seems more useful. This whole algorithm is just a part of SPV verification, so why not just implement SPV?

@Kixunil can we just do it as is for now so we can get the backport done and then look into your suggestion after that?

cc @vincenzopalazzo because I wrote your name in the PR description without an @ mention.

@tcharding I'm not happy about releasing things we intend to break soon but if it's blocking someone that's acceptable to me.

@Kixunil well, we have released broken code (or at least, code which is missing functionality in the last release, and which wasn't missing that functionality before). So the "don't release APIs we intend to replace" ship has sailed and we might as well patch it up so that it's usable.

Glad it's acceptable to you.

@apoelstra I mean I would've preferred to do it right if we have an idea what the right way is but if it takes much longer and is urgent then I'm fine.

The backport at least is urgent. Arguably we don't need to do the fix on master. But I don't like the idea of backporting a change that doesn't also appear on master.

But I don't like the idea of backporting a change that doesn't also appear on master.

I agree.

I'm not happy about releasing things we intend to break soon but if it's blocking someone that's acceptable to me.

Turns out LDK may not actually need this, read the message yesterday. I want to dig into it today personally to work out exactly what we are doing re backporting for 0.30 and 0.31, bitcoincore-rpc just got released so we can do miniscript now and then the backports are the only things stopping upstream upgrades IIUC.

EDIT: ref: lightningdevkit/rust-lightning#2124 (comment)

Yeah, they don't need it but it'd still be nice to have eventually. I'm completely fine with de-prioritizing.

There is any update on this? I am kindly stuck in updating the recent bitcoin version on Nakamoto and this is blocking several projects like bdk and liana

Also if it is not a priority for the team, @tcharding could you give me some estimation? otherwise, I will resurrect my work on it

@vincenzopalazzo IIUC the recently released version with methods to compute min and max target fix lightning at least, do the other crates need something more? If yes, then I'm in favor of bumping priority here.

I am kindly stuck in updating the recent bitcoin version on Nakamoto and this is blocking several projects like bdk and liana

Que? bdk depends on Nakamoto?

This is just blocked by the required brain power to do #2181, if you need it I can prioritise that one.

Replaced by #2740