I've just learned about ControlFlow which I think would be much more clear than Result<Infallible, Result<(), Error> but the MSRV would have to be 1.55. What do you think?

Neat! I am in favor of using ControlFlow. Maybe initially we should add this API in parallel to the existing one and cfg-gate it on 1.55. It seems like on the MSRV-for-stable thread we've achieved rough consensus on something in the mid-1.60s but we're waiting on Debian and the 2-year rule to actually pull the trigger. Something of this scale I think we'd want some real-world testing before fully cutting over to it, so it could make sense to have a parallel API.

cc #1251 where push-based decoding was first discussed.

I'm not sure how much we can feature-gate it if we want to delegate the impls. (Maintaining two different implementations is annoying; though thankfully they shouldn't change much.) If we're going to do that I would limit it to one release or so.

Also I don't think we can reasonably feature gate it on Rust version - cfg in Cargo.toml doesn't support that and putting #[cfg] ins the entire stack seems too much.

Note that 1.55 is already >2 years old (September 2021) and 1.63 is in Debian stable. But if we're going to bump it I'd jump to 1.56 which is still 2 years old and supports edition 2021 (we don't have to use it right away but it's nice to have).

Sounds good to me. cc @tcharding is there an existing issue that we should glom onto to discuss MSRV bumps, or should @Kixunil just make a PR to bump to 1.56 and we'll try to get a bunch of ACKs.

cc #1722.

Lets PR and push forward! I tried bringing up Rust 1.57 a few months ago but I was too early because it was not yet 2 years old, #1983

I'd like to see 1.56, edition 2021 would make my life a bunch easier.

I'm keen for this PR. I don't have any comments or suggestions other than that I'm afraid.

Just in case you go to do it @Kixunil, I did the MSRV bump #2188

I'm keen for this PR. I don't have any comments or suggestions other than that I'm afraid.

That's fine. I was mainly checking that my approach isn't viewed as horribly complicated or something like that.

Updated to use ControlFlow, edition 2021, and a new method I added to clean up the code a bit.

Rebased and a bit improved but not really ready. Working on encoding now.

Just rebase, will address the review later.

Pull Request Test Coverage Report for Build 10075949365

Details

• 726 of 958 (75.78%) changed or added relevant lines in 14 files are covered.
• 23 unchanged lines in 4 files lost coverage.
• Overall coverage decreased (-0.3%) to 82.914%

Totals
Change from base Build 10059904982:	-0.3%
Covered Lines:	20226
Relevant Lines:	24394

💛 - Coveralls

Also I'd love to rebase this on top of #3100 (not that I love rebasing this monstrosity...)

Concept ACK.

Just spent whole morning trying to understand. But after a few hours it started to click. Pretty cool!

TLDR: It's all basically Rust version of Haskell's binary package :-)

data Decoder a = Fail !B.ByteString String
              | Partial (Maybe B.ByteString -> Decoder a)
              | Done !B.ByteString a
              | BytesRead !Int64 (Int64 -> Decoder a)

newtype Get a = C { runCont :: forall r.
                               B.ByteString ->
                               Success a r ->
                               Decoder   r }

type Success a r = B.ByteString -> a -> Decoder r

instance Monad Get where …

But after a few hours it started to click.

I knew I wanted to expand the documentation a lot but damn, I didn't expect it to take people hours without it. :(

Good to know.

I didn't expect it to take people hours without it. :(

There are many details to understand, part of them in push_decode which I was not familiar with; for example: in transaction decoder, what do sub_decode + wrap_sub_decode do and why? how do all the macros work? etc. It takes a while to understand.

I don't think it's problem of documentation, rather it's the size and (inherent) complexity.

Yeah, the thing is, I hope it won't be required to understand all of it to get benefits. Basically understand the basic idea, have examples how to use it and some examples how to implement decoders/encoders. sub_decode and wrap_decode are indeed a bit weird but pretty niche and thankfully also somewhat nice.

Side note: I had some other ideas that would make the crate even bigger but would result in simpler usage. Some of those ideas are quite crazy though :D.

We did https://crates.io/crates/bitcoin-consensus-encoding