Skip to content

[0.32] Bound decode methods on Read, rather than BufRead#3173

Merged
apoelstra merged 1 commit intorust-bitcoin:0.32.xfrom
TheBlueMatt:2024-08-normal-rust
Oct 16, 2024
Merged

[0.32] Bound decode methods on Read, rather than BufRead#3173
apoelstra merged 1 commit intorust-bitcoin:0.32.xfrom
TheBlueMatt:2024-08-normal-rust

Conversation

@TheBlueMatt
Copy link
Copy Markdown
Member

The Rust world has, sadly, standardized on Read for the type from which objects are deserialized. This makes forcing users into BufReads in a library crate somewhat impractical - a rust-bitcoin user may implement some deserialization trait provided by a third-party and be stuck with a Read. Now, to read a rust-bitcoin object they'll have to wrap the Read in some BufRead wrapper that only buffers single bytes, wrecking their performance. While the user who is using this library should pass a BufRead (and may do so), the library using rust-bitcoin may not have any choice in the matter.

Thus, here, we standardize on the same trait that the entire Rust world has standardized on, using Read for bounds in deserialization methods rather than BufRead.

Here we avoid implementing bitcoin_io::Read for all std types to encourage users who do have control over the types being passed to use BufReader where possible, though this decision should be revisited once we have feedback from downstream users.

I only bothered to implement this for 0.32 here (we'll have to do something similar upstream, but its super easy to just rebuild this patch and is probably less error-prone than trying to backport, plus we can discuss further if we want to do this on other version(s)).

See #3172 for further discussion.

The Rust world has, sadly, standardized on `Read` for the type
from which objects are deserialized. This makes forcing users into
`BufRead`s in a library crate somewhat impractical - a
`rust-bitcoin` user may implement some deserialization trait
provided by a third-party and be stuck with a `Read`. Now, to read
a `rust-bitcoin` object they'll have to wrap the `Read` in some
`BufRead` wrapper that only buffers single bytes, wrecking their
performance. While the user who is using this library should pass a
`BufRead` (and may do so), the library using `rust-bitcoin` may not
have any choice in the matter.

Thus, here, we standardize on the same trait that the entire Rust
world has standardized on, using `Read` for bounds in
deserialization methods rather than `BufRead`.

Here we avoid implementing `bitcoin_io::Read` for all `std` types
to encourage users who do have control over the types being passed
to use `BufReader` where possible, though this decision should be
revisited once we have feedback from downstream users.
@github-actions github-actions bot added the C-bitcoin PRs modifying the bitcoin crate label Aug 15, 2024
Copy link
Copy Markdown
Collaborator

@Kixunil Kixunil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This severely conflicts with #2184 which requires BufRead for performance. You should implement #3172 (comment) instead if you really want that. Also this comes off as aggressive since the conversation is not resolved.

@TheBlueMatt
Copy link
Copy Markdown
Member Author

Note that this is against the 0.32 branch and intended for a point release, so it conflicting with other PRs isn't a big deal. We can do something smarter, like you link to, for new releases.

@coveralls
Copy link
Copy Markdown

Pull Request Test Coverage Report for Build 10406462165

Details

  • 55 of 59 (93.22%) changed or added relevant lines in 21 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 83.113%

Changes Missing Coverage Covered Lines Changed/Added Lines %
bitcoin/src/consensus/encode.rs 15 17 88.24%
bitcoin/src/p2p/message.rs 3 5 60.0%
Totals Coverage Status
Change from base Build 9429226336: 0.0%
Covered Lines: 19200
Relevant Lines: 23101

💛 - Coveralls

@Kixunil
Copy link
Copy Markdown
Collaborator

Kixunil commented Aug 15, 2024

Let's not change it just to change it again later.

@TheBlueMatt
Copy link
Copy Markdown
Member Author

TheBlueMatt commented Aug 15, 2024

This isn't a change, this is an API relaxation for the current version as people upgrade to it. And, honestly, not sure we'll ever upgrade past 0.32, so I don't care too much about future breakage :).

@Kixunil
Copy link
Copy Markdown
Collaborator

Kixunil commented Aug 15, 2024

I don't care too much about future breakage

You might not but I do.

@TheBlueMatt
Copy link
Copy Markdown
Member Author

Sure, if you care about future breakage and are already on 0.32 this doesn't effect you at all :)

@Kixunil
Copy link
Copy Markdown
Collaborator

Kixunil commented Aug 15, 2024

I was thinking that this cannot be really non-breaking and add the proper fix because of trait objects. So instead I suggest these changes:

  • add consensus_decode_slow() method to Decodable using the Read bound call into the fast one with one-byte buffer
  • by default call consensus_decode_slow() from the fast method
  • Move the current implementations that don't use the buffer from fast method to slow one

Downstream users that want to use Read will call into slow and they can change it later if they want to.

I offer making a PR for this.

@apoelstra
Copy link
Copy Markdown
Member

I'd prefer _unbuffered rather than _slow. It is both less loaded and takes longer to type, so it should discourage use while being easier to find. But concept ACK doing that.

@Kixunil
Copy link
Copy Markdown
Collaborator

Kixunil commented Aug 15, 2024

The doc should still say it may be slower though.

@TheBlueMatt
Copy link
Copy Markdown
Member Author

I was thinking that this cannot be really non-breaking and add the proper fix because of trait objects.

Oh? I don't see why this is breaking.

@apoelstra
Copy link
Copy Markdown
Member

I think it might be ok. This code works on the playground:

fn myfn<T: std::io::Read + ?Sized>(test: &mut T) {
    // blah blah
}

fn myfn2<T: std::io::BufRead + ?Sized>(test: &mut T) {
    // blah blah
}

fn main() {
    let mut works_with_old_code = std::io::BufReader::new(std::io::stdin());
    
    myfn2(&mut works_with_old_code as &mut dyn std::io::BufRead);
    myfn(&mut works_with_old_code as &mut dyn std::io::BufRead);
}

@TheBlueMatt
Copy link
Copy Markdown
Member Author

TheBlueMatt commented Aug 15, 2024

Yea, I went and did this and it seems to work:

trait A {}
trait B: A{}
struct BI {}
impl A for BI {}
impl B for BI {}

fn fb<T: B + ?Sized>(t: &T) {}
fn fa<T: A + ?Sized>(t: &T) {}

fn dynb(t: &dyn B) { fa(t); }

fn main() {
    let b = BI {};
    fb(&b);
    fa(&b);
    let bt: &dyn B = &b;
    fa(bt);
    fb(bt);
}

@Kixunil
Copy link
Copy Markdown
Collaborator

Kixunil commented Aug 15, 2024

I meant adding the associated types.

@TheBlueMatt
Copy link
Copy Markdown
Member Author

Ah, right, okay. Yea, the "proper fix" (assuming we don't end up on a push-based decode total API rewrite anyway) I assume we'd want to do in a new release anyway, leaving this for 0.32.

@apoelstra
Copy link
Copy Markdown
Member

I would like to do this in 0.32.

I think that a proper fix would involve having an API that can take either a Read or a BufRead and (potentially) does different things depending what's available. (As I said in the other issue I don't think it's reasonable to force direct users to propagate BufRead bounds outward, nor is it reasonable to force them to use a Read->BufRead adaptor that "buffers" a byte at a time.)

So we will need to break things relative to master anyway, and changing 0.32 to use Read won't result in extra breakage.

@TheBlueMatt
Copy link
Copy Markdown
Member Author

The failing CI look unrelated to this PR.

/// performance hit.
#[inline]
fn consensus_decode_from_finite_reader<R: BufRead + ?Sized>(
fn consensus_decode_from_finite_reader<R: Read + ?Sized>(
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've just realized this is strictly a breaking change since people who started to rely on it in their own implementations would get broken by it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO making the API usable is still worth it. I am fairly skeptical people implemented Decodable, but if we're super worried about it we should ship an 0.33 with just this change.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we ship 0.33 then I'd also do the breaking bitcoin-io change. But I still prefer to have the unbuffered methods.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's ok to break the API in this way. Agreed that nobody has implemented the Decodable trait themselves.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should've made them sealed. :(

Copy link
Copy Markdown
Member

@apoelstra apoelstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 79df48e successfully ran local tests

Copy link
Copy Markdown
Member

@tcharding tcharding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 79df48e

@tcharding
Copy link
Copy Markdown
Member

tcharding commented Oct 15, 2024

I rebased this locally on top of #3418 and it seems good to go in on top of that.

@tcharding tcharding dismissed Kixunil’s stale review October 15, 2024 03:03

@Kixunil is AWOL at the moment.

@apoelstra apoelstra merged commit b5e9fc4 into rust-bitcoin:0.32.x Oct 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

C-bitcoin PRs modifying the bitcoin crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants