-
Notifications
You must be signed in to change notification settings - Fork 308
Are initial "buffer size" bytes in bincode2 decode_from_slice really required? #714
Description
I don't know if this is a bug, a feature request or a support request…
I am porting a .xm file parser library¹ to support no_std. This means upgrading from bincode to bincode2. The files the library opens are not created with bincode, the format is pre-existing and the library uses bincode to deserialize a struct that has the same fields in the same order as the format. Serde #[attributes] are used to make sure the offsets and such all match². With bincode1 this just works.
To convert to bincode2, I have replaced calls to bincode::deserialize(&decoded_data)?) with calls to bincode::decode_from_slice(&decoded_data, bincode::config::legacy())?.0). It is no longer able to open the same files, and the reason why is bincode2, when decoding, seems to assume that an eight-byte size is present at the beginning of the data. I believe this becuase when I try to decode one of the files that bincode1 could open, it tries to allocate an 8 exabyte buffer and the program crashes. The size of the buffer it tries to allocate is exactly equal to the first eight bytes of the file in binary.
I don't find this eight-byte header in the documentation, and there doesn't seem to be a way to opt out of using it. I don't know if this is something that is already planned to be fixed as part of bringing bincode2 to be backward compatible with bincode 1.33. I don't know if I somehow caused this myself by using decode_with_slice instead of decode_from_reader (what we have is a slice however).
What should I do to decode a file that does not have the 8 byte size header?
We are embedding bincode with bincode = { version = "2.0.0-rc.3", features=["alloc", "derive", "serde"], default-features = false }.
¹ This link is a test program that can be used to reproduce, but it will be a little nontrivial to set up. You will have to check out the PRs for both xmrs and xmrsplayer with the PR branch, then replace the reference to xmrs in the xmrsplayer Cargo.toml with xmrs = { path = "/path/to/xmrs/checkout", default-features = false } so it is using your checked out version, then run the invocation under "I broke it" at the link.
² I think currently I have this set up wrong, but it doesn't matter because currently we aren't even getting as far as parsing the first field.