Skip to content

Push-based consensus decoding #1251

@Kixunil

Description

@Kixunil

I just realized push-based consensus decoding could solve a bunch of issues:

  • Work with APIs that don't enable pull-based decoding
  • Enables trivial async deserialization
  • Potentially enable allocation-free decoding (this is somewhat orthogonal but I guess good to think about together)
  • Reduce the complexity of no-std feature more problematic than anticipated #758 to just having a simple generic Write trait

Design draft (owned decoding only):

pub trait Decodable {
    type Decoder: Decoder<Item = Self>;
    
    /// Decodes `Self` by reading from a buffered reader.
    #[cfg(any(feature = "std", feature = "core"))]
    fn from_reader<R: BufRead>(mut reader: R) -> Result<Self, IoError<Self::Error>> {
        let mut deserializer = Self::Decoder::default();
        while let Ok(buf) = reader.fill_buf()? {
            if buf.is_empty() {
                break;
            }
            let consumed = deserializer.put_bytes(buf).map_err(IoError::Decode)?;
            reader.consume(consumed);
            if consumed < buf.len() {
                 break;
            }
        }
        deserializer.end().map_err(Into::into)
    }

    /// Decodes `Self` from given slice containing *full* data.
    ///
    /// The slice must **not** be partial - use `Decoder` API to handle partial slices.
    /// Returns the number of bytes consumed as the second field of the tuple on success.
    fn from_slice(bytes: &[u8]) -> Result<(Self::Item, usize), DecodeError<Self::Error> {
        let mut decoder = Self::Decoder::default();
        let consumed = decoder.put_bytes(bytes).map_err(DecodeError::Decode)?;
        let item = decoder.end()?;
        Ok((item, consumed))
    }
}

pub trait Decoder: Default {
    type Item;
    type Error; // perhaps we should require our own error trait?

    /// Add these bytes into deserializer state.
    ///
    /// Returns the number of consumed bytes on success, error if data was corrupted.
    /// If the returned number is less than `buf.len()` then this decoder finished decoding and `end()` should be called.
    /// It will no longer accept any bytes.
    ///
    /// ## Panics
    ///
    /// The function should panic if `bytes` is empty.
    fn put_bytes(&mut self, bytes: &[u8]) -> Result<usize, Self::Error>;

    /// Finishes decoding the item.
    ///
    /// Returns decoded item or `UnexpectedEndError` if decoding didn't finish.
    fn end(self) -> Result<Self::Item, UnexpectedEndError>;
}

// not entirely correct macro, just to give you an idea
macro_rules! fixed_size_decodable {
    ($type:ty, $decoder:ident, $len:expr, $conversion_method:ident) => {
        pub struct $decoder {
            // could be MaybeUninit in the future, this should be well-contained
            // or model it with an enum
            buf: [u8; $len],
            len: usize,
        }

        impl $crate::Decoder for $decoder {
            fn put_bytes(&mut self, bytes: &[u8]) -> Result<usize, Self::Error> {
               let dst = &mut self.buf[self.len..];
               let to_copy = dst.len().min(bytes.len());
               dst[..to_copy].copy_from_slice(&bytes[..to_copy]);
               self.len += to_copy;
               Ok(to_copy)
            }

            fn end(self) -> Result<Self, UnexpectedEndError> {
                if self.len == $len {
                    Ok($type::$conversion_method(self.buf))
                } else {
                    Err(UnexpectedEndError::new(self.len, $len))
                }
            }
        }
    }
}

fixed_size_decodable!(u32, U32Decoder, 4, from_be_bytes);

I also envision MinLenDecodable and ExactLenDecodable traits to simplify decoding for some types where they'd only have to implement conversions from byte arrays.

Metadata

Metadata

Assignees

No one assigned

    Labels

    1.0Issues and PRs required or helping to stabilize the APIAPI breakThis PR requires a version bump for the next releasebrainstorm

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions