Overview

This issue describes zerocopy's high-level roadmap both in terms of goals and in terms of concrete steps to achieve those goals.

A slogan often associated with Rust is "Fast, Reliable, Productive. Pick Three." Zerocopy's mission is to make that slogan true by making it so that 100% safe Rust code is just as fast and ergonomic as unsafe Rust code.

In order to live up to that mission, we need to do the following things:

• Hold ourselves to a high standard for soundness, including in the face of future compiler changes
• Frame zerocopy in a way that is legible to various user bases, including:
  • Users who don't conceive of themselves as users of unsafe
  • Users who are especially security-conscious
  • Users who care about the crates.io ecosystem
• Identify gaps which prevent users from choosing zerocopy, and close those gaps
• Identify features which can improve the ergonomics or performance of code which uses zerocopy, and implement those features

Motivation

A user story

Imagine you are a systems programmer. Any sort of systems software will do, but we need a specific example, so let's say you're writing a networking stack. You care about your software's performance, you care about your software's correctness, and you care about your team's productivity. In order to achieve maximum performance, you want your code to do as few things as possible, and that means avoiding any situation where your data must be converted between representations in the course of processing it. For example, if you are parsing a network packet, you want to operate on the packet in-place: so-called "zero-copy" parsing (hey, that's the name of the crate!).

Your first impulse might be to use unsafe code. Perhaps you write a parsing routine like:

struct UdpHeader {
    src_port: u16,
    dst_port: u16,
    length: u16,
    checksum: u16,
}

struct UdpPacket<'a> {
    header: &'a UdpHeader,
    body: &'a [u8],
}

fn parse_udp_packet(bytes: &[u8]) -> Option<UdpPacket<'_>> {
    if bytes.len() <  {
        return None;
    }

    let (header, body) = bytes.split_at(size_of::<UdpHeader>());
    let header = unsafe { &*header.as_ptr().cast::<UdpHeader>() };
    Some(UdpPacket { header, body })
}

One of your goals is performance, and this code is fast! But you also care about your code's correctness, and you know that unsafe is notoriously difficult to get right (in fact, this implementation is unsound in two ways - can you spot them?). So you decide to be more careful. You spend the day poring over the Rustonomicon and the language reference. You find a fix some bugs in your code, and you even write a pseudo-proof of correctness in a "SAFETY" comment so that others can check your work.

#[repr(C)]
struct UdpHeader {
    src_port: [u8; 16],
    dst_port: [u8; 16],
    length: [u8; 16],
    checksum: [u8; 16],
}

struct UdpPacket<'a> {
    header: &'a UdpHeader,
    body: &'a [u8],
}

fn parse_udp_packet(bytes: &[u8]) -> Option<UdpPacket<'_>> {
    if bytes.len() < size_of::<UdpHeader>() {
        return None;
    }

    let (header, body) = bytes.split_at(size_of::<UdpHeader>());

    // SAFETY: We've validated that `bytes` is at least as long as `UdpHeader`. We know
    // that `UdpHeader` has no alignment requirement because all of its fields are `u8`
    // arrays, which don't have any alignment requirement, and it's `#[repr(C)]` so its
    // alignment is equal to the maximum of the alignments of its fields. Thus, the reference
    // we create here satisfies the layout properties of a `&UdpHeader`.
    //
    // We also know that any sequence of bytes of length `size_of::<UdpHeader>()` is a
    // valid instance of `UdpHeader` because that is true of all of its fields. That means that,
    // regardless of the contents of `bytes`, those contents represent a valid `UdpHeader`,
    // and so this conversion is unconditionally sound.
    //
    // Finally, we know that the created reference has the correct lifetime because of Rust's
    // lifetime elision rules. In particular, the type signature of this function guarantees that
    // the argument and return types have the same lifetime. Thus, the returned `UdpPacket`
    // cannot outlive the bytes it was parsed from.
    let header = unsafe { &*header.as_ptr().cast::<UdpHeader>() };
    Some(UdpPacket { header, body })
}

One of your goals is correctness, and this code is much more likely to be correct than the previous version! But you also care about your productivity, and you just spent an entire day writing a few lines of code. And what happens when you need to change the code? How much work will it take to convince yourself that a change is still correct? What if other, less experienced developers want to work on this section of code? Will they feel comfortable following your logic and feel confident in their ability to make changes without introducing bugs? So you decide to commit to never using unsafe. You modify your code to get rid of it and make whatever changes you need to get it to compile:

#[repr(C)]
struct UdpHeader {
    src_port: u16,
    dst_port: u16,
    length: u16,
    checksum: u16,
}

struct UdpPacket<'a> {
    header: UdpHeader,
    body: &'a [u8],
}

fn parse_udp_packet(bytes: &[u8]) -> Option<UdpPacket<'_>> {
    if bytes.len() < size_of::<UdpHeader>() {
        return None;
    }

    let (src_port_bytes, rest) = bytes.split_at(size_of::<u16>());
    let (dst_port_bytes, rest) = bytes.split_at(size_of::<u16>());
    let (length_bytes, rest) = bytes.split_at(size_of::<u16>());
    let (checksum_bytes, rest) = bytes.split_at(size_of::<u16>());

    let mut src_port = [0; 2];
    let mut dst_port = [0; 2];
    let mut length = [0; 2];
    let mut checksum = [0; 2];

    (&mut src_port[..]).copy_from(src_port_bytes);
    (&mut dst_port[..]).copy_from(dst_port_bytes);
    (&mut length[..]).copy_from(length_bytes);
    (&mut checksum[..]).copy_from(checksum_bytes);

    let header = UdpHeader {
        src_port: u16::from_be_bytes(src_port),
        dst_port: u16::from_be_bytes(dst_port),
        length: u16::from_be_bytes(length),
        checksum: u16::from_be_bytes(checksum),
    };

    Some(UdpPacket { header, body: rest })
}

One of your goals is productivity, and this code is easy to verify, so it was fast to write and will be fast to change in the future! But you also care about performance, and you're doing a lot more bounds checking and copying than you were before. Maybe the optimizer will improve things for you, but there's no way to be sure without benchmarking it, and even if the optimizer is smart enough this time, you might get unlucky with a future change that makes the code just confusing enough to stump the optimizer, leading to unexpected performance cliffs.

You think back on all of these attempts. You wanted fast code, so you used unsafe, but that made you worried about correctness. You also wanted correct code, so you spent a long time reasoning about your code's correctness and you wrote down that reasoning so others could check your work, but that took an entire day and resulted in code that would be slow to change in the future. You wanted to be productive, so you got rid of all of the unsafe, but that made your code slow again. It seems like you just can't win!

Moral

The moral of this story is that, when it comes to operations that touch memory directly, the Rust language and standard library are not on their own sufficient to achieve "Fast, Reliable, Productive. Pick Three." While the basic ingredients are all there, putting them together unavoidably requires sacrifices along one of the dimensions of speed, reliability, and productivity. Zerocopy aims to fill this gap. In the Design section, we outline the current state of zerocopy, identify the gaps between zerocopy's current state and its aspirational future, and outline the steps required reach that future.

Design

As mentioned above, zerocopy's mission is to make good on the slogan Fast, Reliable, Productive. Pick Three. by making it so that 100% safe Rust code is just as fast and ergonomic as unsafe Rust code. Using zerocopy, you could write the parsing code from the previous section like this:

use zerocopy::{FromBytes, Ref, Unaligned};

#[derive(FromBytes, Unaligned)]
#[repr(C)]
struct UdpHeader {
    src_port: [u8; 16],
    dst_port: [u8; 16],
    length: [u8; 16],
    checksum: [u8; 16],
}

struct UdpPacket<'a> {
    header: UdpHeader,
    body: &'a [u8],
}

fn parse_udp_packet(bytes: &[u8]) -> Option<UdpPacket<'_>> {
    let (header, body) = Ref::new_unaligned_from_prefix(bytes)?;
    Some(UdpPacket { header: header.into_ref(), body })
}

This is already a huge step above what you can do with just the standard library, and illustrates what it's like to have an API that takes care of all of this for you.

Thanks to ergonomics and safety like this, the building blocks that zerocopy provides are already being used in a diverse array of domains. Networking is zerocopy's origin and its bread and butter, but it is also used in embedded security firmware, in software emulation, in hypervisors, in filesystems, in high-frequency trading, and much more. However, it still has a ways to go before it can replace most of the unsafe code in the Rust ecosystem.

Gaps

User model

In order to identify gaps, it's helpful to say a bit about who we hope to reach with zerocopy.

Not looking to use unsafe code

A lot of use of unsafe code is by programmers who conceive of themselves primarily as trying to solve some practical problem. If they think about it at all, they think about unsafe code as a tool, not as an object of contemplation. They may have a vague sense of what the phrase "memory safe" means, and they may even know that pointers need to be aligned. They likely don't know that, in order to be able to convert a type to a byte slice, the type must not contain any uninitialized bytes, and they almost certainly have never heard of pointer provenance.

Often, these users don't know a priori that unsafe code is a tool they should consider. Instead, in trying to solve a particular problem, they may come across a crate or a Google search result which points them towards unsafe, or at least points them towards a crate which makes use of unsafe.

In order to reach users in this camp, we must:

• Frame our APIs in terms that makes sense for their use cases instead of in terms of the language semantics concepts that underlie them. For example, the AsBytes trait should speak primarily about viewing a type as bytes; details about uninitialized bytes should be saved for the "Safety" section of the doc comment.
• Advertise zerocopy in terms that these users will recognize as describing their needs. This is an area of active development, and threading the needle correctly is difficult.

Security-conscious

On the other end of the spectrum, many of our users come from domains which generally have a high bar for correctness - kernels, hypervisors, cryptography, security hardware, etc. These users are extremely wary of taking external dependencies, and only take dependencies when they absolutely need to or when they have a high degree of trust in an external software artifact.

In order to reach users in this camp, we must:

• Hold ourselves to a high standard for correctness and soundness
• Articulate this standard concisely but in sufficient technical detail that a user in this camp can come away from our docs comfortable with taking a dependency on zerocopy

Care about the open-source ecosystem

Many potential users are the authors of crates which are published on crates.io. These users have concerns which are specific to publishing software in an open-source ecosystem. For example:

• They care about API stability, especially when their use of zerocopy would be visible in their own API
• They care about compile times
• They care about the optics of relying on pre-1.0 crates

In order to reach users in this camp, we must have good open-source hygiene. We must:

• Provide the ability to disable features which are expensive to compile, especially including zerocopy-derive
• Document and test compliance with a minimum supported Rust version (MSRV)
• Decide what it would take for us to reach a 1.0 release; while versioning like this may not matter in some worlds (such as monorepos like Google's, where zerocopy was first developed), version numbers are taken as indicators of quality and stability in the open-source world. We need to think about what sorts of long-term API stability guarantees we're willing to make, and then be serious about it when we make them.

Memory model instability and zerocopy's future-soundness guarantee

Rust doesn't have a well-defined memory model. As a result, it's possible that code which is sound under today's compiler may become unsound at some point in the future. If zerocopy wants to be a trustworthy replacement for unsafe code, and ask its users not to worry about soundness, it needs to promise not only soundness, but soundness under any future compiler behavior and under any future memory model.

This work is tracked in #61.

Feature-completeness

Building-block API

impl<'a, B, T> Ref<B, T>
where
    B: 'a + ByteSlice,
    T: FromBytes,
{
    /// Converts this `Ref` into a reference.
    ///
    /// `into_ref` consumes the `Ref`, and returns a reference to
    /// `T`.
    pub fn into_ref(self) -> &'a T {
        // SAFETY: This is sound because `B` is guaranteed to live for the
        // lifetime `'a`, meaning that a) the returned reference cannot outlive
        // the `B` from which `self` was constructed and, b) no mutable methods
        // on that `B` can be called during the lifetime of the returned
        // reference. See the documentation on `deref_helper` for what
        // invariants we are required to uphold.
        self.deref_helper()
    }
}

impl<B, T> Ref<B, T>
where
    B: ByteSlice,
    T: FromBytes,
{
    /// Creates an immutable reference to `T` with a specific lifetime.
    ///
    /// # Safety
    ///
    /// The type bounds on this method guarantee that it is safe to create an
    /// immutable reference to `T` from `self`. However, since the lifetime `'a`
    /// is not required to be shorter than the lifetime of the reference to
    /// `self`, the caller must guarantee that the lifetime `'a` is valid for
    /// this reference. In particular, the referent must exist for all of `'a`,
    /// and no mutable references to the same memory may be constructed during
    /// `'a`.
    unsafe fn deref_helper<'a>(&self) -> &'a T {
        &*self.0.as_ptr().cast::<T>()
    }
}

impl<'a, B, T> Ref<B, T>
where
    B: ByteSlice + Into<&'a [u8]>,
    T: FromBytes,
{
    /// Converts this `Ref` into a reference.
    ///
    /// `into_ref` consumes the `Ref`, and returns a reference to
    /// `T`.
    pub fn into_ref(self) -> &'a T {
        let bytes = self.0.into();
        // SAFETY: `Ref` upholds the invariant that `.0`'s length is
        // equal to `size_of::<T>()`. `size_of::<ByteArray<T>>() ==
        // size_of::<T>()`, so this call is sound.
        let byte_array = unsafe { ByteArray::from_slice_unchecked(bytes) };
        // SAFETY: `Ref` upholds the invariant that `.0` satisfies
        // `T`'s alignment requirement.
        unsafe { T::ref_from_bytes_unchecked(byte_array) }
    }
}

Simplify ByteSlice's definition and make it un-sealed

Split ByteSlice so that split_at is in a different trait (#1)

Elide length or alignment checks when they can be verified statically

Tracked in #280.

Support types which are not FromBytes, but which can be converted from a sequence of zeroes

Tracked in #30.

Support fallible conversions

Tracked in #5; in progress.

Support conversions in const fn

Tracked in #115.

Support converting &[[u8; size_of::<T>()]] to &[T]

What is says on the tin.

Rename LayoutVerified to Ref (#68)

Miscellaneous features

API polish

• Inline trait methods in derive-generated code #7
• Add zerocopy::byteorder::NE typealias for zerocopy::byteorder::NativeEndian #100
• Automatically implement Unaligned without custom derive #110
• Consider adding #[must_use] annotation to some types, functions, and macros #188
• write_to_prefix (and similar ones) take ownership of the buffer #195
• For types which implement Deref, change some methods to associated functions? #210
• Consider renaming as_bytes_mut to as_mut_bytes #253

Documentation is complete, thorough, and up-to-date (#32)

• Document soundness requirements around references #8
• Test that cargo readme output matches README.md #18
• Test cargo doc in CI #33
• Add comment to README.md stating that it's auto-generated and shouldn't be edited directly #45
• Rename LayoutVerified to Ref #68
• Update AsBytes derive docs #132
• Document why we don't require a repr for an enum to be FromZeroes #146
• Unalign ABI promises are wrong #164
• Unalign::update docs should suggest DerefMut for unaligned types #262
• Document yanked releases #238
• Document that our traits opt-in to creating type instances #260

High confidence in correctness and soundness

• Test Clippy in CI #49
• Deny other Clippy lints? #50
• Use -Z randomize-layout #56
• Tracking issue for proving soundness, preventing regressions, and documenting security ethos #61
• Add defensive programming in FromBytes::new_box_slice_zeroed #64
• Use Miri's -Zmiri-symbolic-alignment-check and -Zmiri-strict-provenance #88
• Don't assume primitive alignments #116
• Test derives on generic, unsized types #129
• Add positive and negative trait impl tests for SIMD types #130
• Be aware of alignment checking issues #182
• write_to_prefix generates panic path #200
• Prevent panics statically #202
• Install OpenSSF Scorecard and consider adopting its recommendations #230
• Could Unalign::update cause issues for types that are aware of their memory location? Is this a soundness hole? #266

Tested and stable on all platforms

• Test more conditions in GitHub actions #12
• Add tests for compilation failure #17
• Run cargo miri test on wasm and riscv target once they're supported #22
• test_as_bytes_methods fails on powerpc #23
• Don't assume primitive alignments #116
• CI step "Set toolchain version" can fail without stopping CI job #225

Usable in Cargo and crates.io ecosystem

• Determine our MSRV #13
• Test in CI that we have the same MSRV in all source files #39
• Test cargo package or cargo publish --dry-run in CI? #105
• Sync zerocopy and zerocopy-derive version numbers, have zerocopy depend on an exact version of zerocopy-derive #107
• Validate cargo package contents in CI #192
• Tag, branch releases #214
• Backport non-breaking features to 0.6.x #215
• Add deprecated LayoutVerified type alias #221
• zerocopy 0.6.2 is not semver compatible with 0.6.1: use zerocopy::U32; | ^^^^^^^^^^^^^ no U32 in the root #228

Compile-time performance

• Simplify zerocopy-derive-generated code #109
• Make zerocopy-derive an optional Cargo dependency #169

Known bugs are fixed

• Enable stdsimd when building with simd-nightly feature #6
• test_new_error fails on i686 #21
• test_as_bytes_methods fails on powerpc #23
• Rust Cache warnings for tests with more than one feature enabled #43
• where_clauses_object_safety future compatibility lint warning #150
• MaybeUninit impls are unsound #299

Code quality

• Document style #16
• Remove deprecated *.out and *.err files #36
• Replace for loop with [u8]::fill method #38
• Optimize caching in CI #85
• Migrate off of deprecated GitHub Actions save-state command #173
• Split into more modules and files #252
• Compute "round down to next multiple of alignment" more efficiently #390

Developer experience

• Make it easier to run CI tests locally #69
• CI step "Set toolchain version" is flaky due to network timeouts #295
• Document private items in CI? #303
• Only run Clippy on nightly? #328
• Optimize expensive Miri tests #391
• Optimize Miri for test_validate_cast_and_convert_metadata #397