Tyler Mandry was quick to point out that this looks just like contexts/capabilities.

In this post I’ll explore the underlying question: what if trait bounds also carried values?

Contexts and Capabilities

We’ll start with a Rust feature idea I’ve been giddy about since I came across it, well-presented by Tyler in his blog post. I recommend giving it a read, but I’ll summarize the core idea.

The feature has three components:

1. You declare a global name with capability my_capability;;
2. You can now write my_capability: Type in a where bound, and this works like an implicit argument: the compiler will pass you a value and error if it can’t find one;
3. A value is provided for a given scope by writing: with my_capability = some_value() { ... }.

This is particularly awesome in trait impls:

capability arena;

impl<'a> Deserialize for &'a Foo
where
    arena: &'a BasicArena,
{
    ...
}

fn main() -> Result<(), Error> {
    let bytes = read_some_bytes()?;
    with arena = &arena::BasicArena::new() {
        let foos: Vec<&Foo> = deserialize(bytes)?;
        println!("foos: {:?}", foos);
    }
    Ok(())
}

What happens here is that the dictionary¹ for <&Foo as Deserialize> now also carries a runtime value. The compiler threads it through any intermediate functions/impls that have a T: Deserialize bound, without them needing to know about it (well, see next section).

Dictionaries carry values now

Implicitly threading values between unsuspecting functions does change things a bit, of course.

Controlling the implicit value

As with any Rust generics, we’ll need a way to control a bit which values we can support. Tyler proposes:

fn deserialize_and_print_later<T>(deserializer: &mut Deserializer)
where with('static + Send) T: Deserialize
{ ... }

In our dictionary world, we may just as well write:

fn deserialize_and_print_later<T>(deserializer: &mut Deserializer)
where
    T: Deserialize,
    <T as Deserialize>: 'static + Send,
{ ... }

where <T as Deserialize> is understood to refer to the dictionary itself, so we may apply bounds on it like any other type².

Linearity

Perhaps the craziest consequence of taking this seriously is that trait bounds need ownership semantics now. Imagine:

impl<'a> Deserialize for &'a Foo
where
    arena: &'a mut Vec<Foo>,
{
    ...
}

Now the dictionary contains a &mut, so we better be careful not to pass it to two functions at once! A function like the following cannot work on our &Foo:

fn bar<T: Deserialize>(bytes: Vec<u8>) {
    // The iterator needs to capture the `&mut` context.
    for item in whatever(bytes).map(|x| T::deserialize(x)) {
        // Trying to use it here too is an aliasing violation.
        let other_item = T::deserialize(something_else(bytes));
        ...
    }
}

We therefore need to distinguish the 4 kinds of ownership semantics we can encounter:

• Today’s default, with no implicit value at all: <T: Trait>: const³;
• &Context-like semantics: <T: Trait>: Copy;
• &mut Context-like semantics: <T: Trait>: Reborrow (using the Reborrow trait from the project goal)⁴;
• Box<Context>-like semantics: <T: Trait> can contain anything.

You’ll have recognized the similarity with the 4 closure traits FnStatic, Fn, FnMut, and FnOnce.

Oh and for the owned case, trait bounds can have significant Drop if not used :3

This is unhinged enough that I’d propose we just limit contexts to being Copy and use interior mutability, but I suspect some delicious APIs could be cooked with the full expressivity 👀. Plz share in the comments I wanna see them.

Methods are closures

Speaking of closure traits, methods are closures now: in our example <T as Deserialize>::deserialize makes use of the implicit parameter, so it cannot be cast to a fn(D) -> Result<.., ..> function pointer.

Depending on the ownership semantics of <T as Deserialize>, its methods will implement the corresponding Fn* closure trait(s) instead.

Scoped impls

As Tyler points out in his blog post, trait bounds can no longer be taken to be global facts! Depending on which capabilities are in scope, the same MyType: Trait may or may hold.

This is a surprisingly expressive new capability, especially if we stretch the feature set a bit:

struct MagicPointer<'a>(PhantomData<&'a ()>);

capability pointer_target;

// This `Deref` impl is only available when the capability is in scope,
// and it has a different target type depending on scope!
impl<'a> Deref for MagicPointer<'a>
where
    pointer_target: impl Sized + 'a // Can be basically anything
{
    type Target = type_of!(pointer_target); // I cheat, don't tell
    fn deref(&self) -> &Self::Target {
        &pointer_target
    }
}

This has far-reaching consequences on how we use traits:

struct MyInt(u32);

capability salt;

// This impl is correct inside a given context. But switching contexts breaks it:
// two equal values may hash differently in different contexts.
impl Hash for MyInt
where
    salt: u32
{
    ... // hash `self.0.xor(salt)`
}

fn main() {
    let mut set: HashSet<MyInt> = Default::default();
    with salt = 42 {
        set.insert(0);
    }
    with salt = 10 {
        if set.contains(&0) {
            // completely not clear whether that's the case.
            // depends on impl details
        }
    }
}

Either the Hash impl above is deemed invalid (seems likely), or datastructures like HashSet would not opt-in to scoped impls. Either way, this opens up a new dimension of expressivity.

Capturing impls

I introduced the article with context/capabilities, but this is not the only way I can think of to make trait bounds carry values. The other one is to have impls capture from their context!

For this, I’ll reuse the idea of move expressions, except I prefer to call them capture expressions. We’ll also still need a notion of scoped impls, I’ll write that local impl.

struct Context;

trait GimmeArena {
    // The crazy lifetime syntax would mean "borrows from the trait dictionary".
    fn gimme() -> &'{Self as GimmeArena} Arena;
}

fn use_arena()
where
    Context: GimmeArena
{
    let arena = Context::gimme();
    // use the arena
}

fn foo() {
    let arena = Arena;

    local impl GimmeArena for Context {
        fn gimme() -> &'{Self as GimmeArena} Arena {
            capture(&arena)
        }
    }

    // In this scope, `Context` implements `GimmeArena`, and the dictionary
    // carries a reference to the arena.
    use_arena();
}

fn bar() {
    // Different scope, so we can make another impl.
    local impl GimmeArena for Context {
        fn gimme() -> &'{Self as GimmeArena} Arena {
            ... // do something else
        }
    }

    use_arena();
}

Feature-wise, this is pretty similar: where Context: GimmeArena is very close to where arena: &'a Arena from before.

This also clashes with another understanding of what a “capturing impl” might be, namely where the captured values are accessible from the &mut self argument, which would allow conveniently defining Iterators, visitors etc.

All in all I’m not too sold on this; I’m showing it because it’s a good illustration that the important notion either way is data-carrying impls.

Conclusion

I hope I got you excited about capabilities, and/or about trait-bounds-as-values! What I find compelling is how naturally “trait bounds carry runtime values” interacts with the rest of the language.

I’ll see you later for more explorations of dictionary-passing-style traits.

You might find this weird: the two parameters feel a bit useless since they’ll always be the same. And you’d be right: them being the same is the whole point of this being an equality. The way this works is that if you need a proof that two types are equal then you just take a a == b a argument.

So what can you do with an a == b? My favorite definition is the one from the HoTT book: from the definition of the type, we automatically get a “destructor” function¹ :

transport: (a: Type) -> (b: Type) -> (f: Type -> Type) -> a == b -> f a -> f b

This function says: if a == b, then I can turn a f a into a f b for any function f. What I find insanely cute, and what compelled me to write this, is that this single function is enough to define a reasonable notion of equality.

Here is a proof of symmetry, in a suspiciously Rust-looking dependent lambda-calculus:

let symmetry(a: Type, b: Type, ab: a == b) -> b == a =
    transport a b (|x: Type| x == a) ab (refl a)

Here’s how it works: the function being transported is |x| x == a. So given a == b, transport will turn a == a into b == a. We can build a a == a using refl, so we win!

Here is transitivity:

let transitivity(a: Type, b: Type, c: Type, ab: a == b, bc: b == c) -> a == c =
    (transport b c (|x: Type| a == x) bc) ab

The idea is similar: we transport |x| a == x from b to c.

That’s all I had to say, I don’t know why this feels so good to my brain but now you can taste it too! If you enjoyed this, go read the HoTT book.

This article is part of an collaboration with the Rust Types team where we’re looking into integrating these ideas into the Rust compiler to make it more robust and more correct.

I’m like babby in the realm of trait solving internals, expect this to be directionally right but missing some important caveats.

In Rust, traits assign behaviors to types in such a way that when I write my_val.clone(), the compiler can figure out the right code to call automatically.

This process is called “trait solving”: given all the traits and all the trait impls found in the current crate and its dependencies, the trait solver figures out for each trait reference whether the trait is indeed implemented for that type, and if so where to find the required methods/associated types/associated constants.

trait Clone {
    fn clone(&self) -> Self;
}

impl Clone for u32 { ... }
impl<T: Clone> Clone for Vec<T> { ... }

let my_vec: Vec<u32> = vec![0, 1, 2];
my_vec.clone(); // uses the two impls above together

In this article I’d like to flesh out an obvious-in-retrospect idea that makes it easier to talk about what it is that the trait solver does.

“Dictionary-Passing Style”

The idea is simple: traits are like struct definitions¹, impls are like struct values, and trait bounds are how you pass such structs from one function to the next. Taking the Clone example above, we can understand it as:

struct Clone<Self> {
    clone: fn(&Self) -> Self,
}

const CLONE_U32: Clone<u32> = Clone {
    clone: |x: &u32| -> u32 { *x },
}

// I'm imagining generic consts because why not. See how the trait bound becomes a
// const generic argument.
const CLONE_VEC<T, const CLONE_T: Clone<T>>: Clone<Vec<T>> = Clone {
    clone: |vec: &Vec<T>| -> Vec<T> { // Dummy clone impl, it's probably not even that bad
        vec
            .iter()
            .map(|x: &T| CLONE_T.clone(x))
            .collect()
    }
}

let my_vec: Vec<u32> = vec![0, 1, 2];
(CLONE_VEC::<u32, CLONE_U32>).clone(&my_vec);

The interesting part is that a trait bound like where T: Clone becomes an argument, something that must be provided by the caller. This reflects trait solving: calling clone on Vec<T> requires the trait solver to figure out whether and why T: Clone.

With a program in this form, it’s easy to track what code gets called where: we just have to follow those impl structs as they’re passed down through function calls and other impls.

[!NOTE] While not necessarily involving actual struct values being passed around, this way of seeing traits/typeclasses/implicit modules is known as “dictionary-passing style” (at least in the Haskell literature) so we’re reusing this standard terminology.

Trait Solving is Elaboration

Armed with this metaphor, we can answer the question: “what does trait solving do?”. The answer: it finds for each trait clause a corresponding trait proof, which may come from an impl, a trait bound, or other sources I haven’t mentioned yet.

We could imagine a Rust that has syntax to express this². From our example we see that the main ingredients are:

• Giving a name to impls,
• Giving a name to trait bounds,
• Passing a trait proof to an item that has a corresponding trait bound.

If you’ll allow me to pull a whole bunch of syntax out of my hat:

trait Clone {
    fn clone(&self) -> Self;
}

// This syntax gives a name to the impl.
impl "clone_u32" Clone for u32 { ... }

// This takes two arguments: the explicit `T`, and the implicit `T: Clone`.
impl<T> "clone_vec" Clone for Vec<T>
where
    clone_t: [T: Clone] // weird syntax don't @ me
{
    fn clone(&self) -> Self {
        vec
            .iter()
            // We use the trait bound explicitly here
            .map(|x: &T| clone_t::clone(x))
            .collect()
    }
}

let my_vec: Vec<u32> = vec![0, 1, 2];
// square brackets is my syntax for passing implicit params:
clone_vec::<u32>[clone_u32]::clone(&my_vec);

To cover all of Rust we’ll need a few more ingredients, but not that many: naming parent traits, naming clauses on associated types, associated type equalities (see next section), and not that much more.

[!NOTE] Another terminology drop: such a process of “filling in implicit information” is typically called “elaboration” in type theory circles. We can thus call this process “trait elaboration”.

The Tricky Part: Associated Types and Equality

The main ingredient we’re missing is associated types. Our “dictionaries” can actually contain types. They definitely aren’t normal structs anymore but that won’t stop us:

trait Iterator {
    type Item;
    fn next(&mut self) -> Option<Self::Item>;
}
// behaves like:
struct Iterator<Self> {
    item: Type, // look ma, types inside values
    next: fn(&mut Self) -> Option<self.item>, // a little bit of self-reference magic 🤫
}

What associated types add to our system is equality bounds, as in T: Iterator<Item = u32>. We can turn these into “passing-style” too, using a neat trick:

// This would be a built-in trait understood by the compiler.
trait Is<T> {}
impl<T> "is_impl" Is<T> for T {}

// Then an equality bound looks like:
fn foo<T>()
where
    iter_t: [T: Iterator],
    item_is_u32: [iter_t::Item: Is<u32>],
{ ... }

Just like a trait bound, the caller of foo would then have to provide a proof that T::Item is indeed u32. In the simple case that proof is given by is_impl, and in more complex cases we can imagine making things like transitivity of equality available³.

This topic is really the tricky part of our endeavour. The trait solver and type checker use type equalities all the time, so trying to track them explicitly would require a whole bunch of new shenanigans. But it can, at least in theory, be done.

Can this even work?

There’s a classic example that illustrates why dictionary-passing style causes trouble in today’s Rust:

trait Trait {
    type Item;
}
impl<T> Trait for T {
    type Item = T;
}

fn callee<T>(t: T) -> <T as Trait>::Item {
    t
}
fn caller<T: Trait<Item = U>, U>(t: T) -> U {
    // The return type of `callee` is `<T as Trait>::Item` which we know to be `U`
    // so all looks good.
    callee(t)
}

Let’s elaborate it to understand the trouble:

trait Trait {
    type Item;
}
impl<T> "the_impl" Trait for T {
    type Item = T;
}

fn callee<T>(t: T) -> the_impl<T>::Item {
    t
}

fn caller<T, U>(t: T) -> U
where
    t_trait: [T: Trait]
    item_is_u: [t_trait::Item: Is<U>]
{
    // ERROR: `the_impl<T>::Item` is `T`, yet we need a `U`.
    callee::<T>(t)
}

This code is accepted today, yet to typecheck in dictionary-passing style we would need to know that T = U. A way to understand the problem is that because of the global impl, caller can only be called with T = U. But caller doesn’t know this: it lives in a world where T and U may be different, yet calls a function that knows them to be the same.

The reason this works in today’s Rust is “coherence”, i.e. the knowledge that there can be only one Trait impl for T. It’s not obvious how to fit that into the dictionary approach.

Our answer to this problem is: we may find a clean solution, or we may find good-enough hacks, or we may even choose to reject such code⁴, because dictionaries could just be worth it.

All This For Soundness

You may now be asking “why would we be doing all this?”. The answer is “soundness”⁵, in two ways:

• From the implementation point of view, we have high hopes that doing things like this will avoid a lot of soundness bugs in the compiler.
• From the research point of view, this could be a way to prove soundness, i.e. to find sufficient conditions that ensure (at least) that the trait solver doesn’t allow safe code to cause UB.

With this blog post, we get a first such condition: trait elaboration must produce appropriate proofs. For example, using a T: Clone bound where T: PartialEq is expected is obviously incorrect.

This by itself isn’t enough however. I’ll take this known unsoundness as an example (playground):

trait Trait<R>: Sized {
    type Proof: Trait<R, Proof = Self>;
}
impl<L, R> Trait<R> for L {
    type Proof
        = R
    where
        L: Trait<R>,
        R: Trait<R, Proof = <L::Proof as Trait<R>>::Proof>;
}
fn transmute_inner<L: Trait<R>, R>(r: L) -> <L::Proof as Trait<R>>::Proof { r }
fn transmute<L, R>(r: L) -> R { transmute_inner::<L, R>(r) } // oops

Here, each individual trait bound and associated type equality can find a justification (I promise). The issue is that the overall dependency of proofs on each other is too recursive, so we end up “proving X by assuming X”.

And so we also need a more global notion of “the overall elaborated program isn’t too ridiculously recursive”. Defining this properly is what we’re working on at the moment.

One question I don’t know the answer to yet is: are these conditions enough? Most likely we’d also need a condition related to coherence⁶ (the fact that there’s at most one impl for each trait and type); is that all we’re missing? I’m looking forward to finding out.

Summary

Dictionaries are, in my opinion, a very cute and useful way of understanding traits. They bring the massive benefit of making it much more obvious to reason about what is or isn’t sound: if you can express it as a dictionary thing, it’s probably sound.

One example is specialization, which has been stalled for a decade due to unsoundnesses. lcnr writes about a “maybe bounds” idea here, which would be a way to get sound specialization, and we know it’s sound because it’s trivially dictionary-passing.

I’m hoping we can integrate this into rustc and advance the grand vision of formally specifying Rust! Stay tuned for more developments; you may also follow our project goal when it gets approved.

Thanks to lcnr and Boxy teaching me so much about these topics. And particular thanks to lcnr for mentoring me through this project.

Over the recent months, lively discussions have been happening around teaching the borrow-checker to understand more things than just “shared borrow”/”mutable borrow”. We’re starting to have a few ideas floating around, so I thought I put them all down in a table so we can see how they interact. I’ll start with the tables and explain the new reference types after.

To clarify the vocabulary I’ll be using:

• A “place” is a memory location, represented by an expression like x, *x, x.field etc¹;
• “Taking a borrow” is the action of evaluating &place, &mut place, &own place etc to a value;
• A “loan” happens when we take a borrow of a place. The loan remembers the place that was borrowed and the kind of reference used to borrow it. When we take that borrow, the resulting reference type gets a fresh lifetime; as long as that lifetime is present in the type of a value somewhere, the loan is considered “live”;
• While a loan is live, further operations on the borrowed place are restricted;
• After the loan expires, operations on the borrowed place may be restricted too.

Note: These tables are not enough to understand everything about these new references. E.g. after you write to a &uninit you can reborrow it as &own, and vice-versa; this is not reflected in the tables. Another example: dropping the contents of a place removes any pinning restriction placed on it.

Table 1: Given a reference, what actions are possible with it

How to read this table: the reference I have gives me the column, the action I want to do with it gives me the row (the reference types mean that the action is a reborrow with that type).

For example, if I have a &own T I can reborrow it into a &mut T but not a &pin mut T. If I have a &mut T I may write a new value to it but not drop its contents.

“Move out” means “move the value out without putting a new one in”. “Drop” means “run the drop code in-place”.

Table 2: If a loan was taken and is live, what can I still do to the place

How to read this table: a borrow of place p was taken and is still live; the kind of borrow gives me the column. The operations I may still do on the place (without going through that borrow) give me the row.

For example, if a &-borrow was taken and is live, I may still read the place.

Unsurprisingly, most of the time we can’t do anything else to the place, because the borrow is exclusive.

Table 3: If a loan was taken and expired, what can I now do to the place

How to read this table: a borrow of place p was taken and has expired; the kind of borrow gives me the column. The operations I may now do on the place give me the row.

For example, if a &own-borrow was taken and expired, I may no longer read the place.

&own and &uninit both have the property that when they expire the place is considered to have been uninitialized. So we the only actions available are those that work on uninitialized places: writing and &uninit borrows.

The other point to note are the pinning loans: after a pinning loan expires, non-pinning mutable loans can’t be taken of that same place until the value is dropped.

What’s with all these new reference types?

All of these are speculative ideas, but at this point they’ve been circulating a bunch so should be pretty robust.

The owning reference &own T

&own T² (which has also been called &move T) is a reference that says “I have full ownership over this value, in particular I am responsible for dropping it”. It feels like Box except that we don’t control the allocation the value resides in. In particular, we can move the T out of a &own T.

Like Box, &own T drops the contained value when it is dropped.

When I owning-borrow &own x a place, I have given up full ownership of the value, and thus must assume that the place is uninitialized when the borrow expires.

It makes for some interesting APIs:

impl<T> Vec<T> {
    // Pop the last value and return a reference to it (instead of moving it out directly).
    fn pop_own(&mut self) -> Option<&own T> { .. }
    // Iterate over the contained values, emptying the Vec as we go.
    fn drain_own(&mut self) -> impl Iterator<Item = &own T> { .. }
}

The uninitialized reference &uninit T

&uninit T³ (which has also been called &out T) is a reference to an allocated but not-yet-initialized location. Much like when we do let x;, the only thing one can do with that reference is write to it. Once written to, it can be reborrowed into everything we want.

// Typical usage is to initialize a value:
impl MyType {
    fn init(&uninit self) -> &own Self {
        *self = new_value();
        &own *self
    }
}
let x: MyType;
let ptr: &own MyType = MyType::init(&uninit x);
// `ptr` can be used much like a `Box` would. It cannot be returned from the
// current function though.

It has a nice synergy with &own T: we can get from &uninit T to &own T by writing a value into the reference, and get from &own T to &uninit T by moving the value out of the reference.

Both have the property that when they expire, the original place is considered uninitialized⁴.

The pinning references &pin T/&pin mut T/&pin own T

Pinning is a notoriously subtle notion, if you’re not familiar with it I recommand the std docs on the topic. If you are familiar with it you may instead enjoy this blog post of mine that shines an original light on the notion.

Pinning references are variants of the existing reference types that also add a “pinning requirement” to the borrowed place. This requirement forbids moving the value out or deallocating the place without running Drop on the value first. This applies even after the pinning borrow has expired.

&pin mut T⁵ is the most common of these, it exists in today’s Rust as Pin<&mut T> and is crucial to our async story. &pin T would be Pin<&T>; it’s less clear how useful it is but I can imagine usecases.

&pin own T finally would be the owning variant. This one has the tricky requirement that it must not be passed to mem::forget as that would break the drop invariant. This isn’t possible in today’s Rust, but there have been proposals over the years as such non-forgettable types are needed for other things.

One funky aspect of &pin own T is that I think it’s ok to reborrow a &own T into a &pin own T: since the only way for the borrow to expire entails dropping the pointed-to value, there’s no way to break the pin guarantee so it doesn’t matter how we got that reference.

You’ll notice I didn’t list &pin uninit T. That’s because pinning is a property of a value, and &pin uninit T doesn’t point to a value. To pin-initialize a value, one can just write the value to a &uninit T then reborrow &uninit T -> &own T -> &pin own T.

In a recent meeting, Niko proposed that we should add language features to enable desugaring more implicit things into valid Rust. This blog post is me taking his idea and running with it.

Edit: I have turned this idea into a book! This book describes ~30 desugaring steps that go from surface Rust to a MIR-like subset of it. I intend this to be a reference document or even a spec kind of thing. Do have a look! I came up with even more language features to make it work.

How does one specify a language like Rust? At the base of understanding a Rust program today, we have the MIR. It’s a representation of a program as a control-flow graph (i.e. with gotos between blocks) and basic operations like “compute a+b and store it in c” or “call this function with these arguments”.

MIR is simple enough that its semantics can be formally described, and in fact the MiniRust project aims to do just that. So, given a MIR program, we can¹ know exactly what it means to run it, and all the rest of the compilation to machine code is but an implementation detail.

The question then is how we get from a Rust program to its MIR. Today a lot of that is hard to explain; the idea of this blog post is that we would like to explain it as a desugaring, into code that is also valid Rust yet precise enough that the mapping from it to MIR is obvious².

What features are we missing to make this desugaring possible? Here are some ideas. (Disclaimer: none of these are my ideas³, but instead things I’ve seen floating around in the Rust ideaspace).

Identifier hygiene

Macros have hygiene, which avoids accidentally shadowing names:

let x = 1;
macro_rules! check {
    () => { assert_eq!(x, 1); }; // Uses `x` from the definition site.
}
let x = 2;
check!(); // the test passes
println!("{x}") // prints 2

To desugar that to valid rust code, we would need to make this information explicit⁴, maybe by renaming identifiers or providing extra disambiguation information:

let x#1 = 1;
let x#2 = 2;
assert_eq!(x#1, 1); // the test passes
println!("{x#2}") // prints 2

Non-dropping assignment

The assignment *x = y; does two things: it drops the previous value inside *x, then writes y to it. To desugar this we need to make the drop explicit, but then we need something new to express “an assignment that doesn’t drop first”.

*x = y;
// desugars to:
drop(*x);
*x ??? y; // would need a special assignment operator that doesn't drop first

One possible solution would be for the borrow-checker to track that *x has been moved out and therefore *x = y; no longer needs to drop the value. This already happens if we had a local variable instead of *x, so why not. The trouble is that determining whether a drop is required is non-trivial, which is in tension with our goal of “the mapping to MIR is obvious”.

Enum projections

In this example, we’re taking a borrow inside an enum:

let opt: &mut Option<u32> = ...;
if let Some(x) = opt {
    // x: &mut u32 here
}

The meaning of this is simple: we first check the enum variant, then borrow the right subplace. We’re only missing syntax for that subplace:

let opt: &mut Option<u32> = ...;
let discriminant = std::mem::discriminant(&*opt);
if discriminant == discriminant_of!(Option<u32>, Some) {
    let x = unsafe { &mut (*opt).Some.0 };
}

I’d propose <place>.<variant_name> as a syntax, and that would be an unsafe operation because you must have checked that the variant is the right one first.

Phased initialization

How does one construct a new value? Built-in values have their built-in constructors (42, true, 67.5, &mut x). For structs, enums and unions we have constructors like Struct { field: 42 }, which can be desugared into assignments to the individual fields. Borrowck could easily figure out that the value is initialized once all the fields are written to⁴ :

let x = Struct { field: 42 };
// would desugar to:
let x: Struct;
x.field = 42;
// Here `x` can be used normally

For enums we can use the enum projections above, combined with a way to set the discriminant (I quite like this syntax combined with these semantics:

let x: Option<u32>;
unsafe {
    x.Some.0 = 42;
    x.enum#discriminant = discriminant_of!(Option<u32>, Some));
    // The discriminant is only allowed to be set once the rest of the fields are
    // initialized, so borrowck can use that to know that `x` is initialized now.
}

I wonder if we could allow this in safe code somehow.

Naming the return place

Not strictly necessary but fun: the expression return x mixes two things: a control-flow construct and setting the return value. We could separate the two by making the return place nameable:

fn foo() -> u32 {
    if check() {
        return 42;
    }
    0
}
// would become:
fn foo() -> u32 {
    if check() {
        return#place = 42;
        return;
    }
    return#place = 0;
}

No idea of a good syntax here. The way I imagine this working is that the return place starts out uninitialized, like let x;, and a plain return is allowed if borrowck determines that the return place has been initialized.

Explicit pointer metadata and explicit vtables

Pointer metadata for unsized types is handled invisibly today:

trait Trait {
    fn method(&self);
}

struct Struct;
impl Trait for Struct {
    fn method(&self) {
        println!("hi!")
    }
}

let x: Struct = ...;
let x: &dyn Trait = &x; // now the pointer carries a pointer to the right vtable.

We could make that explicit in two parts: making metadata explicit, and making vtables explicit:

struct TraitVTable {
    method: unsafe fn(&dyn Trait),
}

static ImplTraitForStructVTable: TraitVTable = TraitVTable {
    // Safety: only call if `this` actually points to a `Struct`.
    method: |this: &dyn Trait| {
        let this: &Struct = unsafe { transmute(this) };
        this.method()
    }
};

// Assuming a method like `std::ptr::from_raw_parts` but for references:
let x: &dyn Trait = unsafe { std::ref::from_raw_parts(&x, &ImplTraitForStructVTable) };

How does this extend to e.g. Rc<Struct> -> Rc<dyn Trait>? Unclear, one option would be to add a method to the (unstable) CoerceUnsize trait that would contain auto-generated code that unpacks the pointer, adds the vtable metadata, and repacks it.

continue operator for matches

Operationally, pattern matching expressions just stand for a series of comparisons of discriminants or integers. So in principle we could desugar a match to a big series of ifs. However that would not give us the same MIR as we get today: the lowering of patterns to MIR is a bit sophisticated⁵, to emit more performant code. Let’s try to preserve that.

Let’s start with match guards:

match opt {
    Some(x) if foo(x) => ..,
    None => ..,
    Some(_) => ..,
}

In that match, if foo(x) returns false we keep trying the arms below. We could give a syntax for that:

'a: match opt {
    Some(x) => if foo(x) {
        ..
    } else {
        continue 'a; // tries the arms after this one
    },
    None => ..,
    Some(_) => ..,
}

This is interestingly not more expressive than today’s matches, since all of that code could have just been in the match guard⁶. Match exhaustiveness would simply ignore arms that use continue, just like it ignores arms with a guard today.

For more complex matches, we can reuse continue to decompose them into layers:

match val {
    (Some(x), None) => .., // branch 1
    (_, Some(y)) => .., // branch 2
    (None, None) => .., // branch 3
}
// could desugar to:
// This shape might look weird but that's how this `match` expression is compiled to
// MIR today. It has that shape to avoid duplicating branch 2.
'a: match val.0.enum#discriminant {
    discriminant_of!(Option, Some) => match val.1.enum#discriminant {
        discriminant_of!(Option, None) => {
            let x = val.0.Some.1;
            // branch 1
        }
        discriminant_of!(Option, Some) => continue 'a,
    },
    _ => match val.1.enum#discriminant {
        discriminant_of!(Option, Some) => {
            let y = val.1.Some.1;
            // branch 2
        }
        discriminant_of!(Option, None) => match val.0.enum#discriminant {
            discriminant_of!(Option, None) => .., // branch 3
            _ => unsafe { unreachable_unchecked() },
        },
    },
}

Can we desugar all matches that way? In principle yes because we can do arbitrary⁷ control-flow using break, but that can get ugly:

match val {
    (Some(x), None) | (None, Some(x)) => .., // branch 1
    (Some(_), Some(_)) | (None, None) => .., // branch 2
}
// naively desugars to:
match val.0.enum#discriminant {
    discriminant_of!(Option, Some) => match val.1.enum#discriminant {
        discriminant_of!(Option, None) => {
            let x = val.0.Some.0;
            // branch 1
        }
        discriminant_of!(Option, Some) => .., // branch 2
    },
    discriminant_of!(Option, None) => match val.1.enum#discriminant {
        discriminant_of!(Option, Some) => {
            let x = val.1.Some.0;
            // also branch 1
        }
        discriminant_of!(Option, None) => .., // also branch 2
    },
}
// to avoid duplication, could do something like:
'match_end: {
    'branch1: {
        'branch2: {
            match val.0.enum#discriminant {
                discriminant_of!(Option, Some) => match val.1.enum#discriminant {
                    discriminant_of!(Option, None) => {
                        let x = val.0.Some.0;
                        break 'branch1;
                    }
                    discriminant_of!(Option, Some) => break 'branch2,
                },
                discriminant_of!(Option, None) => match val.1.enum#discriminant {
                    discriminant_of!(Option, Some) => {
                        let x = val.1.Some.0;
                        break 'branch1;
                    }
                    discriminant_of!(Option, None) => break 'branch2,
                },
            }
        }
        // code for branch 2
        break 'match_end;
    } 
    // code for branch 1
    break 'match_end;
}

I don’t have great ideas here.

Unchecked builtin indexing

As we know, Rust inserts bound checks when accessing arrays/slices. On first approximation, we might desugar like this:

let slice: &mut [T] = ...;
let x = &mut slice[0];
// becomes
assert!(slice.len() > 0);
let x = unsafe { slice.get_unchecked_mut(0) };

However, a little known fact is that indexing is somewhat understood by the borrow-checker:

let slice: &mut [(A, B)] = ...;
let x = &mut slice[i].0;
let y = &mut slice[j].1;
do_something(x);
do_something(y);

This is allowed because regardless of i and j the borrows are guaranteed to be disjoint. But if we go through a function call like get_unchecked_mut, we need full exclusivity of the whole of slice! So if we want to preserve this behavior, we need a built-in operator for unchecked indexing (that the borrow-checker understands).

Constant, borrowck-tracked, indexing

To continue on the interactions between borrow-checking and indexing, you might believe that the borrow-checker never takes into account indices: it refuses to borrow &mut slice[i] and &mut slice[j] simultaneously even when i and j are 0 and 1 which are obviously distinct.

However there is a funky little corner of Rust semantics where it does track indices, namely slice patterns:

let slice: &mut [T] = ...;
if let [a, .., b] = slice
    && let [_, c, e @ .., d, _] = slice {
    // a,b,c,d: &mut T
    // e: &mut [T]
}
// compiles to, morally:
if slice.len() >= 2 && slice.len() >= 4 {
    let a = &mut slice[const 0];
    let b = &mut slice[const -1];
    let c = &mut slice[const 1];
    let d = &mut slice[const -2];
    let e = &mut slice[const 2..const -2];
}

It can tell that these are distinct because they use a special constant-indexing primitive in MIR. The reason for the notation I chose is because it supports also indices that are constant-starting-from-the-end, which are needed here. I used python-like negative indices for illustration.

If we wanted to desugar that, we’d need a syntax for these.

In-place drop

At the end of a scope, all the live locals and temporaries are automatically dropped. To desugar that, one could naively want to insert calls to drop(local), but that’s actually incorrect: drop actually happens in-place, which is soundness-critical for pinned types.

{
    let x = String::new();
}
// would want to desugar to:
{
    let x = String::new();
    unsafe { std::ptr::drop_in_place(&raw mut x); }
    // But then borrowck will try to drop the place again!
}

The issue is that we also need to tell borrowck “this place is dropped now, don’t try to drop it again”. Today I think mem::forget(place) works, but if we get ever get a feature for pinned places that would not be allowed. In that case we’ll need “in-place mem::forget” or something along these lines.

Etc

What I would love to see eventually is a cargo desugar command⁸, or a code action in my editor, that shows the fully desugared version of a piece of code. Then we’d make the Reference describe these desugarings, we’d make Miri run on this subset of Rust instead of on MIR, and understanding the behavior of a piece of Rust code would become much easier!

That’s a bunch of ideas, there a likely a lot of other implicit transformations that can’t be expressed in plain Rust. Please share your ideas!

This blog post is part of the discussions around the Field Projections project goal. Thanks to Benno Lossin and everyone involved for the very fruitful discussions!

In a my first post on this blog I outlined a solution for making custom smart pointers as well integrated into the language as references are today. I had left the exact rules for autoref and autoderef unspecified; this blog post is my attempt to write them down precisely.

The basic tenets I have for the whole feature are:

• Expressions have a type that doesn’t depend on their context.
• To understand what operation (e.g. method call) is being done on an expression, one only needs to know the type of that expression.

PlaceWrap and non-indirected place operations

One of the recent ideas we’ve added to the proposal is this trait¹, which we’ll need to explain the desugarings:

/// If `X: PlaceWrap` and `X::Target` has a field `field`, then `X` itself acquires a virtual field
/// named `field` as well. That field has type `<X as
/// PlaceWrap<proj_ty!(X::Target.field)>>::Wrapped`, and `WrappedProj` is the
/// projection used when we refer to that field.
pub unsafe trait PlaceWrap<P: Projection<Source = Self::Target>>: HasPlace {
    /// The type of the virtual field. This is necessarily a transparent wrapper around `P::Target`.
    type Wrapped: HasPlace<Target = P::Target>;
    type WrappedProj: Projection<Source = Self, Target = Self::Wrapped>;
    fn wrap_proj(p: &P) -> Self::WrappedProj;
}

This is implemented for “non-indirected containers” such as MaybeUninit, RefCell, Cell, etc. What it does is that if Struct has a field field: Field, then MaybeUninit<Struct> has a virtual field field: MaybeUninit<Field>.

In the next section I explain how that interacts with the existing place operations, and at the end we’ll see examples of how they work together for very nice expressivity.

To explain the computations I propose a strawman syntax @@Type p which is allowed iff Type is a transparent wrapper around the type of p. This expression is a place expression too, it behaves like basically a transmute of the target without doing anything else. In particular this is how PlaceWrap operates: if x: MaybeUninit<Struct>, x.field desugars to @@MaybeUninit (*x).field.

Computing the type of a place

Every place expression starts with a local or a temporary, with a known type. We then apply one or more of the pure place operations, recursively:

• deref *p;
• field access p.field;
• indexing p[i].

Deref is simple: *p requires that p: T: HasPlace, and then *p: T::Target.

Field access is the tricky one; I propose the following. Let p be a place expression of type T.

• If T has a field field: F, p.field: F and we’re done;
• If T: !HasPlace, error.
• If T: HasPlace, we first descend through T::Target::Target::etc until we find a type that has a field field: F. We get the intermediate expression tmp_place := (****p).field: F with the appropriate number of derefs.
• We then “go back up” as long as the intermediate T::Target::etc implements PlaceWrap<the_right_thing>. Every time we go back up in such a way, we wrap our target place in tmp_place := @@Wrapped tmp_place.
• The first time we can’t PlaceWrap, we’re done.
• If T: !HasPlace, error.

Finally, indexing is easy because we’re only talking about built-in indexing here. It’s exactly like a field access, where [T] and [T; N] have one field per index. The tricky part is just that the index is not known at compile-time. That’s the reason why Projections don’t make the offset available as a const actually.

Examples, assuming Struct has a field field: Field:

• p: MaybeUninit<Struct>: p.field desugars to @@MaybeUninit (*p).field with type MaybeUninit<Field>;
• p: MaybeUninit<MaybeUninit<Struct>>: p.field desugars to @@MaybeUninit @@MaybeUninit (**p).field with type MaybeUninit<MaybeUninit<Field>>;
• p: &&&MaybeUninit<Struct>: p.field desugars to @@MaybeUninit (****p).field with type MaybeUninit<Foo>;
• p: MaybeUninit<&Struct>: p.field desugars to (**p).field with type Foo²;
• p: MaybeUninit<[u8]>: p[42] desugars to @@MaybeUninit (*p)[42] with type MaybeUninit<u8>.

Note that because we resolve place expressions one operation at a time, we ensure that e.g. p.a.b is always the same as (p.a).b.

Computing the type of borrows

Let p be a place expression of type T. The type of @Ptr p is easy: it’s always Ptr<Something>, with the guarantee that Ptr<Something>: HasPlace<Target=T>. This means p cannot change type when this happens. There is no extra autoderef or anything at this stage.

Method autoref

In this section, I will assume that T: Receiver => T: HasPlace<Target=<T as Receiver>::Target>>³ and that T: Deref => T: HasPlace<Target=<T as Deref>::Target>>.

Let p be a place expression of type T, and assume we want to typecheck p.method(). We first compute the set {T, T::Target, T::Target::Target, ..} as long as the types implement HasPlace.

For each such type U, we look through all the impl U and impl Trait for U for a method with the right name. This gives us a list of “method candidates”. If there are none, error; if there are several, pick one in some way. Which one to pick is important for ergonomics but irrelevant for us now.

If the selected method takes fn method(self, ..) directly, we desugar to <..>::method(***p) (with enough derefs to get to the right type) and we’re done.

Otherwise the method takes fn method(self: X, ..) where X: HasPlace (by the assumption on Receiver above). If X::Target is one of the candidate types above, let q := ***p be p suitably derefed to get to that candidate type; we then desugar to <..>::method(@X q). If X::Target is not one of the candidate types, we go back and pick another method.

This draft is possibly quite naive, I’ve heard that method resolution is quite tricky. Whatever I might be missing, the core ideas I’m trying to convey are this:

1. We only ever consider the type of the place. The pointer the place came from does not come into play until after we’ve desugared, to check if the borrow was allowed after all;
2. We search only impl blocks for T, T::Target, T::Target::Target, etc.
3. This works wonderfully with arbitrary_self_types: when we find an arbitrary self type we can just attempt to borrow with that pointer. This means e.g. that for x: CppRef<Struct> and fn method(self: CppRef<Self>) on Field, x.field.method() Just Works.

Desugaring the place operations

Recall that the operations we can do on a place are: borrow, read, write, in-place-drop⁴. Each of these comes with a corresponding PlaceOp trait. Once we know which operation to do on the place, we can desugar the operation to a call to the appropriate trait method, which will also check if that operation is allowed by the pointer in question.

Let’s desugar a PlaceOp operation on a place p. A place expression is made of three things: locals, derefs and projections, where “projections” means field accesses, indexing, and either of these mediated by PlaceWrap.

So our place p can always be written as p = q.proj where .proj represents all the non-indirecting projections (including PlaceWrap ones), and q is a place expression that’s either a local or a deref. Let U be the type of q. Then an operation on p desugars to PlaceOp::operate(get!(q), proj_val!(U.proj)), where get! is defined as:

• if q is a local, get!(q) is &raw const @@LocalPlace q;
• otherwise q is a deref which we can write *(r.proj2), and we can get the right pointer using PlaceDeref::deref⁵. This applies recursively if r itself contains a deref, etc.

Where PlaceWrap comes into play is in this proj_val! macro: that macro computes the value of the appropriate P: Projection type. If PlaceWrap is involved, then it will be used in computing that projection.

Canonical reborrows

As a special case of the borrows above, the official proposal includes a notion of “canonical reborrows”, whereby each pointer can declare the default type with which to be reborrowed, and the (possibly temporary) syntax @$place uses it.

The way it works is simple: @$place desugars just like PlaceBorrow above, except when we get to PlaceOp::operate we use <PlaceBorrow<'_, _, <Ptr as CanonicalReborrow<proj_ty!(U.proj)>>::Output>>::borrow where Ptr is the type of *get!(q). This is equivalent to @Output $place with that same Output type.

Putting it all together

Let’s go through a bunch of examples. In what follows e is the expression of interest that we want to desugar and typecheck. We also assume the obvious place operations on standard library types, as well as:

struct Struct {
    field: Field,
}
struct Field {
    value: u32,
}

// Implements `PlaceWrap`.
struct W<T> {
  value: PhantomData<()>,
  wrapped: T,
}

1. p: &mut MaybeUninit<Struct>, e := &mut p.field

   We get e = &mut @@MaybeUninit (**p).field : &mut MaybeUninit<Field>, and the two traits involved are PlaceWrap for MaybeUninit and PlaceBorrow<P, &mut P::Target> for &mut P::Source. Note how &mut is entirely unaware of anything special happening, and how that would work with many nested wrappers.

2. x: Struct, impl Field { fn method(self: CppRef<Self>) }, e := x.field.method()

   We get e = Field::method(@CppRef x.field). Per the section on borrows, @CppRef x.field becomes @CppRef (*@@LocalPlace x).field, which is allowed iff LocalPlace<Struct>: PlaceBorrow<P, CppRef<Field>>. The smart pointer can opt-in to that, and of course they can choose the nature of the resulting borrow (owning, exclusive, shared, etc).

3. x: &mut CppRef<Struct>, impl Struct { fn method(self: &CppRef<Self>) }, e := x.method()

   We get e = Struct::method(&*x).

4. x: &mut CppRef<Struct>, impl Field { fn method(self: &CppRef<Self>) }, e := x.field.method()

   I made this an error, but in theory we could desugar this to Field::method(&(@CppRef (**x).field)), i.e. create a temporary CppRef and borrow that. We’ll pick whatever’s consistent with the rest of Rust I guess.

5. x: W<Struct>, e := w.field.value

   We get e = (@@W (*x).field).value : PhantomData<()> because the real field on W takes precedence over the virtual field. If we wanted to access the value field of Field, we’d have to write @@W (*w).field.value.

6. x: &Box<Arc<Struct>>, impl Field { fn method(self: ArcRef<Self>) }, e := x.field.method()

   We get e = Field::method(@ArcRef (***x).field). The final desugaring looks like:

    let tmp: &raw const LocalPlace<&Box<Arc<Struct>>> = &raw const @@LocalPlace x;
    let tmp: &raw const &Box<Arc<Struct>> = <LocalPlace<_> as PlaceDeref<_>>::deref(tmp, trivial_proj_val!(&Box<Arc<Struct>>));
    let tmp: &raw const Box<Arc<Struct>> = <&_ as PlaceDeref<_>>::deref(tmp, trivial_proj_val!(Box<Arc<Struct>>));
    let tmp: &raw const Arc<Struct> = <Box<_> as PlaceDeref<_>>::deref(tmp, trivial_proj_val!(Arc<Struct));
    let arc_ref: ArcRef<Field> = <PlaceBorrow<'_, _, ArcRef<_>>>::borrow(tmp, proj_val!(Struct.field));
    Field::method(arc_ref)
   

   Note how only the last deref (the one of Arc) is involved in the reborrow. The rest are just PlaceDerefed through.

7. x: Arc<Box<Struct>>, e := @ArcRef x.field

   That’s an error. We get e = @ArcRef (**x).field, which uses Arc as PlaceDeref then Box as PlaceBorrow<'_, _, ArcRef<_>> which doesn’t exist. This is unfortunate because in principle we can make this ArcRef<Field>. But this would need something like Arc<Box<Struct>> as PlaceBorrow<'_, P, ArcRef<Field>> where P includes a deref. Projections are just an offset in our model currently, so that’s not allowed¹.

8. x: &Arc<[Struct]>, e := @x[42].field

   This desugars to @ArcRef x[42].field. The final desugaring looks like:

    let tmp: &raw const LocalPlace<&Arc<[Struct]>> = &raw const @@LocalPlace x;
    let tmp: &raw const &Arc<[Struct]> = <LocalPlace<_> as PlaceDeref<_>>::deref(tmp, trivial_proj_val!(&Arc<[Struct]>));
    let tmp: &raw const Arc<[Struct]> = <&_ as PlaceDeref<_>>::deref(tmp, trivial_proj_val!(Arc<[Struct]));
    let arc_ref: ArcRef<Field> = <PlaceBorrow<'_, _, ArcRef<_>>>::borrow(tmp, proj_val!([Struct][42].field));
    arc_ref
   

   Note again how the last derefed pointer is the one used to determine the reborrow.

Below are the footnotes, this theme does not distinguish them very clearly:

The obvious way to make the feature work is to say that in <expr>.macro!(), the macro gets the tokens for <expr> and does what it wants with them.

This however allows macros to break the so-called “no-backtracking rule” (coined by Tyler Mandry IIRC): in x.is_some().while! { ... }, reading the while makes us realize that the is_some() call wasn’t just a boolean value, it was an expression to be evaluated every loop. So we sort of have to go back and re-read the beginning of the line. For purposes of reducing surprise and code legibility, we’d like to avoid that.

Hence the question that the feature stalled on: can we design postfix macros that always respect the no-backtracking rule? We would need to somehow evaluate <expr> once and pass the result to the macro instead of passing <expr> itself. Apart from that I’ll assume that we want maximal expressiveness.

This post is centrally about places and the implicit operations that surround them; check out my recent blog post on the topic for an overview of that vocabulary.

Partial Place Evaluation

To get the obvious out of the way: we can’t just desugar <expr>.method() to let x = <expr>; x.method(); that may give entirely the wrong behavior, e.g.:

struct Foo { count: Option<u32> }
impl Foo {
    fn take_count(&mut self) -> Option<u32> {
        // That's fine
        self.count.take()
        // That creates a copy
        // let tmp = self.count;
        // tmp.take() // modifies the copy instead of the original
    }
}

In technical terms, that’s because the LHS of a method call is a place expression. Storing <expr> in a temporary adds an incorrect place-to-value coercion. The same applies to postfix macros.

I think that the behavior we ideally want is to pre-evaluate all temporaries (that arise from value-to-place coercion), and pass whatever remains of the expression as-is to the macro. I’ll call that “partial place evaluation”.

Some examples:

let x: Foo = ...;
x.field.macro!()
// becomes (there are no temporaries)
macro!(x.field)

impl .. { fn method(&self) -> Foo { .. } }
x.method().field.macro!()
// becomes
let mut tmp = x.method();
macro!(tmp.field)

Looks easy enough, except for autoderef¹.

let x: Box<Foo> = ...;
x.field.macro!()

Depending on the contents of macro!(), this may need to expand to a call to deref or deref_mut:

let tmp = Box::deref(&x);
macro!((*tmp).field)
// or
let tmp = Box::deref_mut(&mut x);
macro!((*tmp).field)

At this point it’s hopefully clear that no simple syntactic transformation will give us what we want.

Place aliases, aka let place

What we’re trying to express is “compute a place once and use it many times”. let place is an idea I’ve seen floating around² which expresses exactly that: let place p = <expr>; causes <expr> to be evaluated as a place, and then p to become an alias for the place in question. In particular, this does not cause a place-to-value coercion.³

let place p = x.field; // no place-to-value, so this does not try to move out of the place
something(&p);
something_else(p); // now this moves out
// would be identical to:
something(&x.field);
something_else(x.field); // now this moves out

let place p = x.method().field;
something(&p);
// would be identical to:
let tmp = x.method();
something(&tmp.field);

This is exactly what we need for postfix macros: <expr>.macro!() would become (using a match to make the temporary lifetimes work as they should 🤞):

match <expr> {
    place p => macro!(p),
}

This would have the effect I propose above: any side-effects are evaluated early, and then we can do what we want with the resulting place.

One of my litmus tests of expressivity for postfix macros is this write! macro, which ends up working pretty straighforwardly:

macro_rules! write {
    ($self:self, $val:expr) => ({
        $self = $val; // assign to the place
        &mut $self // borrow it mutably
    })
}
let mut x; // borrowck understands that `write!` initializes the place!
let _ = x.write!(Some(42)).take();
// desugars to:
let _ = match x {
    place p => write!(p, Some(42)).take(),
};
// desugars to:
let _ = write!(x, Some(42)).take();
// desugars to:
let _ = {
    x = Some(42);
    (&mut x).take()
};

let place and custom autoderef

The hard question is still autoderef¹ :

let mut x: Box<Foo> = ...;
let place p = x.field; // should this use `deref` or `deref_mut`?
something(&p);
something_else(&mut p); // causes `deref_mut` to be called above

For that to work, we infer for each place alias whether it is used by-ref, by-ref-mut or by-move (like closure captures I think), and propagate this information to its declaration so that we can know which Deref variant to call ⁴.

let place isn’t too powerful

Turns out let place is a rather simple feature when we play with it:

// Place aliases can't be reassigned:
let place p = x.field;
// Warning, this assigns to `x.field` here! that's what we want place aliases to do
// but it's admittedly surprising.
p = x.other_field;

// You can't end the scope of a place alias by hand:
let place p = x.field;
drop(p); // oops you moved out of `x.field`
// `p` is still usable here, e.g. you can assign to it

// Place aliases can't be conditional.
let place p = if foo() { // value-to-place happens at the assignment
    x.field // place-to-value happens here
} else {
    x.other_field
};
// This won't mutate either of the fields, `p` is fresh from a value-to-place coercion. I propose
// that this should just be an error to avoid sadness.
do_something(&mut p);

In particular it’s easy to statically know what each place alias is an alias for.

The caveat is that all of those are surprising if you think of p as a variable. This is definitely not a beginners feature.

let place doesn’t need to exist in MIR

The big question that let place raises is what this even means in the operational semantics of Rust. Do we need a new notion of “place alias” in MiniRust?

I think not. The reason is that the “store intermediate values in temporaries” happens automatically when we lower to MIR. All place coercions and such are explicit, and MIR place expressions do not cause side-effects. So whenever we lower a let place p to MIR, we can record what mir::Place p stands for and substitute it wherever it’s used.

To ensure that the original place doesn’t get used while the alias is live, we insert a fake borrow where the let place is taken and fake reads when it’s referenced. That’s already a trick we use in MIR lowering for exactly this purpose⁵.

So the only difficulty seems to be the mutability inference mentioned in previous section. The rest of typechecking let place is straighforward: let place p = <expr>; makes a place with the same type as <expr>, and then it behaves pretty much like a local variable.

All in all this is looking like a much simpler feature that I expected when I started playing with it.

let place is fun

I kinda of want it just because it’s cute. It makes explicit something implicit in a rather elegant way. Here are some fun things I discovered about it.

To start with, it kind of subsumes binding modes in patterns: if let Some(ref x) = ... is the same thing as if let Some(place p) = ... && let x = &p. One could even use place x instead of x in patterns everywhere and let autoref set the right binding mode! That’s a funky alternative to match ergonomics.

We can also use it to explain this one weird corner case of borrow-checking. This code is rejected by the borrow-checker, can you tell why?

let mut x: &[_] = &[[0, 1]];
let y: &[_] = &[];
let _ = x[0][{x = y; 1}];
//      ^^^^ value is immutable in indexing expression

What’s happening is that we do the first bound-check on x before we evaluate the second index expression. So we can’t have that expression invalidate the bound-check on pain of UB. We can use let place to explain the situation via a desugaring:

x[0][{x = y; 1}]
// desugars to:
let place p = x[0]; // bounds check happens here
p[{x = y; 1}]
// desugars to:
let place p = x[0];
let index = {x = y; 1}; // x modified here
p[index] // but place alias used again here

Can this be used to explain closure captures? I don’t think so because closures really do carry borrows of places, not just places. It does feel like a related kind of magic though.

Conclusion

I started out writing this blog post not knowing where it would lead, and I’m stoked of how clean this proposal ended up looking. I kinda want let place even independently from postfix macros. The one weird thing about let place is this “mutability inference” for autoderef, hopefully that’s an acceptable complication.

I’m most looking forward to everyone’s feedback on this; let place is rather fresh and I wanna know if I missed anything important (or anything fun!).

In this post I’ll give a brief overview of what places are, the implicit operations that surround them, and the vocabulary we have for them. I’ll talk more about postfix macros in a later post.

A lot of this blog post is a retelling of this blog post from Ralf in my own words; do check out his post if you want a more rigorous presentation or are interested in how this is relevant for unsafe code.

Places and place expressions

Rust expressions typically evaluate to a value: to evaluate x + 1 we first evaluate x to its value, say 42, then compute 42 + 1 which results in the value 43. But we don’t evaluate &mut x to &mut 42, that would make no sense. We want the result of &mut x to be about x as a memory location, not x as a value.

This is what places are: a place is a memory location, and some Rust expressions refer to places. We saw that a local variable x denotes a place, whereas 42 only denotes a value.

Rust expressions are of two kinds: they denote either a place or a value¹. We call them “place expressions” or “value expressions” depending on which. The following are all the place expressions in Rust:

• <ident>, where the ident is the name of a local variable or static;
• Deref *<expr>;
• Field access <expr>.field;
• Indexing <expr>[<expr>].

All the other expressions (e.g. method call <expr>.method(), arithmetic operation <expr> + <expr>, constant 42, etc) are value expressions.

On the other side of this, each operation takes either a place or a value. Each operand of an operation is either a “place context” or “value context” depending on which. The criterion is, roughly: if the operation cares only about the value of that expression then it’s a value context, if it also cares about where the value is stored then it’s a place context.

The following operations are all place contexts:

• Borrows &mut <expr>, &<expr>, &raw const <expr>, &raw mut <expr>;
• The LHS² of a write assignment <expr> = ...;;
• The RHS of an assignment let ... = <expr>;, ... = <expr>;³;
• The scrutinee of a match match <expr> { ... };
• The LHS of a field access <expr>.field;
• The LHS of an indexing operation <expr>[...];
• The LHS of a method call <expr>.method(..).

The following operations are all value contexts:

• <expr> + <expr>, <expr> - <expr> etc;
• *<expr>;
• (<expr>, <expr>) as well as struct and enum constructors;
• { <expr> }.

These lists are not exhaustive.

Place-to-value coercion

So what happens when you put a place expression in a value context? Rust inserts an implicit read of the value inside the place. This is called “place-to-value coercion” and following Ralf I’ll write it “load”:

let z = x + y + 1; // `x` and `y` are place expressions in a value context
// actually means:
let z = load x + load y + 1;

let x = Some(*ptr); // `*ptr` is a place expression in a value context
// actually means:
let x = Some(load *ptr);

If the place expression has a non-Copy type, then place-to-value coercion will move the value out (or raise an error). E.g.:

let x = Box::new(42);
let y = Some(x);
// actually means:
let y = Some(load x); // this moves out of `x`

In fact whenever you get the “cannot move out of a shared reference” error, you know there was a place-to-value coercion somewhere.

Value-to-place coercion

How about the other way around, can a value expression be put in a place context? Absolutely, and we then get “value-to-place coercion”, also called “storing the value in a temporary place”. A simple example is:

let x = f(&String::from('🦀'));
// actually means something like:
let x = {
    let tmp = String::from('🦀');
    f(&tmp)
};

This actually happens quite often, with method autoref (which I’ll go into in a moment), e.g.:

if x.method().is_some() {
    ...
}
// method resolution + autoref desugars this to:
if Option::is_some(&x.method()) { // `x.method()` is a value expression in a place context
    ...
}

This direction of coercion is much trickier than the other, because it raises the thorny question of how long that implicit temporary place should live. That topic is called “temporary lifetime extension rules” and you should check out Mara’s blog post on the topic⁴ to get a sense of the space.

Autoderef

Autoderef is, to start with, what allows you to write things like x.field when x: &T. The compiler will desugar this to (*x).field. This works with any number of references: x: &&mut &T gives (***x).field.

This also happens on method calls, inside function arguments (you can pass a &&mut &T to a function expecting a &T), and a bunch of other cases I couldn’t list exactly⁵.

Autoderef is in fact more powerful than this: it applies not only to built-in references but to any smart pointer that implements Deref. So if x: Box<T>⁶, x.field becomes (*Box::deref(&x)).field (note the automatic borrow of x). Well, unless you’re about to use x.field mutably, in which case it becomes (*Box::deref_mut(&mut x)).field.

And this is where autoderef is very magical: if x is a smart pointer then *x by itself is a place, whose type is known, but that we won’t know how to compute until we know what we’re doing to the place. A mutable borrow or assignment causes deref_mut to be called, otherwise deref is called⁷.

Autoref and method resolution

The final piece of the puzzle is what happens on method calls. This might be the most magical desugaring we have: method resolution. Two things happen for <expr>.method(): we have to figure out what method to call, and in the process may have to change <expr>.

Take a simple example:

let mut x = Some(42);
let y = x.take();

Here x has type Option<i32>, so we look at all the methods on Option and find fn take(&mut self). To make the type match, we insert a borrow of x. The desugared call is Option::take(&mut x). This process of “adding extra references when needed” is called “autoref”.

Method resolution can also involve autoderef: if x: Rc<i32>, x.is_some() will first look for an is_some method on Rc, then fallback to autoderef and try again. We end up with Option::is_some(Rc::deref(&x)). The full algorithm involves a mix of autoderef and autoref.

Closure capture

Closures add another layer of magic to places: if you mention inside a closure a place that comes from outside the closure, the place will automatically get carried around along with the closure. We say the place is “captured”, and this means either that we place-to-value coerce and store the resulting value inside the closure, or that we store a & or &mut borrow of the place inside the closure. Much like for autoderef, the way we capture x depends on how the place is used.

For example, this:

let x: Foo = ...;
let f = || {
    x.field.is_some()
};

causes x.field to be captured, in this case as a shared borrow. The resulting code is equivalent to the following, where we make the closure object explicit:

struct Closure<'a> {
    p: &'a Field,
}
impl Fn<()> for Closure<'_> { // I'm cheating a bit on the shape of this trait
    type Output = bool;
    fn call(&self) -> bool {
        Option::is_some((*self).p)
    }
}
// We store a borrow of `x.field` inside the closure.
let f = Closure { p: &x.field };

The rules for what we capture exactly are subtle, see the Reference for details.

Conclusion

In this whirlwind tour, we saw that places are at the center of a number of implicit operations:

• place-to-value coercion;
• value-to-place coercion (with temporary lifetime extension);
• autoderef;
• autoref along with method resolution;
• closure capture.

Places get implicitly borrowed, created, moved out of and discarded all of the time implicitly. This all comes together to “just work” most of the time, and I’d say play a big role in Rust’s renowned expressivity power, but is far from obvious when you start digging.

I hope this post gave you a clearer picture, and I for one know I’ll be referencing this blog post in the future. I expect to keep it up-to-date/add more detail to it, more like a reference document than a one-off blog post.

Disclaimer: I am by no means a Pin expert so this may be incorrect in subtle ways. Please let me know if so!

Pin is notoriously subtle: once pinned, a place has restricted access forever, even after the Pin<&mut T> reference goes out of scope. The reason is simple: we want to allow pointers to the place to be stored in locations we know nothing about, so the data there better stay consistent.

In other words, a place has restricted access because some pointers to it may exist. Well this is typically the kind of thing the borrow-checker tracks!

I propose the following idea, for the sole purpose of making Pin easier to understand: weak references. &weak T would be a new reference kind, with the following properties:

• it doesn’t allow any safe accesses;
• while it exists, no contained !Unpin type is allowed to be moved out of the place; other mutations are fine;
• while it exists the pointed-to place may be deallocated, on the condition that the value is dropped first.

So it’s basically a raw pointer, except that while I hold a &weak T to a place, I know the place will maintain some consistency. In particular it won’t be deallocated without calling Drop first, so the Drop impl has the opportunity to let me know that my &weak reference can’t be used anymore (this is illustrated in the example below).

The way Pin fits into this is that we can define it as:

struct Pin<P: Deref>(P, &'static weak P::Target);

Here the weirdness of Pin is made clear: it makes the pointed-to place weakly-borrowed forever. The API surface and safety requirements of Pin are all there to make sure only use the P pointer in ways that don’t break the invariants imposed by the &'static weak reference.

Note that it’s ok to take a 'static reference of a local variable here because &weak allows the pointed-to place to be deallocated. Of course it can then only be used if you have some mechanism to know whether the target is still allocated.¹

Here’s how it would work in an example: intrusive linked lists (based on Ralf’s post on the topic).

struct Collection<T> {
    // The `Drop` impl of `Entry` guarantees that the entries listed can be accessed.
    objects: RefCell<Vec<&'static weak Entry<T>>>,
}
impl<T> !Unpin for Collection<T> {}

struct Entry<T> {
    x: T,
    // Set to `Some` if we are part of some collection.
    // The `Drop` impl of `Collection` guarantees that the collection can be accessed.
    collection: Cell<Option<&'static weak Collection<T>>>,
}
impl<T> !Unpin for Entry<T> {}

impl<T> Collection<T> {
    fn new() -> Self {
        Collection { objects: RefCell::new(Vec::new()) }
    }

    // Add the entry to the collection.
    fn insert(mut self: Pin<&mut Self>, entry: Pin<&mut Entry<T>>) {
        if entry.collection.get().is_some() {
            panic!("Can't insert the same object into multiple collections");
        }
        // Pointer from collection to entry. This `&mut` is unsafe: not all mutations
        // through it are allowed.
        let mut_this : &mut Self = unsafe { Pin::get_mut(&mut self) };
        mut_this.objects.borrow_mut().push(Pin::get_weak(&entry));
        // Pointer from entry to collection.
        let weak_this: &weak Self = Pin::get_weak(&self);
        entry.collection.set(Some(weak_this));
    }

    // Show all entries of the collection.
    fn print_all(self: Pin<&mut Self>)
    where T: Debug
    {
        print!("[");
        for entry in self.objects.borrow().iter() {
            // Safety: the `&weak` ref guarantees:
            // 1. that `entry.collection` cannot be changed and keeps pointing to this
            //   collection;
            // 2. that the entry won't be deallocated without running `Drop`.
            // The `Drop` impl of `Entry` will remove itself from its `entry.collection`,
            // so combined with the guarantees above we know that the weak refs we hold
            // here can be used.
            let entry : &Entry<T> = unsafe { &**entry };
            print!(" {:?},", entry.x);
        }
        println!(" ]");
    }
}

impl<T> Drop for Collection<T> {
    fn drop(&mut self) {
        // Go through the entries to remove pointers to the collection.
        for entry in self.objects.borrow().iter() {
            // Safety: the `&weak` ref guarantees:
            // 1. that `entry.collection` cannot be changed and keeps pointing to this
            //   collection;
            // 2. that the entry won't be deallocated without running `Drop`.
            // The `Drop` impl of `Entry` will remove itself from its `entry.collection`,
            // so combined with the guarantees above we know that the weak refs we hold
            // here can be used.
            let entry : &Entry<T> = unsafe { &**entry };
            entry.collection.set(None);
        }
    }
}

impl<T> Entry<T> {
    fn new(x: T) -> Self {
        Entry { x, collection: Cell::new(None) }
    }
}

impl<T> Drop for Entry<T> {
    fn drop(&mut self) {
        // Go through collection to remove this entry.
        if let Some(collection) = self.collection.get() {
            // Safety: the `&weak` ref guarantees:
            // 1. that `collection.objects` cannot be changed without cooperation from
            //   the `Collection` API;
            // 2. that the collection won't be deallocated without running `Drop`.
            // The `Drop` impl of `Collection` will remove itself from all the entries
            // it contains, so combined with the guarantees above we know that the
            // weak ref we hold here can be used.
            let collection : &Collection<T> = unsafe { &*collection };
            collection.objects.borrow_mut().retain(|ptr| ptr.addr() != self.addr());
        }
    }
}

Here the collection keeps a &weak reference to each entry, and we rely on the Drop guarantee of &weak for the safety of our API. We could imagine using real lifetimes instead of 'static: the entry only needs to stay pinned as long as it exists inside the collection. Once we remove an entry, its &weak reference goes out of scope, so we could in theory go back to doing whatever we want with the entry.²

Of note, which wasn’t obvious to me, is that the collection too needs to be pinned. That’s because the entries need to keep a pointer to it. This highlights the two ingredients for a safe Pin-based API:

• a lifetimeless pointer to a place;
• a mechanism for the Drop impl of the pinned type to make sure we don’t use the pointer any longer.

This typically means a reference cycle, but that’s not necessary. The following example doesn’t involve a reference cycle but requires pinning:

struct A<'a> {
    // While this is `true`, `ptr` is valid for accesses.
    is_ptr_valid: &'a AtomicBool,
    ptr: &'static weak B<'a>,
}
struct B<'a> {
    flag: &'a AtomicBool,
    some_data: Data,
}
impl !Unpin for B<'_> {}

impl Drop for B<'_> {
    fn drop(&mut self) {
        self.flag.store(false, Ordering::Relaxed);
    }
}

We can also understand the need for !Unpin: without it, one could swap two Entrys, which would make their collection field no longer point to the right thing. In some way the “cause” of Entry: !Unpin is the contained &weak Collection; so when entry.collection.is_none() we could in theory safely move the entry around.³ Similarly the reason that B is !Unpin is to ensure the &AtomicBool doesn’t get swapped out while B is weakly borrowed.

So here we go, I personally found this enlightening: Pin<P> is basically a way to manage a particular kind of untracked static borrow in a safe API. When I have a Pin<P> I can store the attached weak reference wherever I want, and holding such a weak reference limits what can happen to the place just enough that we can keep things consistent and safe.⁴

Let me know if you found this mental model helpful!

I propose we could make this a more explicit part of the language by giving crate authors a common language to control their present API surface and opt-in/opt-out of future API changes.

The proposed keyword choices are very much not what I expect to be accepted; I’m just trying to share the idea for now. Please share other compelling examples! (I will edit this blog post to include more examples).

// Equivalent to `#[non_exhaustive]` on enums.
#[semver_compatible(allow(author, add_variants)]
enum Enum { ... }

enum Enum {
    // Kinda opposite of `#[non_exhaustive]`; forbids user from matching or
    // constructing this variant.
    #[semver_compatible(allow(author, remove_variant)]
    Variant1,
    ..
}

// Equivalent to `#[non_exhaustive]` on structs.
#[semver_compatible(allow(author, add_fields)]
struct Struct { ... }

// The author can add fields but only public ones. This means downstream
// crates can use FRU (idea from scottmcm, ty!).
#[semver_compatible(allow(author, add_fields(pub))]
struct Struct { ... }

// The author can add fields but only with default values. This means downstream
// crates can construct it (idea from scottmcm, ty!).
#[semver_compatible(allow(author, add_fields(with_defaults))]
struct Struct { ... }

// Allows downstream crates to rely on the layout of this struct. Could be used
// for safe transmutation to reason about API/ABI stability.
#[semver_compatible(forbid(author, change_layout))]
#[repr(C)]
struct Struct { ... }

// Bound the size of the struct.
#[semver_compatible(size <= 42)]
struct Struct { ... }

// Commit to keeping these auto traits implemented.
#[semver_compatible(implements(Send))]
#[semver_compatible(implements(const Destruct))]
struct Struct { ... }

struct Struct {
    // Forbid mutation by downstream crates. See also
    // https://rust-lang.github.io/rfcs/3323-restrictions.html .
    // This isn't a semver kind of thing so I picked another keyword but I don't like it much.
    #[api(forbid(downstream, write)]
    pub x: Data,
}

// Prevents downstream crates from implementing this trait. Basically builtin "sealed traits", see
// also https://rust-lang.github.io/rfcs/3323-restrictions.html .
// This isn't a semver kind of thing so I picked another keyword but I don't like it much.
#[api(forbid(downstream, impl)]
trait Trait { ... }

// Ensures a trait is and stays dyn safe.
#[semver_compatible(is_dyn_safe)]
trait Trait { ... }

// Prevents adding a new method if it's not const.
// IIUC, should be enough to allow `const Trait` bounds. If so, that's an
// alternative to the `const trait Trait` syntax.
#[semver_compatible(forbid(author, add_method(not_const))]
trait Trait { ... }

trait Trait {
    // Prevents downstream crates from overriding this method. Replaces `final`
    // methods (https://github.com/rust-lang/rfcs/pull/3678).
    #[api(forbid(downstream, override)]
    fn method() { ... }
    
    // Prevents downstream crates from calling this method.
    #[api(forbid(downstream, call)]
    fn method() { ... }
}


// Ensures the lifetime/type param stays covariant.
#[semver_compatible(covariant('a))]
#[semver_compatible(covariant(T))]
struct Foo<'a, T> { ... }

// Prevents `foo::<...>` syntax as the generic parameters may change, e.g. going to `impl Trait` instead of an explicit param.
#[semver_compatible(allow(author, change_generics))]
fn foo<T: Trait>(..) { ... }

// Ensures the generated coroutine implements `Send`.
#[semver_compatible(implements(Send))]
async fn foo() { ... }

I would imagine, in a future edition, having a lint that warns if you’re relying on un-committed-to API facts, e.g. you use dyn Trait for a trait that didn’t guarantee it would stay dyn-safe.

I feel like there are quite a number of features that would fit in this, and like we could find a nice common language to talk about those things. This is a call for contributions: do you see other examples?