Skip to content

Using Rust enums to represent C enums can result in undefined behavior #667

@mjbshaw

Description

@mjbshaw

This is similar to #225, but different in that this is about C enums that are used as actual enums (not bit flags).

Consider, for example, the following C code:

typedef enum {
  STATUS_OK,
  STATUS_BAD_ARG,
  STATUS_NO_MEMORY,
} Status;

This gets translated by bindgen into:

#[repr(u32)]
#[derive(Debug, Copy, Clone, PartialEq, Eq, Hash)]
pub enum Status { STATUS_OK = 0, STATUS_BAD_ARG = 1, STATUS_NO_MEMORY = 2, }

The problem is that the C project could add enum fields (which is often considered to not be an API/ABI breaking change in the C world) to the end (e.g. a new STATUS_BAD_AUTH = 3 field), at which point using the (old) Rust enum would result in undefined behavior due to Rust's strict enum range requirements. If a function in the C API ever returned STATUS_BAD_AUTH then the Rust code would have undefined behavior. Static linking the C library will help mitigate this, but if the library is dynamically linked then the Rust code really can't be considered safe.

Here are my current thoughts on how to solve this, though other solutions are likely possible:

  • By default, don't use Rust enums to represent C enums. Introduce a new type (e.g. a simple struct) that can be used instead.
  • Allow the bindgen user to specify a flag that will make bindgen generate Rust enums to represent C enums only for C enums that are used as input parameters. For example, if the C function is enum foo do_something(enum bar) then the bar enum could be represented safely as a Rust enum, since it would only be constructed in the Rust code. If all the C functions that bindgen finds only take an enum by value (or a pointer to const enum), then this should be a safe optimization to make.
  • Allow the bindgen user to specify a different flag that will make bindgen generate Rust enums for all C enums. This would be an unsafe option, but would allow the user to take advantage of potential compiler optimizations, and in a controlled environment (e.g. statically linking to a known library version) can be done safely.

This would be a nontrivial amount of work, but I think at least the first point is important for ensuring the safety of Rust programs that have to interact with C libraries.

See rust-lang/rust#36927 for the relevant issue filed against Rust itself.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions