Skip to content

Emitting &[u8] literal is very slow #2

@SOF3

Description

@SOF3

Issue

rustc takes forever to parse the generated &[u8] literal. I have checked that it is indeed compiled into a raw byte array (no delimiters, no boxing, etc.) in the output executable (thus satisfying the size requirements), but rustc doesn't seem to know to parse the contents in a faster way.

Alternatives

An alternative would be to write the deflated stream into a temp file and use include_bytes!(), but I don't know where the file should be created.

  • If I write it to the OS temp directory, it would result in an OS-level resource leak before rebooting (because nobody is deleting the temp files).
  • If I always write to the same file in the target directory, concurrent compilations might overwrite each other.
  • If I write to somewhere next to the included file, it corrupts the user filesystem. Rust does not recommend build scripts to affect files beyond the OUT_DIR. However, for macros, we don't even have OUT_DIR or anything within the target/ directory.

A solution (still bad but affects less negatively) is to create a hash based on the full file name in the operating system, and save the file content hash + compressed contents using a file within target uniquely identified by the hash. A unique file lock can be acquired while writing. If the file content hash is already identical to the compressed contents, this means (although unlikely) that another compilation has already compressed the file, and there is no need to recompile, preventing reducing the chance of race condition between rustc's handling include_bytes!() and another process's macro compilation.
But where within the target directory? I am not sure.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghelp wantedExtra attention is needed

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions