stdenv: allow for jobservers across multiple nix builds #314888

RaitoBezarius · 2024-05-26T16:48:27Z

Description of changes

Retake of #143820 where I unfortunately fucked up trying to debug kernel issues and btrfs.

Original motivations by @pennae:

make -jN -lN in stdenv is a very blunt instrument. it works well when max-jobs=1, but as nix-level paralellism increases it becomes increasingly deficient. starting from a low-load situation we start max-jobs * N compilers, loadavg goes through the roof, the -lN load limit kicks in and inhibits new compilers starting until loadavg has fallen below N—at which point all make instances spawn a lot of new compilers and loadavg goes through the roof again. this oscillation leaves the system underutilized in low phases and overcommitted in high phases.

testing the current stdenv against a jobserver with 26 tokens on a 12C/24T machine shows that parallel builds of llvm_{8..11} run about 7% faster (35:52min for stdenv, 33:30min with jobserver), a larger build of llvm{5..13} is about about 11% faster (1:27h for stdenv, 1:17h with jobserver). (removing the -l from stdenv also improves utilization but is less efficient. preliminary testing here shows that -l${1.5 * N} may be a good alternative to -lN as used currently, #141266 could be a good vector to go for that instead of this whole mess. [more testing says that -l2N would be a minimum to get better utilization, but so far every -l setting we've tried has produced some underutilization except excessive large numbers like 6N or higher])

nothing in here should be regarded as a final suggestion in any way, it's more of a "hey look, this might just work". as such it's extremely rough around the edges, eg to use the jobserver the experimenter currently has to bring a /jobserver fifo filled with tokens into the nix sandbox:

nix-build -E 'with import ./. {}; pkgs.callPackage ./pkgs/os-specific/linux/nixos-jobserver {}'
touch /tmp/jobserver
sudo ./result -t 24 -g nixbld /tmp/jobserver&
nix build --extra-sandbox-paths "/jobserver=/tmp/jobserver" #...

is this something worth pursuing? a 10% speedup for hydra does seem tempting

todos before this is more generally usable:

re-add $NIX_BUILD_CORES support (ioctl on the jobserver fd should do it)
maybe port from cuse to fuse to not require root for everything (mmap problems may be solved by reporting jobserver file size as 0 at all times, needs testing)
usability of tools (error messages, runtime configuration, etc)
add support for other build systems (cargo can honor MAKEFLAGS, maybe others too)
nixos module
make configurable which env var(s) jscall adds jobserver information to
metrics?
<your suggestion here>

Things done

pennae · 2024-05-27T02:32:42Z

doc/stdenv/stdenv.chapter.md

Suggested change

### `runInJobServer` \<command\> \-\-\-\- \<defArgs\> \-\-\-\- \<args\> {#fun-runInJobServer}

### `runInJobserver` \<command\> \-\-\-\- \<defArgs\> \-\-\-\- \<args\> {#fun-runInJobServer}

(multiple places)

pennae · 2024-05-27T02:33:35Z

nixos/modules/services/misc/nixos-jobserver.nix

running this as root is most unwise in practice but running it as another user seems to be nearly impossible. when fuse doesn't fuck us over systemd does.

pennae · 2024-05-27T02:34:04Z

pkgs/by-name/ni/nixos-jobserver/nixos-jobserver.cpp

Suggested change

.st_mode = S_IFREG | 0660,

.st_mode = S_IFREG | 0666,

won't work otherwise

pennae · 2024-05-27T02:34:49Z

pkgs/stdenv/generic/setup.sh

Suggested change

-j${NIX_BUILD_CORES} -l${NIX_BUILD_CORES} ---- \

-j${NIX_BUILD_CORES} ---- \

multiple places

pennae · 2024-05-27T19:06:39Z

cargo apparently no longer accepts jobserver fds that aren't pipes. that's somewhat problematic.

initially only make and cargo support using the jobserver. other build systems may follow suit later.

Co-authored-by: Raito Bezarius <masterancpp@gmail.com>

aviallon · 2024-07-30T19:11:01Z

Is this work dead? It looked very promising.

yu-re-ka · 2024-08-12T09:56:10Z

I have some patches lying around locally to make it work with ninja, so anyone interested in picking this up, feel free to write me about it.

lf- · 2025-02-15T22:30:07Z

FYI for anyone looking at this in the future: the state of this is that various people are talking about this in Lix and something better may appear eventually, but the approach of Make jobservers really does not work and a new protocol needs to exist. Here's a pad where it is being discussed: https://pad.lix.systems/lix-jobserver

Remove the requirement that the jobserver "fifo" is actually a named pipe. Named pipes are essentially stateless, and therefore carry a high risk of a killed process leaving the server with no tokens left, and no clear way to reclaim them. Therefore, multiple jobserver implementations use FUSE instead: - [nixos-jobserver](NixOS/nixpkgs#314888) (WIP) uses simple file on FUSE - [steve](https://gitweb.gentoo.org/proj/steve.git) uses a character device via CUSE - [guildmaster](https://codeberg.org/amonakov/guildmaster) uses a character device via CUSE This is compatible with GNU make and Ninja, since they do not check the file type, and seems to be the only solution that can achieve state tracking while preserving compatibility. Signed-off-by: Michał Górny <mgorny@gentoo.org>

The jobserver specification [0] currently suggests that the FIFO must be a genuine FIFO. For some work we're doing [1][2], we're emulating a FIFO using CUSE/FUSE to allow tracking when consumers disappear to avoid lost tokens. nixos had a similar idea in the past too [3]. There doesn't seem to be a good reason to check that any FIFO passed by the user is actually identifiable as such by `stat()`, so drop the check. make already does not perform such a check, just the specification isn't clear about it, so we've asked them to clarify it [4]. [0] https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html [1] https://codeberg.org/amonakov/guildmaster [2] https://gitweb.gentoo.org/proj/steve.git/ [3] NixOS/nixpkgs#314888 [4] https://savannah.gnu.org/bugs/index.php?67726

The jobserver specification [0] currently suggests that the FIFO must be a genuine FIFO. For some work we're doing [1][2], we're emulating a FIFO using CUSE/FUSE to allow tracking when consumers disappear to avoid lost tokens. nixos had a similar idea in the past too [3]. make already does not perform such a check, just the specification isn't clear about it, so we've asked them to clarify it [4]. [0] https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html [1] https://codeberg.org/amonakov/guildmaster [2] https://gitweb.gentoo.org/proj/steve.git/ [3] NixOS/nixpkgs#314888 [4] https://savannah.gnu.org/bugs/index.php?67726 Signed-off-by: Sam James <sam@gentoo.org>

Remove the requirement that the jobserver "fifo" is actually a named pipe. Named pipes are essentially stateless, and therefore carry a high risk of a killed process leaving the server with no tokens left, and no clear way to reclaim them. Therefore, multiple jobserver implementations use FUSE instead: - [nixos-jobserver](NixOS/nixpkgs#314888) (WIP) uses simple file on FUSE - [steve](https://gitweb.gentoo.org/proj/steve.git) uses a character device via CUSE - [guildmaster](https://codeberg.org/amonakov/guildmaster) uses a character device via CUSE This is compatible with GNU make and Ninja, since they do not check the file type, and seems to be the only solution that can achieve state tracking while preserving compatibility. Signed-off-by: Michał Górny <mgorny@gentoo.org>

Do not require that the jobserver path corresponds to an actual named pipe. Token-accounting jobserver implementations such as steve [1] and guildmaster [2] use a character device instead, while the draft version of nixos-jobserver [3] uses a pseudo-file via FUSE. GNU make does not enforce a named pipe either. [1] https://gitweb.gentoo.org/proj/steve.git [2] https://codeberg.org/amonakov/guildmaster [3] NixOS/nixpkgs#314888 Signed-off-by: Michał Górny <mgorny@gentoo.org>

@amonakov

Remove the requirement that the jobserver "fifo" is actually a named pipe. Named pipes are essentially stateless, and therefore carry a high risk of a killed process leaving the server with no tokens left, and no clear way to reclaim them. Therefore, multiple jobserver implementations use FUSE instead: - [nixos-jobserver](NixOS/nixpkgs#314888) (WIP) uses simple file on FUSE - [steve](https://gitweb.gentoo.org/proj/steve.git) uses a character device via CUSE - [guildmaster](https://codeberg.org/amonakov/guildmaster) uses a character device via CUSE This is compatible with GNU make and Ninja, since they do not check the file type, and seems to be the only solution that can achieve state tracking while preserving compatibility. CC @amonakov --------- Signed-off-by: Michał Górny <mgorny@gentoo.org>

@amonakov

…(#169154) Remove the requirement that the jobserver "fifo" is actually a named pipe. Named pipes are essentially stateless, and therefore carry a high risk of a killed process leaving the server with no tokens left, and no clear way to reclaim them. Therefore, multiple jobserver implementations use FUSE instead: - [nixos-jobserver](NixOS/nixpkgs#314888) (WIP) uses simple file on FUSE - [steve](https://gitweb.gentoo.org/proj/steve.git) uses a character device via CUSE - [guildmaster](https://codeberg.org/amonakov/guildmaster) uses a character device via CUSE This is compatible with GNU make and Ninja, since they do not check the file type, and seems to be the only solution that can achieve state tracking while preserving compatibility. CC @amonakov --------- Signed-off-by: Michał Górny <mgorny@gentoo.org>

@amonakov

Remove the requirement that the jobserver "fifo" is actually a named pipe. Named pipes are essentially stateless, and therefore carry a high risk of a killed process leaving the server with no tokens left, and no clear way to reclaim them. Therefore, multiple jobserver implementations use FUSE instead: - [nixos-jobserver](NixOS/nixpkgs#314888) (WIP) uses simple file on FUSE - [steve](https://gitweb.gentoo.org/proj/steve.git) uses a character device via CUSE - [guildmaster](https://codeberg.org/amonakov/guildmaster) uses a character device via CUSE This is compatible with GNU make and Ninja, since they do not check the file type, and seems to be the only solution that can achieve state tracking while preserving compatibility. CC @amonakov --------- Signed-off-by: Michał Górny <mgorny@gentoo.org>

@amonakov

Remove the requirement that the jobserver "fifo" is actually a named pipe. Named pipes are essentially stateless, and therefore carry a high risk of a killed process leaving the server with no tokens left, and no clear way to reclaim them. Therefore, multiple jobserver implementations use FUSE instead: - [nixos-jobserver](NixOS/nixpkgs#314888) (WIP) uses simple file on FUSE - [steve](https://gitweb.gentoo.org/proj/steve.git) uses a character device via CUSE - [guildmaster](https://codeberg.org/amonakov/guildmaster) uses a character device via CUSE This is compatible with GNU make and Ninja, since they do not check the file type, and seems to be the only solution that can achieve state tracking while preserving compatibility. CC @amonakov --------- Signed-off-by: Michał Górny <mgorny@gentoo.org>

RaitoBezarius requested review from Ericson2314, figsoda, winterqt and zowoq as code owners May 26, 2024 16:48

lilyinstarlight added the 1.severity: significant Novel ideas, large API changes, notable refactorings, issues with RFC potential, etc. label May 26, 2024

pennae requested changes May 27, 2024

View reviewed changes

stdenv: allow for jobservers across multiple nix builds

742725b

initially only make and cargo support using the jobserver. other build systems may follow suit later.

pennae force-pushed the stdenv-jobserver branch from 8a4a67a to cd4036f Compare May 27, 2024 19:27

nixos/nixos-jobserver: init

e97220e

Co-authored-by: Raito Bezarius <masterancpp@gmail.com>

pennae force-pushed the stdenv-jobserver branch from cd4036f to e97220e Compare May 28, 2024 19:49

h7x4 added the 8.has: module (new) This PR adds a module in `nixos/` label Jun 1, 2024

wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Sep 10, 2024

wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jan 2, 2025

nixpkgs-ci bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 25, 2025

github-project-automation bot added this to Stdenv Jun 25, 2025

nixpkgs-ci bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Aug 16, 2025

mgorny mentioned this pull request Nov 22, 2025

[LLVM] Remove the requirement for named pipe in jobserver llvm/llvm-project#169154

Merged

thesamesam mentioned this pull request Nov 22, 2025

Don't require a FIFO to be identifiable as such ninja-build/ninja#2692

Merged

mgorny mentioned this pull request Dec 6, 2025

Remove the FIFO requirement tommilligan/pytest-jobserver#149

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

stdenv: allow for jobservers across multiple nix builds #314888

stdenv: allow for jobservers across multiple nix builds #314888

RaitoBezarius commented May 26, 2024

Uh oh!

pennae May 27, 2024

Uh oh!

pennae May 27, 2024

Uh oh!

pennae May 27, 2024

Uh oh!

pennae May 27, 2024

Uh oh!

pennae commented May 27, 2024

Uh oh!

aviallon commented Jul 30, 2024

Uh oh!

yu-re-ka commented Aug 12, 2024

Uh oh!

lf- commented Feb 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

	### `runInJobServer` \<command\> \-\-\-\- \<defArgs\> \-\-\-\- \<args\> {#fun-runInJobServer}
	### `runInJobserver` \<command\> \-\-\-\- \<defArgs\> \-\-\-\- \<args\> {#fun-runInJobServer}

	-j${NIX_BUILD_CORES} -l${NIX_BUILD_CORES} ---- \
	-j${NIX_BUILD_CORES} ---- \

Uh oh!

stdenv: allow for jobservers across multiple nix builds #314888

Are you sure you want to change the base?

stdenv: allow for jobservers across multiple nix builds #314888

Conversation

RaitoBezarius commented May 26, 2024

Description of changes

Original motivations by @pennae:

Things done

Uh oh!

pennae May 27, 2024

Choose a reason for hiding this comment

Uh oh!

pennae May 27, 2024

Choose a reason for hiding this comment

Uh oh!

pennae May 27, 2024

Choose a reason for hiding this comment

Uh oh!

pennae May 27, 2024

Choose a reason for hiding this comment

Uh oh!

pennae commented May 27, 2024

Uh oh!

aviallon commented Jul 30, 2024

Uh oh!

yu-re-ka commented Aug 12, 2024

Uh oh!

lf- commented Feb 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants