I'm opening this mainly for discussion for now. I believe that at least some load limit is better than none.

I was testing this on a 24-core machine to observe the behavior. Using $(nproc) * 3/2 was still leaving some periods where there was a noticeable under-utilization of CPU (by atop) due to the oscillation – though not large amount and not with all builds (llvm builds were doing this but not linux builds).

So I chose $(nproc) * 2 to improve on this. Also weakening the limit seemed more in line with my perception of community's reaction around the PR #192447.

Ideally, I think should have this support on the nix side. Similar to how you can configure jobs and cores.

That being said, I think this is better than what we had.

Yes, adding a similar config to nix to populate NIX_LOAD_LIMIT could be done later, if this route gets consensus (in nixpkgs first). Now it doesn't take so much time for nix changes to get into the default version in nixpkgs.

For my setup a load limit twice the cores is too high. It is more likely that my machines run out of memory or starting to swap.
I think we have to introduce NIX_LOAD_LIMIT first (see NixOS/nix#6855) or set the value to a more conservative value, e.g. NIX_LOAD_LIMIT=$(ncores).

For my setup a load limit twice the cores is too high. It is more likely that my machines run out of memory or starting to swap.

Yea, there's a big difference between a 2-core CI machine, and something like my 128 vCPU server :/

A good default for everyone seems impossible with the mechanisms at hand. Generally I believe it helps to set zramSwap.enable = true;. (stalls due to zram-swapping will increase load, too)

I created NixOS/nix#7091 to make a longer term solution viable.

Yea, there's a big difference between a 2-core CI machine, and something like my 128 vCPU server :/

Note that most machines in Hydra have between 24 and 80 cores, but have between 2 and 80 cores allocated to the build. I agree there isn't a heuristic here that will be robust across hardware configurations.

People who have RAM issues still have the option of reducing cores, and I'd argue that that's more appropriate. Load doesn't rise immediately, so it's likely that there's a burst with lots of processes at the beginning of a build (observed on hydra before the removal of the load limit). This can easily exhaust RAM too.

Yea, looked into what load actually provides, and I don't think it's as useful for burst workloads :(.

But by reducing --cores you slow down big builds, and many other builds are single-core. It's hard to find a good fixed split, unless you can be patient like Hydra (--cores 2 for almost all jobs).

Linking #328677 for those subscribed to this.