I have tested this PR in https://github.com/Atry/nix-ld-so-cache-test, it resolves cachix/devenv#773

Hello there, please ensure you read the CONTRIBUTING guide, this is a change touching glibc, therefore mass rebuilding, therefore it should target at least staging, please read the CONTRIBUTING guide on how to perform retargeting.

On the changes themselves, I am not so sure about the triviality of adding a new environment variable, this makes us diverge from vanilla glibc, while it's true, it fixes a pain point, any new attempt should take into account the previous attempts at this, e.g. NIX_LD_LIBRARY_PATH, etc. (please look for the PRs, I don't have the number in mind).

Furthermore, I am not sure how this interacts with #207893 which is clearly a change that benefits a larger number of folks.

Either case, I don't feel good about this PR without having enough of our expert matter looking into it, I will add them as reviewers, please understand that it may take some time to proceed regarding this.

Please don't add any new ways for getenv() to influence library loading.

This PR definitely creates a trivial exploit, since you didn't add NIX_LD_SO_CACHE to UNSECURE_ENVVARS in sysdeps/generic/unsecvars.h.

Even if you do add it to that list, it is far from clear that it does not create other exploits. We don't really have the resources to do this kind of analysis, and we would have to repeat the analysis on every glibc version bump.

Please don't add any new ways for getenv() to influence library loading.

Do you suggest that the only secure way to add an environment variable for the interpreter is to add it to upstream glibc and make ensure that the upstream maintainer is aware of it in version bumping, like #248547 ?

Do you suggest that the only secure way to add an environment variable for the interpreter is to add it to upstream glibc and make ensure that the upstream maintainer is aware of it in version bumping, like #248547 ?

I would agree with @amjoseph-nixpkgs analysis here, it seems saner to move it to a "non vanilla glibc" variant that can be used "exceptionally" rather than "default", or, this needs to be upstreamed or brought forward to glibc folks.

We do know some stuff, but we are not glibc developers, and if anything, we probably know how wicked those things can be.

Furthermore, I am not comfortable creating a precedent to accept this type of change in vanilla glibc.
I am more convinced by external approaches as https://github.com/Mic92/nix-ld and bringing them in a meaningful way to make them actually usable.

nix-ld is a nice tool, but it does not solve #327854 , especially when a nix binary loads a shared library, which does not have runpaths to nix store.

Vanilla Python packages in manylinux2014 ABI do not have runpaths.

nix-ld is a nice tool, but it does not solve NixOS/nixpkgs#327854 , especially when a nix binary loads a shared library, which does not have runpaths to nix store.

Vanilla Python packages in manylinux2014 ABI do not have runpaths.

If you have a Nix-built binary which does not have its RUNPATH completely filled and set, there's a bigger problem here. The toolchain was not correctly set for success.

You have two worlds:

• Nix-built binaries: correct-by-construction, they have all their RUNPATH set properly.
• External binaries on NixOS: will read the FHS dynamic loader, with nix-ld, they will just work because nix-ld is exactly that: a compatibility FHS dynamic loader.

If you need more than this model, you have to explain why you are sitting in the "in-between" which is hard to control.

If by this, you mean running Nix's Python and mixing with manylinux2014 ABI binaries, hah, I would argue that the fact that it won't work by default, is a feature to me. Python PyPI is full of malware and wheels is an opaque binary format unreviewed by PyPA, NixOS is nice to thwart any attempt to run arbitrary code that I don't trust.

If folks want this model to work out of the box, I would urge them to build a second package set for those, I am definitely against them by default.

I think the authors intention here is to extend interpreters library path. This is not covered by nix-ld yet. Let's take for example installing a native python extension with pip that needs an additional native library. Just now one can set LD_LIBRARY_PATH to fix this but than this LD_LIBRARY_PATH is also picked up by other applications where it overrides the RPATH of our programs. This can lead to crashes if incompatible libraries are loaded (i.e. library compiled against different glibc versions). I haven't checked yet the implications of NIX_LD_SO_CACHE, but some environment variable that has lower priority than the RPATH of the binary would be great.

I think the authors intention here is to extend interpreters library path. This is not covered by nix-ld yet. Let's take for example installing a native python extension with pip that needs an additional native library. Just now one can set LD_LIBRARY_PATH to fix this but than this LD_LIBRARY_PATH is also picked up by other applications where it overrides the RPATH of our programs. This can lead to crashes if incompatible libraries are loaded (i.e. library compiled against different glibc versions). I haven't checked yet the implications of NIX_LD_SO_CACHE, but some environment variable that has lower priority than the RPATH of the binary would be great.

That's what the linked PR #248547 offers I believe.

Yes, this PR and #248547 are very similar

You have two worlds:

Nix-built binaries: correct-by-construction, they have all their RUNPATH set properly.
...

...have their runpath set properly, unless they use OpenGL or CUDA. There's at least one in-between world: "Nix-built in not-a-NixOS runtime"

I think nix-ld is an absolutely wonderful approach to solve "FHS-on-NixOS" (definitely preferable to fhs user namespaces): it's a decent UX, and it only affects applications that actually expect an FHS environment. However, OpenGL/CUDA/etc programs built by Nix aren't quite "correct" or "cross-platform" today. They are usable outside NixOS with the help of extra tools, but if we wanted to actually support FHS environments, we'd have to patch ld.so more?

P.S. I'm still unfamiliar with #207893 and with how far it's progressed

P.P.S. @amjoseph-nixpkgs what is your opinion about nix-ld? It does, in principle, expose a new variable...

"Nix-built in not-a-NixOS runtime"

Nitpick: this should be "Nix-built in FHS runtime". There are plenty of non-NixOS nixpkgs-based systems out there. Spectrum is one.

P.P.S. @amjoseph-nixpkgs what is your opinion about nix-ld? It does, in principle, expose a new variable...

I haven't tried nix-ld myself, but it appears to be an example of the approach that I prefer: don't modify the "standard" linker/elf-interpreter; instead provide a modified one that is used if/when necessary.

Nitpick: this should be "Nix-built in FHS runtime"

👍🏻 ❤️

I haven't tried nix-ld myself, but it appears to be an example of the approach that I prefer: don't modify the "standard" linker/elf-interpreter; instead provide a modified one that is used if/when necessary.

Yes and no: from the point of view of the FHS app, nix-ld "replaces" /lib64/ld-linux-x86-64.so.2, which is easy because it's linked impurely. We, otoh, link a concrete loader, and that's exactly what would (and should) be used in the FHS environment as in any other environment, so we likely want it to support an escape hatch in the lines of LD_FALLBACK_LIBRARY_PATH.

So, what does the process of "talking to glibc people" look like if we want one of these solutions upstreamed?

I'd recommend sending an email explaining the whole problem and situation and trying to convince them of re-introducing something like RPATH (not RUNPATH).

It was the nix community's decision to use RUNPATH instead of RPATH. GNU's dynamic linker supports both.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/on-nixpkgs-and-the-ai-follow-up-to-2023-nix-developer-dialogues/37087/2