Speed up application startup by using `ld.so.cache` by winterqt · Pull Request #207061 · NixOS/nixpkgs

winterqt · 2022-12-21T00:14:48Z

Description of changes

This PR does two things:

Patch glibc's dynamic loader to load the ld.so cache from $ORIGIN/../etc/ld.so.cache instead of /etc/ld.so.cache, and to prefer the cache over RPATH entries
Introduce a new setup hook, generate-ld-cache, to facilitate the generation of these caches

This was originally implemented by Guix's Ludovic Courtès in August of 2021 (after Ricardo came up with the idea) -- see their blog post and their discussion on the change for more information + some benchmarks. The glibc patch is copied verbatim from the Guix codebase.

As noted in the linked Guix issue, there are a few things to consider here:

The hard-coded $ORIGIN/../etc/ld.so.cache path means that it can only be used with first-level sub-directories like bin and sbin
There is a possibility that users could be tricked into loading a malicious ld.so.cache
If you read the discussion, you'll note that I omitted one of their points here: that transient dependencies aren't added to the cache. Our implementation actually does include transient dependencies, because we use ldd to get the required dependencies, as opposed to just using the data from the ELF.

When discussing this with pennae, they brought up the idea of embedding an absolute path to the cache in an ELF section, and adjusting the patch to do that. We may want to do this instead; I'd appreciate thoughts on the matter.

I'd like to thank the Guix folks mentioned above for originally coming up with and implementing this idea, as well as pennae for nerd sniping me into looking at perf stuff, and their thoughts on the matter.

Things done

winterqt · 2022-12-21T01:25:55Z

After looking around some more, I stumbled across @fzakaria'a nix-harden-needed, which may be a better choice over Guix's solution (per the README, they want to integrate it into Nixpkgs).

(Farid: hi, figured I'd loop you in here, considering you have interest in this problem :)

fzakaria · 2022-12-21T02:56:40Z

After looking around some more, I stumbled across @fzakaria'a nix-harden-needed, which may be a better choice over Guix's solution (per the README, they want to integrate it into Nixpkgs).

(Farid: hi, figured I'd loop you in here, considering you have interest in this problem :)

Hi Winter!

Thank you for pinging me.
This is something I would love to solve at nixpkgs level -- I've just been intimated doing the solution although I've audited the code and have a good sense of what to do.

My hot-take:
1 - we can take https://github.com/fzakaria/shrinkwrap and rewrite it in C (it's using LIEF C-library) so the closure is very small and we can include it in the bootstrap binary.
With shrinkwrap we can take a much simpler "patchelf" style approach and fixup the binaries.
2 - overtime do harden-needed approach to make it more "robust"
3 - I really want to investigate using $ORIGIN in the path so that the binaries are relocatable for any nix-store path.
(This is something Spack package manager accomplishes already)
(We actually need to upstream $ORIGIN support for PT_INTERP)

Edit: This is a great PR btw -- very concise if it works. Only unfortunate thing is patching but maybe you can upstream something to make it configurable based on an ELF file section.

It has the added benefit of supporting LD_LIBRARY_PATH which the other options don't (maybe though that's a feature?)

fzakaria · 2022-12-21T02:58:35Z

Here is a document of some other much more "visionary" ideas I have for Nix + NixOS.

Breaking_Apart_the_Shared_Object_Monolith.pdf

winterqt · 2022-12-21T03:02:36Z

So I hope you don't mind me asking this here (if you'd rather discuss over e.g. IRC, Matrix, or email to avoid cluttering up this PR, let me know), but what does Shrinkwrap provide that this cache approach doesn't? They seemingly both eliminate the intensive RPATH traversal.

I know there's one advantage you list in the README:

Secondly, shared dynamic objects may be found due to the fact that they are cached during the linking stage. Meaning, if another shared object requires the same dependency but failed to specify where to find it, it may still properly resolved if discovered earlier in the linking process. This is extremely error prone and changing any of the executable's dependencies can change the link order and potentially cause the binary to no longer work.

but I admit I'm confused as to what a practical example of this would look like/what the problem is in general 😅. Do you mind explaining?

winterqt · 2022-12-21T03:04:05Z

Edit: This is a great PR btw -- very concise if it works. Only unfortunate thing is patching but maybe you can upstream something to make it configurable based on an ELF file section.

It does work; I've tested it with hello, and any Guix user has been using it since last year. Not sure how much credit I can take, since it's just some Bash + the Guix folks' patch, but thank you :)

I think an advantage to this solution is definitely that it's minimally invasive and concise, as you mention.

It has the added benefit of supporting LD_LIBRARY_PATH which the other options don't (maybe though that's a feature?)

(Why doesn't Shrinkwrap support it, since it just lifts dependencies up to the parent ELF?)

fzakaria · 2022-12-21T03:07:54Z

Reached out on Matrix.
(@winterqt:nixos.dev)

Mindavi · 2022-12-21T07:16:04Z

Some questions:

Does this work when cross-compiling?
Are there any implications for content-addressed derivations with this?

raphaelr · 2022-12-21T09:28:00Z

pkgs/stdenv/generic/default.nix

      ../../build-support/setup-hooks/set-source-date-epoch-to-latest.sh
      ../../build-support/setup-hooks/strip.sh
-    ] ++ lib.optionals hasCC [ cc ];
+    ] ++ lib.optionals hostPlatform.isLinux [ ../../build-support/setup-hooks/generate-ld-cache.sh ]


hostPlatform.isLinux is no specific enough, for example pkgsMusl doesn't have a ldconfig. Maybe hostPlatform.isLinux && hostPlatform.isGnu?

generating LD cache for ELF executables in /nix/store/bgm2nivmwfa94pw6xlial071z9 0vv5pg-nano-7.1 found lib: /nix/store/62hdd7jfmj90wrwl6dp94hj6xzl9cgvq-ncurses-6.3-p20220507/lib /libncursesw.so.6 found lib: /nix/store/mpgmlwjdd9g61w92q5d9g8y7r66fp8j6-musl-1.2.3/lib/ld-musl-x8 6_64.so.1 found lib dirs: /nix/store/62hdd7jfmj90wrwl6dp94hj6xzl9cgvq-ncurses-6.3-p2022050 7/lib /nix/store/mpgmlwjdd9g61w92q5d9g8y7r66fp8j6-musl-1.2.3/lib /nix/store/6j095nxqj7qylwbbp4m00kb640gca06x-generate-ld-cache.sh: line 41: ldcon fig: command not found failed to generate LD cache

(nix-build -A pkgsMusl.nano)

Good point this only works for Glibc

pennae · 2022-12-21T10:47:07Z

building stdenv comes up with some errors (that are probably not fatal only because shell is terrible):

gnum4> /nix/store/6j095nxqj7qylwbbp4m00kb640gca06x-generate-ld-cache.sh: line 5: ldd: command not found
xz> /nix/store/6j095nxqj7qylwbbp4m00kb640gca06x-generate-ld-cache.sh: line 5: ldd: command not found
zlib> /nix/store/6j095nxqj7qylwbbp4m00kb640gca06x-generate-ld-cache.sh: line 5: ldd: command not found
gettext> ldd: $warning: you do not have execution permission for `/nix/store/w8dzi3li1zx3dnpsilp3s9wah18g0f2b-gettext-0.21/lib/preloadable_libintl.so'
# etc

might be better to add this hook to the final linux stdenv stage only.

cross-building an armv7 test system from scratch fails at python:

error: builder for '/nix/store/2wx96x4ci01yw9p2wi3ad1kg78wijnp1-python3-3.10.9-env.drv' failed with exit code 1;
       last 1 log lines:
       > error: collision between `/nix/store/lc038xi6in0x4i11ypm4dcbhg1lla2ny-dtc-1.6.1/etc/ld.so.cache' and `/nix/store/yyvlli12hj3i442ww29pdcbmvdg2kh0k-python3-3.10.9/etc/ld.so.cache'
       For full logs, run 'nix log /nix/store/2wx96x4ci01yw9p2wi3ad1kg78wijnp1-python3-3.10.9-env.drv'.

buildEnv (and presumably symlinkJoin and friends) don't seem to like the approach :(

edit
cross-building does not look like it's going so well:

nix> /nix/store/6j095nxqj7qylwbbp4m00kb640gca06x-generate-ld-cache.sh: line 41: /nix/store/1ff0pbnn4jx6amdf9jxn0y0i39v02ngm-glibc-armv7l-unknown-linux-gnueabihf-2.35-224-bin/bin/ldconfig: cannot execute binary file: Exec format error

winterqt · 2022-12-21T12:05:45Z

buildEnv (and presumably symlinkJoin and friends) don't seem to like the approach :(

Adding etc/ldconfig.so.cache to this block will fix buildEnv:
https://github.com/NixOS/nixpkgs/blob/339ff0387aa51495586b69c9052e573465126bcb/pkgs/build-support/buildenv/builder.pl#L125-133

Not sure what to do about symlinkJoin, though; maybe it doesn't suffer from this issue (or at least won't fail to build)? Will test ASAP.

fzakaria · 2022-12-21T17:22:03Z

pkgs/build-support/setup-hooks/generate-ld-cache.sh

+
+fixupOutputHooks+=('if [ -z "${dontGenerateLDCache-}" ]; then generateLDCache "$prefix"; fi')
+
+generateLDCache() {


why not just use patchelf to list the dependencies rather than this bash ?
(does patchelf list the needed transitive? if not, you can write a tiny C binary using https://github.com/lief-project/LIEF)

Might be more maintainable.

we've done some investigating and it looks like a purpose-built tool that crawls a dep tree and generates a cache file will be the best thing to do. ld and ldconfig apparently can't be made to work in cross-builds (ldd certainly not, for ldconfig we've found nothing doing it with glibc. only one instance with uclibc, which doesn't want to build for armv7 right now (and that doesn't bode well for using its ldconfig))

would very much like to make this rust though, C ought to disappear and rustc is in our closures anyways

nixos-discourse · 2023-01-01T21:26:08Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/reducing-stat-calls-for-library-loading-during-application-startup/24358/1

tomberek · 2023-01-02T04:15:46Z

There was a similar patch and discussion from NixOS/patchelf#357 (comment) that started to fold this into the patchelf phase. I'm a fan of the ld.so.cache approach due to the possibility of providing a different cache en-masse, or to mutable locations for testing/debugging!) (but the shrinkwrap also has merits.

Looks like the ldd searching for transitive deps is the main difference. I fear this makes the approach more fragile. What are the pros/cons here?

When discussing this with pennae, they brought up the idea of embedding an absolute path to the cache in an ELF section, and adjusting the patch to do that. We may want to do this instead; I'd appreciate thoughts on the matter.

This would make it easier to then patch or redirect to a new place later. I can imagine a patchelf --set-ld-so-cache /some-project/etc/ld.so.cache or as a more focused alternative to lib.replaceDependency:

replaceLibDependency drv {
  drv = drv;
  from = openssl_old;
  to = openssl_new;
}

ncfavier · 2023-01-02T10:03:05Z

pkgs/development/libraries/glibc/dl-cache.patch

+      static const char store[] = @STORE_DIRECTORY@;
+      const char *origin = _dl_get_origin ();
+
+      /* Check whether ORIGIN is something like "/gnu/store/…-foo/bin".  */


Suggested change

+ /* Check whether ORIGIN is something like "/gnu/store/…-foo/bin". */

+ /* Check whether ORIGIN is something like "/nix/store/…-foo/bin". */

nixos-discourse · 2023-01-03T12:59:13Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2022-01-03-nixpkgs-architecture-team-meeting-23/24404/1

nixos-discourse · 2023-08-07T20:26:19Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/where-did-you-get-stuck-in-the-nix-ecosystem-tell-me-your-story/31415/2

nyabinary · 2023-12-19T15:47:13Z

What the status on this currently?

nixos-discourse · 2024-08-22T20:06:10Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/where-did-you-get-stuck-in-the-nix-ecosystem-tell-me-your-story/31415/72

diniamo · 2024-09-17T20:45:32Z

I don't understand, why aren't fzakaria's solutions plain better, compared to a cache?

fzakaria · 2024-09-17T21:11:17Z

I have some interesting work in similar vein for relocation processing that is a clear win for Nixpkgs as well; I plan to speak + write about it.

LordGrimmauld · 2025-03-28T12:48:11Z

This looks interesting, is there any progress still being made? Did everyone involved just leave the nix ecosystem, or what happened? And what would be needed to get this to a usable state?

fzakaria · 2025-03-31T23:46:33Z

Biggest wins seem to be if you are on NFS; I guess they are a minor (non-vocal) community :)

LordGrimmauld · 2025-04-01T07:42:45Z

Why? because of latency/bandwidth limit of network? In that case i'd expect some improvements for my external USB based setups too...

fzakaria · 2025-04-03T16:20:37Z

Why? because of latency/bandwidth limit of network? In that case i'd expect some improvements for my external USB based setups too...

Yes latency; I wrote more in depth about it in https://arxiv.org/abs/2211.05118
tl;dr; you have n^2 stat calls to check if a library exists. Each stat call on NFS is a network call.
(Sometimes negative-cached which mitigates it but if not... 💥 )

winterqt added 2 commits December 20, 2022 18:42

glibc: load $ORIGIN/../etc/ld.so.cache when available

27c7c0a

build-support/setup-hooks/generate-ld-cache: init

ee79e4b

winterqt requested review from Ericson2314 and matthewbauer as code owners December 21, 2022 00:14

github-actions bot added the 6.topic: stdenv Standard environment label Dec 21, 2022

winterqt changed the title ~~glibc: load $ORIGIN/../etc/ld.so.cache when available~~ Speed up application startup by using ld.so.cache Dec 21, 2022

ofborg bot added the 10.rebuild-linux-stdenv This PR causes stdenv to rebuild on Linux and must target a staging branch. label Dec 21, 2022

ofborg bot requested review from Ma27 and edolstra December 21, 2022 02:18

raphaelr reviewed Dec 21, 2022

View reviewed changes

fzakaria reviewed Dec 21, 2022

View reviewed changes

pennae mentioned this pull request Dec 27, 2022

draft: per-dso .so resolution cache on glibc #207893

Closed

13 tasks

ncfavier reviewed Jan 2, 2023

View reviewed changes

infinisil added the 1.severity: significant Novel ideas, large API changes, notable refactorings, issues with RFC potential, etc. label Apr 19, 2023

SomeoneSerge mentioned this pull request May 9, 2023

singularity cannot use nvidia gpu. #230851

Closed

wegank added 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 2.status: merge conflict This PR has merge conflicts with the target branch labels Mar 19, 2024

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 20, 2024

wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 4, 2024

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Sep 17, 2024

wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jan 1, 2025

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 31, 2025

RossSmyth mentioned this pull request Sep 9, 2025

Where should DLLs be installed? #431820

Open

nixpkgs-ci bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Oct 1, 2025

github-project-automation bot added this to Stdenv Oct 1, 2025

ilsubyeega mentioned this pull request Jan 19, 2026

Address /nix/store recursive lookup/syscall overhead #481620

Open


		fixupOutputHooks+=('if [ -z "${dontGenerateLDCache-}" ]; then generateLDCache "$prefix"; fi')

		generateLDCache() {

	+ /* Check whether ORIGIN is something like "/gnu/store/…-foo/bin". */
	+ /* Check whether ORIGIN is something like "/nix/store/…-foo/bin". */

Uh oh!

Conversation

winterqt commented Dec 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes

Things done

Uh oh!

winterqt commented Dec 21, 2022

Uh oh!

fzakaria commented Dec 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fzakaria commented Dec 21, 2022

Uh oh!

winterqt commented Dec 21, 2022

Uh oh!

winterqt commented Dec 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fzakaria commented Dec 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mindavi commented Dec 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

raphaelr Dec 21, 2022

Choose a reason for hiding this comment

Uh oh!

fzakaria Dec 21, 2022

Choose a reason for hiding this comment

Uh oh!

pennae commented Dec 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

winterqt commented Dec 21, 2022

Uh oh!

fzakaria Dec 21, 2022

Choose a reason for hiding this comment

Uh oh!

pennae Dec 21, 2022

Choose a reason for hiding this comment

Uh oh!

nixos-discourse commented Jan 1, 2023

Uh oh!

tomberek commented Jan 2, 2023

Uh oh!

ncfavier Jan 2, 2023

Choose a reason for hiding this comment

Uh oh!

nixos-discourse commented Jan 3, 2023

Uh oh!

nixos-discourse commented Aug 7, 2023

Uh oh!

nyabinary commented Dec 19, 2023

Uh oh!

nixos-discourse commented Aug 22, 2024

Uh oh!

diniamo commented Sep 17, 2024

Uh oh!

fzakaria commented Sep 17, 2024

Uh oh!

LordGrimmauld commented Mar 28, 2025

Uh oh!

fzakaria commented Mar 31, 2025

Uh oh!

LordGrimmauld commented Apr 1, 2025

Uh oh!

fzakaria commented Apr 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

winterqt commented Dec 21, 2022 •

edited

Loading

fzakaria commented Dec 21, 2022 •

edited

Loading

winterqt commented Dec 21, 2022 •

edited

Loading

fzakaria commented Dec 21, 2022 •

edited

Loading

Mindavi commented Dec 21, 2022 •

edited

Loading

pennae commented Dec 21, 2022 •

edited

Loading