Skip to content

treewide: Allow the linker to be chosen independently#122778

Merged
Ericson2314 merged 2 commits intoNixOS:stagingfrom
Ericson2314:choose-linker
May 14, 2021
Merged

treewide: Allow the linker to be chosen independently#122778
Ericson2314 merged 2 commits intoNixOS:stagingfrom
Ericson2314:choose-linker

Conversation

@Ericson2314
Copy link
Copy Markdown
Member

Motivation for this change

This will begin the process of breaking up the useLLVM monolith. That
is good in general, but I hope will be good for NetBSD and Darwin in
particular.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@github-actions github-actions bot added the 6.topic: stdenv Standard environment label May 12, 2021
@matthewbauer matthewbauer changed the title treewide: All the linker to be chosen independently treewide: Allow the linker to be chosen independently May 12, 2021
@ofborg ofborg bot added 8.has: clean-up This PR removes packages or removes other cruft 8.has: package (new) This PR adds a new package labels May 12, 2021
@ofborg ofborg bot added 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. labels May 12, 2021
@r-rmcgibbo
Copy link
Copy Markdown

Result of nixpkgs-review pr 122778 at 168194cc run on x86_64-linux 1

38 packages skipped due to time constraints:
  • adoptopenjdk-icedtea-web
  • chromium
  • clickhouse
  • firefox-esr-78-unwrapped (firefoxPackages.firefox-esr-78)
  • xulrunner (firefox-unwrapped)
  • llvmPackages_9.clangNoCompilerRt (haskellPackages.llvmPackages.clangNoCompilerRt)
  • llvmPackages_9.clangNoCompilerRtWithLibc (haskellPackages.llvmPackages.clangNoCompilerRtWithLibc)
  • llvmPackages_9.clangNoLibc (haskellPackages.llvmPackages.clangNoLibc)
  • llvmPackages_9.clangNoLibcxx (haskellPackages.llvmPackages.clangNoLibcxx)
  • llvmPackages_9.clangUseLLVM (haskellPackages.llvmPackages.clangUseLLVM)
  • ...
3 packages built successfully:
  • bintools
  • bintools-unwrapped
  • bintoolsNoLibc

@mweinelt
Copy link
Copy Markdown
Member

This breaks the firefox build on x86_64-linux.

 0:09.91(B checking for linker...(B
 0:09.92(B DEBUG: Executing: `/nix/store/l9fklk5dj4b4gzpc28d16ckfq5y1i1xk-clang-wrapper-11.1.0/bin/clang -std=gnu99 -fuse-ld=lld -Wl,--version`(B
 0:09.92(B ERROR: Could not use lld as linker(B
Error running mach:

    ['configure', '--prefix=/nix/store/dr9fplxh68mc0314d1x8fvrz1ggvkvln-firefox-unwrapped-88.0.1', '--enable-application=browser', '--with-system-jpeg', '--with-system-zlib', '--with-system-libevent', '--with-system-libvpx', '--with-system-png', '--with-system-icu', '--enable-system-ffi', '--enable-system-pixman', '--disable-tests', '--disable-necko-wifi', '--disable-updater', '--enable-jemalloc', '--enable-default-toolkit=cairo-gtk3-wayland', '--with-libclang-path=/nix/store/jlzl1c2w1d884ykzlmpby102y1rdzzw3-clang-11.1.0-lib/lib', '--with-system-nspr', '--with-system-nss', '--enable-lto', '--disable-elf-hack', '--enable-linker=lld', '--enable-alsa', '--enable-pulseaudio', '--enable-ffmpeg', '--enable-negotiateauth', '--enable-webrtc', '--disable-crashreporter', '--disable-debug', '--enable-optimize', '--enable-strip', '--enable-release', '--enable-official-branding', '--with-google-location-service-api-keyfile=/build/ga', '--with-google-safebrowsing-api-keyfile=/build/ga']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file configure| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Process executed with non-0 exit code 1: ['/nix/store/pxhyfgz2c7y5n35hjphdi0ady7xdl4sm-python3-3.8.9/bin/python3', '/build/firefox-88.0.1/configure.py', '--prefix=/nix/store/dr9fplxh68mc0314d1x8fvrz1ggvkvln-firefox-unwrapped-88.0.1', '--enable-application=browser', '--with-system-jpeg', '--with-system-zlib', '--with-system-libevent', '--with-system-libvpx', '--with-system-png', '--with-system-icu', '--enable-system-ffi', '--enable-system-pixman', '--disable-tests', '--disable-necko-wifi', '--disable-updater', '--enable-jemalloc', '--enable-default-toolkit=cairo-gtk3-wayland', '--with-libclang-path=/nix/store/jlzl1c2w1d884ykzlmpby102y1rdzzw3-clang-11.1.0-lib/lib', '--with-system-nspr', '--with-system-nss', '--enable-lto', '--disable-elf-hack', '--enable-linker=lld', '--enable-alsa', '--enable-pulseaudio', '--enable-ffmpeg', '--enable-negotiateauth', '--enable-webrtc', '--disable-crashreporter', '--disable-debug', '--enable-optimize', '--enable-strip', '--enable-release', '--enable-official-branding', '--with-google-location-service-api-keyfile=/build/ga', '--with-google-safebrowsing-api-keyfile=/build/ga']

  File "/build/firefox-88.0.1/python/mozbuild/mozbuild/build_commands.py", line 177, in configure
    return driver.configure(
  File "/build/firefox-88.0.1/python/mozbuild/mozbuild/controller/building.py", line 1529, in configure
    status = self._run_command_in_objdir(
  File "/build/firefox-88.0.1/python/mozbuild/mozbuild/base.py", line 897, in _run_command_in_objdir
    return self.run_process(cwd=self.topobjdir, **args)
  File "/build/firefox-88.0.1/python/mach/mach/mixin/process.py", line 176, in run_process
    raise Exception(
builder for '/nix/store/lq1xykqwb7nz6n79ax480kwshggw82lh-firefox-unwrapped-88.0.1.drv' failed with exit code 1

@Ericson2314
Copy link
Copy Markdown
Member Author

Yeah i found I think workarounds for everything but Firefox. Hmmm.

@Ericson2314
Copy link
Copy Markdown
Member Author

Ericson2314 commented May 13, 2021

Firefox now has a (somewhat ugly) workaround. I don't really like the override, but writing down every possible combination is also a bit much. That's when I wish we had a relational language / solver :D.

@Ericson2314
Copy link
Copy Markdown
Member Author

@matthewbauer It's been a minute, but I'd love for you to way on this. I first mentioned doing something like this quite some time ago :).

Add a comment with explanation, which I should have done all along.
@Ericson2314
Copy link
Copy Markdown
Member Author

OK the fix for firefox is a lot less ugly now.

Copy link
Copy Markdown
Member

@lovesegfault lovesegfault left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left two small nits, but the changes look good overall to me. I suspect it may be worth it to do a separate hydra run of this just to be sure, even if this is going into staging and not master. Up to the author.

Copy link
Copy Markdown
Member

@sternenseemann sternenseemann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Not sure if bootBintools is the clearest terminology in this case, but OTOH I am also unable to express what I think “boot” is supposed to mean in that context.

This will begin the process of breaking up the `useLLVM` monolith. That
is good in general, but I hope will be good for NetBSD and Darwin in
particular.

Co-authored-by: sterni <sternenseemann@systemli.org>
@ofborg ofborg bot requested a review from lovesegfault May 14, 2021 21:42
@Ericson2314
Copy link
Copy Markdown
Member Author

@sternenseemann Yeah I don't really like the name either, but I don't intend for it to be really stable so we can just change it later. I somewhat hope as the bootstrapping logic becomes more standardized across all the combinations, we can move it out of the default.nix anyways.

@Ericson2314 Ericson2314 merged commit 7dd53fe into NixOS:staging May 14, 2021
@Ericson2314 Ericson2314 deleted the choose-linker branch May 14, 2021 22:53
sternenseemann added a commit to sternenseemann/nixpkgs that referenced this pull request Aug 3, 2021
PR NixOS#122778 allowed the linker being chosen independently from useLLVM
which also affected pkgsLLVM where we were relying on this behavior.
For platform sets assembled from scratch useLLVM still implies
linker == "lld", however in the case of pkgsLLVM we update the current
platform via the set update operator which means that `linker` won't
be re-evaluated. Using ld.bfd with pkgsLLVM is okay to a certain extent,
but with C++ things begin to break.

We fix this by setting linker explicitly.
sternenseemann added a commit that referenced this pull request Aug 3, 2021
PR #122778 allowed the linker being chosen independently from useLLVM
which also affected pkgsLLVM where we were relying on this behavior.
For platform sets assembled from scratch useLLVM still implies
linker == "lld", however in the case of pkgsLLVM we update the current
platform via the set update operator which means that `linker` won't
be re-evaluated. Using ld.bfd with pkgsLLVM is okay to a certain extent,
but with C++ things begin to break.

We fix this by setting linker explicitly.

stdenvNoLibs = mkStdenvNoLibs stdenv;
stdenvNoLibs =
if stdenv.hostPlatform != stdenv.buildPlatform && (stdenv.hostPlatform.isDarwin || stdenv.hostPlatform.isDarwin.useLLVM or false)
Copy link
Copy Markdown
Member

@Artturin Artturin Feb 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if stdenv.hostPlatform != stdenv.buildPlatform && (stdenv.hostPlatform.isDarwin || stdenv.hostPlatform.isDarwin.useLLVM or false)

should be

if stdenv.hostPlatform != stdenv.buildPlatform && (stdenv.hostPlatform.isDarwin || stdenv.hostPlatform.useLLVM or false)

nix-repl> pkgsCross.x86_64-freebsd.stdenv.hostPlatform.useLLVM
true

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops! Thanks for catching!

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Artturin added a commit to Artturin/nixpkgs that referenced this pull request Dec 13, 2023
@Artturin Artturin mentioned this pull request Dec 13, 2023
13 tasks
github-actions bot pushed a commit that referenced this pull request Dec 14, 2023
@rrbutani rrbutani added the 6.topic: llvm/clang Issues related to llvmPackages, clangStdenv and related label May 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: llvm/clang Issues related to llvmPackages, clangStdenv and related 6.topic: stdenv Standard environment 8.has: clean-up This PR removes packages or removes other cruft 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants