waf: fix cross-compilation by AndersonTorres · Pull Request #252874 · NixOS/nixpkgs

AndersonTorres · 2023-09-02T02:29:10Z

Description of changes

Things done

AndersonTorres · 2023-09-02T02:32:31Z

lopsided98 · 2023-09-02T02:49:40Z

This fixes evaluating pkgsCross.armv7l-hf-multiplatform.python3Packages.cairosvg. I'm currently checking whether it builds.

AndersonTorres · 2023-09-02T03:59:59Z

I am a bit afraid that Python3 should be propagated inside waf.hook.

lopsided98 · 2023-09-02T17:58:44Z

The root cause of this issue is that waf.hook breaks splicing. This fix makes things evaluate, but it almost certainly doesn't build correctly. stdenv.hostPlatform.system != stdenv.targetPlatform.system will be false and waf will be from the host platform.

Here is a diff of the waf hook derivations with this PR applied. broken.txt is what you get when you use waf.hook, while good.txt is what you get when you use the wafHook alias. Note that broken.txt doesn't include --cross-compile and references waf-armv7l-unknown-linux-gnueabihf-2.0.26 (built for the host platform).

Diff

diff --git a/good.txt b/broken.txt
index 44168667146f..33f1385f33c9 100644
--- a/good.txt
+++ b/broken.txt
@@ -1,5 +1,5 @@
 {
-  "/nix/store/86byz9qa5an1cy774aqbikn8cd7w4vq2-waf-setup-hook.drv": {
+  "/nix/store/d3l1w04pdk7jmy8f7bcfcrbvzlsqciv9-waf-setup-hook.drv": {
     "args": [
       "-e",
       "/nix/store/6xg259477c90a229xwmb53pdfkn6ig3g-default-builder.sh"
@@ -10,9 +10,9 @@
       "buildCommand": "mkdir -p $out/nix-support\ncp /nix/store/7hqndxkvykifrd4dbqvsi79xmxrpyxh0-setup-hook.sh $out/nix-support/setup-hook\nrecordPropagatedDependencies\nsubstituteAll /nix/store/7hqndxkvykifrd4dbqvsi79xmxrpyxh0-setup-hook.sh $out/nix-support/setup-hook\n",
       "buildInputs": "",
       "builder": "/nix/store/m36d29gn5gm9bk0g7fcln1v8171hvn95-bash-5.2-p15/bin/bash",
-      "cmakeFlags": "",
-      "configureFlags": "",
-      "crossFlags": "--cross-compile \"--cross-execute=/nix/store/lxrcjkl3angz4bm7ys1f2cywqlgj9ayk-qemu-8.0.4/bin/qemu-arm\"",
+      "cmakeFlags": "-DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=armv7l -DCMAKE_HOST_SYSTEM_NAME=Linux -DCMAKE_HOST_SYSTEM_PROCESSOR=x86_64",
+      "configureFlags": "--build=x86_64-unknown-linux-gnu --host=armv7l-unknown-linux-gnueabihf",
+      "crossFlags": "",
       "depsBuildBuild": "",
       "depsBuildBuildPropagated": "",
       "depsBuildTarget": "",
@@ -26,42 +26,40 @@
       "enableParallelBuilding": "1",
       "enableParallelChecking": "1",
       "enableParallelInstalling": "1",
-      "mesonFlags": "",
+      "mesonFlags": "--cross-file=/nix/store/liv5rkz44z1cy77w4a92fg2ibnf852ng-cross-file.conf",
       "name": "waf-setup-hook",
       "nativeBuildInputs": "",
-      "out": "/nix/store/zsfa3fvqinzllv8c1jiyjfjrlabdpzgi-waf-setup-hook",
+      "out": "/nix/store/ra8x0v1s6197y2k03wwggh11hmjhnzd4-waf-setup-hook",
       "outputs": "out",
       "passAsFile": "buildCommand",
       "patches": "",
       "propagatedBuildInputs": "",
       "propagatedNativeBuildInputs": "",
-      "stdenv": "/nix/store/x6rqmyq0ni35krzp3fs44k36dfin1zhn-stdenv-linux",
+      "stdenv": "/nix/store/ckraqx6fiwami5cg96gn5j266rdzqr7q-stdenv-linux",
       "strictDeps": "1",
       "system": "x86_64-linux",
-      "waf": "/nix/store/syn1dhm73zd4d0jng4krqc17akwsmvb6-waf-2.0.26"
+      "waf": "/nix/store/h5bv4b0a65byh2lfinwndmg0jxc4013k-waf-armv7l-unknown-linux-gnueabihf-2.0.26"
     },
     "inputDrvs": {
       "/nix/store/0sawbbskl41aq4ii9kl5w77r4m92das8-bash-5.2-p15.drv": [
         "out"
       ],
-      "/nix/store/1qxgnnxb8ggpgq84jhmgl8351a8v0h03-qemu-8.0.4.drv": [
+      "/nix/store/j0392zmavpa52y6yr32v6wi1v6n7grbc-waf-armv7l-unknown-linux-gnueabihf-2.0.26.drv": [
         "out"
       ],
-      "/nix/store/76krn5snp1b82i3d95sz11fpnqlkwrl3-waf-2.0.26.drv": [
-        "out"
-      ],
-      "/nix/store/ks92ys95hq4c5aq5rqncl4qn9nak7haf-stdenv-linux.drv": [
+      "/nix/store/kpmcs593zvgcdsblr6fb1v27nhijnpzp-stdenv-linux.drv": [
         "out"
       ]
     },
     "inputSrcs": [
       "/nix/store/6xg259477c90a229xwmb53pdfkn6ig3g-default-builder.sh",
-      "/nix/store/7hqndxkvykifrd4dbqvsi79xmxrpyxh0-setup-hook.sh"
+      "/nix/store/7hqndxkvykifrd4dbqvsi79xmxrpyxh0-setup-hook.sh",
+      "/nix/store/liv5rkz44z1cy77w4a92fg2ibnf852ng-cross-file.conf"
     ],
     "name": "waf-setup-hook",
     "outputs": {
       "out": {
-        "path": "/nix/store/zsfa3fvqinzllv8c1jiyjfjrlabdpzgi-waf-setup-hook"
+        "path": "/nix/store/ra8x0v1s6197y2k03wwggh11hmjhnzd4-waf-setup-hook"
       }
     },
     "system": "x86_64-linux"

You could work around these issues by adjusting the platform comparison and using more buildPackages, but this wouldn't solve the root splicing problem.

AndersonTorres · 2023-09-03T13:59:02Z

@lopsided98 can you test it now?

lopsided98 · 2023-09-05T01:22:32Z

jack2 pulls in a bunch of dependencies that are broken for other reasons, libjack2 is the minimal version that should work

@ofborg build pkgsCross.armv7l-hf-multiplatform.libjack2

AndersonTorres · 2023-09-05T15:25:06Z

Timeout failure :(

As shown in a79a8f2, tha variable can be overriden by other functions, especially those related to cross or static compilation.

Standardizing the behaviour of commit a79a8f2.

- Reword and reorganize sections

AndersonTorres · 2023-09-06T18:40:00Z

@ofborg build pkgsCross.armv7l-hf-multiplatform.libjack2

AndersonTorres · 2023-09-07T11:32:43Z

Yay, it worked!

Artturin · 2023-09-07T18:25:26Z

@ofborg build pkgsCross.aarch64-multiplatform.tests.cc-wrapper

pkgsCross.aarch64-multiplatform.tests.cc-wrapper has a infinite recursion currently, hopefully this PR fixes it

$ nix build ".#pkgsCross.aarch64-multiplatform.tests.cc-wrapper" --show-trace
error:
       … while calling the 'derivationStrict' builtin

         at <nix/derivation-internal.nix>:9:12:

            8|
            9|   strict = derivationStrict drvAttrs;
             |            ^
           10|

       … while evaluating derivation 'cc-wrapper-test-aarch64-unknown-linux-gnu'
         whose name attribute is located at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:300:7

       … while evaluating attribute 'buildCommand' of derivation 'cc-wrapper-test-aarch64-unknown-linux-gnu'

         at /home/artturin/nixgits/my-nixpkgs/pkgs/test/cc-wrapper/default.nix:15:3:

           14|
           15|   buildCommand = ''
             |   ^
           16|     set -o pipefail

       … from call site

         at /home/artturin/nixgits/my-nixpkgs/pkgs/test/cc-wrapper/default.nix:11:14:

           10|   staticLibc = lib.optionalString (stdenv.hostPlatform.libc == "glibc") "-L ${glibc.static}/lib";
           11|   emulator = stdenv.hostPlatform.emulator buildPackages;
             |              ^
           12| in stdenv.mkDerivation {

       … while calling 'emulator'

         at /home/artturin/nixgits/my-nixpkgs/lib/systems/default.nix:248:20:

          247|
          248|         emulator = pkgs:
             |                    ^
          249|           if (final.emulatorAvailable pkgs)

       … from call site

         at /home/artturin/nixgits/my-nixpkgs/lib/systems/default.nix:249:15:

          248|         emulator = pkgs:
          249|           if (final.emulatorAvailable pkgs)
             |               ^
          250|           then selectEmulator pkgs

       … while calling 'emulatorAvailable'

         at /home/artturin/nixgits/my-nixpkgs/lib/systems/default.nix:246:29:

          245|       in {
          246|         emulatorAvailable = pkgs: (selectEmulator pkgs) != null;
             |                             ^
          247|

       … from call site

         at /home/artturin/nixgits/my-nixpkgs/lib/systems/default.nix:246:36:

          245|       in {
          246|         emulatorAvailable = pkgs: (selectEmulator pkgs) != null;
             |                                    ^
          247|

       … while calling 'selectEmulator'

         at /home/artturin/nixgits/my-nixpkgs/lib/systems/default.nix:216:26:

          215|       let
          216|         selectEmulator = pkgs:
             |                          ^
          217|           let

       … while evaluating derivation 'qemu-8.1.0'
         whose name attribute is located at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:300:7

       … while evaluating attribute 'buildInputs' of derivation 'qemu-8.1.0'

         at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:347:7:

          346|       depsHostHost                = lib.elemAt (lib.elemAt dependencies 1) 0;
          347|       buildInputs                 = lib.elemAt (lib.elemAt dependencies 1) 1;
             |       ^
          348|       depsTargetTarget            = lib.elemAt (lib.elemAt dependencies 2) 0;

       … while evaluating derivation 'pipewire-0.3.79'
         whose name attribute is located at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:300:7

       … while evaluating attribute 'buildInputs' of derivation 'pipewire-0.3.79'

         at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:347:7:

          346|       depsHostHost                = lib.elemAt (lib.elemAt dependencies 1) 0;
          347|       buildInputs                 = lib.elemAt (lib.elemAt dependencies 1) 1;
             |       ^
          348|       depsTargetTarget            = lib.elemAt (lib.elemAt dependencies 2) 0;

       … while evaluating derivation 'libjack2-1.9.22'
         whose name attribute is located at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:300:7

       … while evaluating attribute 'nativeBuildInputs' of derivation 'libjack2-1.9.22'

         at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:344:7:

          343|       depsBuildBuild              = lib.elemAt (lib.elemAt dependencies 0) 0;
          344|       nativeBuildInputs           = lib.elemAt (lib.elemAt dependencies 0) 1;
             |       ^
          345|       depsBuildTarget             = lib.elemAt (lib.elemAt dependencies 0) 2;

       … while evaluating derivation 'waf-setup-hook'
         whose name attribute is located at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:300:7

       … while evaluating attribute 'crossFlags' of derivation 'waf-setup-hook'

         at /home/artturin/nixgits/my-nixpkgs/pkgs/development/tools/build-managers/waf/hook.nix:16:5:

           15|     inherit waf;
           16|     crossFlags = lib.optionalString (stdenv.hostPlatform.system != stdenv.targetPlatform.system)
             |     ^
           17|       ''--cross-compile "--cross-execute=${stdenv.targetPlatform.emulator pkgs}"'';

       … from call site

         at /home/artturin/nixgits/my-nixpkgs/pkgs/development/tools/build-managers/waf/hook.nix:16:18:

           15|     inherit waf;
           16|     crossFlags = lib.optionalString (stdenv.hostPlatform.system != stdenv.targetPlatform.system)
             |                  ^
           17|       ''--cross-compile "--cross-execute=${stdenv.targetPlatform.emulator pkgs}"'';

       … while calling 'optionalString'

         at /home/artturin/nixgits/my-nixpkgs/lib/strings.nix:243:5:

          242|     # String to return if condition is true
          243|     string: if cond then string else "";
             |     ^
          244|

       … from call site

         at /home/artturin/nixgits/my-nixpkgs/pkgs/development/tools/build-managers/waf/hook.nix:17:44:

           16|     crossFlags = lib.optionalString (stdenv.hostPlatform.system != stdenv.targetPlatform.system)
           17|       ''--cross-compile "--cross-execute=${stdenv.targetPlatform.emulator pkgs}"'';
             |                                            ^
           18|   };

       … while calling 'emulator'

         at /home/artturin/nixgits/my-nixpkgs/lib/systems/default.nix:248:20:

          247|
          248|         emulator = pkgs:
             |                    ^
          249|           if (final.emulatorAvailable pkgs)

       … from call site

         at /home/artturin/nixgits/my-nixpkgs/lib/systems/default.nix:249:15:

          248|         emulator = pkgs:
          249|           if (final.emulatorAvailable pkgs)
             |               ^
          250|           then selectEmulator pkgs

       … while calling 'emulatorAvailable'

         at /home/artturin/nixgits/my-nixpkgs/lib/systems/default.nix:246:29:

          245|       in {
          246|         emulatorAvailable = pkgs: (selectEmulator pkgs) != null;
             |                             ^
          247|

       … from call site

         at /home/artturin/nixgits/my-nixpkgs/lib/systems/default.nix:246:36:

          245|       in {
          246|         emulatorAvailable = pkgs: (selectEmulator pkgs) != null;
             |                                    ^
          247|

       … while calling 'selectEmulator'

         at /home/artturin/nixgits/my-nixpkgs/lib/systems/default.nix:216:26:

          215|       let
          216|         selectEmulator = pkgs:
             |                          ^
          217|           let

       … while evaluating derivation 'qemu-8.1.0'
         whose name attribute is located at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:300:7

       … while evaluating attribute 'buildInputs' of derivation 'qemu-8.1.0'

         at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:347:7:

          346|       depsHostHost                = lib.elemAt (lib.elemAt dependencies 1) 0;
          347|       buildInputs                 = lib.elemAt (lib.elemAt dependencies 1) 1;
             |       ^
          348|       depsTargetTarget            = lib.elemAt (lib.elemAt dependencies 2) 0;

       error: infinite recursion encountered

       at «none»:0: (source not available)

Artturin · 2023-09-07T18:34:56Z

Ofborg shows that it fails, but I tested and rebased to staging locally and it worked.

Mindavi · 2023-09-07T20:45:42Z

Actually, I'm trying this locally but still seeing infinite recursion issues.

nix build .#pkgsCross.aarch64-multiplatform.ffmpeg -L

~/nixpkgs$ nix build .#pkgsCross.aarch64-multiplatform.ffmpeg -L; echo $?
error:
       … while calling the 'derivationStrict' builtin

         at /builtin/derivation.nix:9:12: (source not available)

       … while evaluating derivation 'ffmpeg-aarch64-unknown-linux-gnu-5.1.3'
         whose name attribute is located at /nix/store/irskqk7rkl275n7v9ga3278qrlxafsk2-source/pkgs/stdenv/generic/make-derivation.nix:300:7

       … while evaluating attribute 'buildInputs' of derivation 'ffmpeg-aarch64-unknown-linux-gnu-5.1.3'

         at /nix/store/irskqk7rkl275n7v9ga3278qrlxafsk2-source/pkgs/stdenv/generic/make-derivation.nix:347:7:

          346|       depsHostHost                = lib.elemAt (lib.elemAt dependencies 1) 0;
          347|       buildInputs                 = lib.elemAt (lib.elemAt dependencies 1) 1;
             |       ^
          348|       depsTargetTarget            = lib.elemAt (lib.elemAt dependencies 2) 0;

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: infinite recursion encountered

       at «none»:0: (source not available)

Artturin · 2023-09-07T21:17:13Z

@Mindavi

no mention of waf in the strace so that should be a separate issue

i rebased this branch on to 872e54e (staging on sep6) and it worked but now rebasing on to 6f51712 it fails

Artturin · 2023-09-07T21:20:24Z

likely caused by 5b0ed68

Artturin · 2023-09-07T21:26:32Z

#253913

This reverts commit 9f52275. Caused breakage on staging-next (2023-09-07) due to the following error: ``` waf: error: no such option: --sbindir ``` Reverted on staging after the cut, refer to NixOS#252874

...into staging-next This is a topologically earlier remerge of PR #252874

kjeremy · 2023-09-25T13:45:31Z

https://nixpk.gs/pr-tracker.html?pr=252874 Shows that this still isn't in master or nixos-unstable which is blocking me. Is the PR tracker right?

AndersonTorres · 2023-09-25T16:49:53Z

Yes, it is.

https://github.com/NixOS/nixpkgs/pull/255658/commits

ofborg bot added the 6.topic: cross-compilation Building packages on a different platform than they will be used on label Sep 2, 2023

lopsided98 mentioned this pull request Sep 2, 2023

Waf: refactor #249251

Merged

12 tasks

github-actions bot added the 6.topic: xfce The Xfce Desktop Environment label Sep 3, 2023

ofborg bot removed 10.rebuild-linux: 1 This PR causes 1 package to rebuild on Linux. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Sep 3, 2023

github-actions bot added the 8.has: documentation This PR adds or changes documentation label Sep 3, 2023

AndersonTorres marked this pull request as ready for review September 3, 2023 19:24

AndersonTorres added 6 commits September 6, 2023 10:41

wafHook: resurrect alias

1abf09f

treewide: waf.hook -> wafHook

f2f9262

waf: use wafConfigureFlags instead of configureFlags in setup-hook.sh

cad7cea

As shown in a79a8f2, tha variable can be overriden by other functions, especially those related to cross or static compilation.

treewide: convert configureFlags to wafConfigureFlags when appropriate

e24107b

waf: prefix all setup-hook variables with waf

b3a1dd8

Standardizing the behaviour of commit a79a8f2.

doc/hooks/waf.section.md: update

83b98f9

- Reword and reorganize sections

Artturin merged commit 6946977 into NixOS:staging Sep 7, 2023

AndersonTorres mentioned this pull request Sep 8, 2023

staging-next 2023-09-07 #253854

Merged

fabianhjr mentioned this pull request Sep 8, 2023

Revert "treewide: change wafConfigureFlags to configureFlags" #254111

Closed

12 tasks

vcunat added a commit that referenced this pull request Sep 9, 2023

Merge #254182: merge waf changes

479f1af

...into staging-next This is a topologically earlier remerge of PR #252874

ashkitten mentioned this pull request Sep 11, 2023

Fix cross-compilation of firefox for powerpc64le-unknown-linux-musl #253225

Closed

12 tasks

nikstur mentioned this pull request Sep 14, 2023

systemd: 253.5 -> 254.3 #243242

Merged

12 tasks

Uh oh!

Conversation

AndersonTorres commented Sep 2, 2023

Description of changes

Things done

Uh oh!

AndersonTorres commented Sep 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lopsided98 commented Sep 2, 2023

Uh oh!

AndersonTorres commented Sep 2, 2023

Uh oh!

lopsided98 commented Sep 2, 2023

Uh oh!

AndersonTorres commented Sep 3, 2023

Uh oh!

lopsided98 commented Sep 5, 2023

Uh oh!

AndersonTorres commented Sep 5, 2023

Uh oh!

AndersonTorres commented Sep 6, 2023

Uh oh!

AndersonTorres commented Sep 7, 2023

Uh oh!

Artturin commented Sep 7, 2023

Uh oh!

Artturin commented Sep 7, 2023

Uh oh!

Mindavi commented Sep 7, 2023

Uh oh!

Artturin commented Sep 7, 2023

Uh oh!

Artturin commented Sep 7, 2023

Uh oh!

Artturin commented Sep 7, 2023

Uh oh!

kjeremy commented Sep 25, 2023

Uh oh!

AndersonTorres commented Sep 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

AndersonTorres commented Sep 2, 2023 •

edited

Loading

AndersonTorres commented Sep 25, 2023 •

edited

Loading