qt6: 6.3.0 -> 6.3.1 by milahu · Pull Request #178171 · NixOS/nixpkgs

milahu · 2022-06-18T18:02:19Z

Description of changes

use latest patch release of qt

https://www.qt.io/blog/qt-6.3.1-released

15919c5bda QBuffer: fix writing more than two GiB of data
Fixed silent data truncation when writing more than two GiB at once on
64-bit platforms.

4f45c0c5d8 QTextStream: fix streaming of char16_t's
Added op<<(char16_t).

c00b8652b5 Fix int/qsizetype mismatches in qstring.h
Fixed result truncation mod INT_MAX in fromStdSstring(),
fromStdU16string(), fromStdU32string(), and fromStdWstring().

...

upstream issues

patches for jobclient support were sent to ...

qtwebengine https://bugreports.qt.io/browse/QTBUG-95176
- probably qt will delegate to chromium
chromium https://groups.google.com/a/chromium.org/g/chromium-discuss (marked as spam?)
gn https://groups.google.com/a/chromium.org/g/gn-dev/c/qKtD0TDl00c (marked as spam?)
devtools-frontend https://groups.google.com/a/chromium.org/g/devtools-reviews/c/TJchABRWq1g (marked as spam?)
jest-worker feat(jest-worker): add gnumake jobclient to limit number of workers jestjs/jest#12968

Child packages

tdesktop 3.7.3 throws build error → update to pre-release 3.7.6
tdesktop 3.7.6 throws build error → update tdesktop.tg_owt (upstream issue)

Review

Details

$ nixpkgs-review pr 178171

41 packages updated:
cutemaze fcitx5-chinese-addons fcitx5-configtool fcitx5-qt fcitx5-qt fcitx5-qt fcitx5-unikey fcitx5-with-addons qt6.qt3d (6.3.0 → 6.3.1) qt6.qt5compat (6.3.0 → 6.3.1) qt6.qtbase (6.3.0 → 6.3.1) qt6.qtcharts (6.3.0 → 6.3.1) qt6.qtconnectivity (6.3.0 → 6.3.1) qt6.qtdatavis3d (6.3.0 → 6.3.1) qt6.qtdeclarative (6.3.0 → 6.3.1) qt6.qtdoc (6.3.0 → 6.3.1) qt6.qtimageformats (6.3.0 → 6.3.1) qt6.qtlanguageserver (6.3.0 → 6.3.1) qt6.qtlottie (6.3.0 → 6.3.1) qt6.qtmultimedia (6.3.0 → 6.3.1) qt6.qtnetworkauth (6.3.0 → 6.3.1) qt6.qtpositioning (6.3.0 → 6.3.1) qt6.qtquick3d (6.3.0 → 6.3.1) qt6.qtquicktimeline (6.3.0 → 6.3.1) qt6.qtremoteobjects (6.3.0 → 6.3.1) qt6.qtscxml (6.3.0 → 6.3.1) qt6.qtsensors (6.3.0 → 6.3.1) qt6.qtserialbus (6.3.0 → 6.3.1) qt6.qtserialport (6.3.0 → 6.3.1) qt6.qtshadertools (6.3.0 → 6.3.1) qt6.qtsvg (6.3.0 → 6.3.1) qt6.qttools (6.3.0 → 6.3.1) qt6.qttranslations (6.3.0 → 6.3.1) qt6.qtvirtualkeyboard (6.3.0 → 6.3.1) qt6.qtwayland (6.3.0 → 6.3.1) qt6.qtwebchannel (6.3.0 → 6.3.1) qt6.qtwebengine (6.3.0 → 6.3.1) qt6.qtwebsockets (6.3.0 → 6.3.1) qt6.qtwebview (6.3.0 → 6.3.1) quazip tdesktop (3.7.3 → 3.7.6)

2 packages marked as broken and skipped:
libsForQt512.fcitx5-qt libsForQt514.fcitx5-qt

2 packages failed to build:
qt6.qtwebengine qt6.qtwebview

37 packages built:
cutemaze fcitx5-chinese-addons fcitx5-configtool fcitx5-unikey fcitx5-with-addons libsForQt5.fcitx5-qt qt6.qt3d qt6.qt5compat qt6.qtbase qt6.qtcharts qt6.qtconnectivity qt6.qtdatavis3d qt6.qtdeclarative qt6.qtdoc qt6.qtimageformats qt6.qtlanguageserver qt6.qtlottie qt6.qtmultimedia qt6.qtnetworkauth qt6.qtpositioning qt6.qtquick3d qt6.qtquicktimeline qt6.qtremoteobjects qt6.qtscxml qt6.qtsensors qt6.qtserialbus qt6.qtserialport qt6.qtshadertools qt6.qtsvg qt6.qttools qt6.qttranslations qt6.qtvirtualkeyboard qt6.qtwayland qt6.qtwebchannel qt6.qtwebsockets qt6Packages.quazip tdesktop

Things done

pkgs/development/libraries/qt-6/modules/qtwebengine.nix

pkgs/applications/networking/instant-messengers/telegram/tdesktop/tg_owt.nix

milahu · 2022-06-19T14:55:04Z

qtwebengine build error: internal compiler error: Segmentation fault

thrown from random locations ...

Details

[25974/29280] CXX obj/third_party/blink/renderer/core/core/core_jumbo_207.o
FAILED: obj/third_party/blink/renderer/core/core/core_jumbo_207.o 

/nix/store/j5vhrywqmz1ixwhsmmjjxa85fpwryzh0-gcc-11.3.0/include/c++/11.3.0/tuple:1069:36: internal compiler error: Segmentation fault
 1069 |         : _Inherited(__a1, __a2) { }
      |                                    ^

or

[25885/29280] CXX obj/third_party/blink/renderer/core/core/core_jumbo_262.o
FAILED: obj/third_party/blink/renderer/core/core/core_jumbo_262.o 

../../../../../src/3rdparty/chromium/third_party/blink/renderer/core/timing/performance.h:318:8: internal compiler error: Segmentation fault
  318 |   void Trace(Visitor*) const override;
      |        ^~~~~

qtwebengine-6.3.1.error.log.gz

similar to QTBUG-103573: GCC internal compiler error: Segmentation fault when trying to compile Qt 6.3

simlar to gentoo bug 663770: dev-qt/qtwebengine-5.11.1 fails to build with insufficient memory

caused by setting NINJAFLAGS, even with high values like NINJAFLAGS="-j128 -l128"

→ reverting "fix ninja flags"

milahu · 2022-06-20T18:23:06Z

review: qt6 compiles + telegram works

NickCao · 2022-06-22T08:38:02Z

Would the ninja fix solve the problems we are having building qtwebengine on hydra?
Oh, it's commented out now, never mind.

SuperSandro2000 · 2022-06-22T12:50:19Z

pkgs/development/libraries/qt-6/modules/qtwebengine.nix

webengine tends to hog even really good build servers. We shouldn't merge this like this.

im working on this in my branch qt6-631-debug-qtwebengine

the main build process is ninja
so currently im exploring ninja-tokenpool (implements jobclient + jobserver)
to limit the job count across multiple ninja processes

todo: add jobclients to python and node builders in qtwebengine

src/3rdparty/chromium/third_party/blink/renderer/bindings/scripts/bind_gen/task_queue.py

src/3rdparty/chromium/third_party/devtools-frontend/src/node_modules/rollup-plugin-terser/rollup-plugin-terser.js

by default, these builders use all cpu cores

I think building is better than being well behaved. This build break blocks quite a few packages. qtwebengine is expensive to build but it isn't that bad.

status update:
i have added jobclient to

src/3rdparty/chromium/third_party/devtools-frontend/src/node_modules/rollup-plugin-terser/rollup-plugin-terser.js src/3rdparty/chromium/third_party/blink/renderer/bindings/scripts/bind_gen/task_queue.py src/3rdparty/chromium/mojo/public/tools/mojom/mojom_parser.py src/3rdparty/gn/src/util/worker_pool.cc

this is mostly working ...
in rare cases, the workers dont release their tokens back to the jobserver
which results in underloading, for example load is 20 when it should be 32

i hope to fix/workaround this in the next ~2 days
but feel free to merge this PR as is

in rare cases, the workers dont release their tokens back to the jobserver

What is the chance that the number of tokens drops to zero or a very low number? It would be annoying if that could hang a build.

here, the problem are bugs in my jobclients
the load slowly drops to 1 = no parallelism (but the build does not hang)

but its also a well-known weakness/bug in the jobserver protocol
milahu/gnumake-tokenpool#1 (comment)

I guess that almost any time a job fails the whole build is going to fail anyways so it shouldn't be too big of a problem. In that I case I think it would be fine with a warning. But if you think you can fix it that is even better.

fixed in milahu/gnumake-tokenpool@247c42b
in javascript, fs.readSync can block when the pipe is empty

now the max load is 32 == NIX_BUILD_CORES
some outliers around T=500, probably cmake, but meh

@kevincox

I tried building e206f87 and it has stalled out as well

same here : (

with qt6-631-debug-qtwebengine-2 i see that the javascript workers are blocking ...

load Load rss time spr cpr proc 20.0 18.1 9869 471 123 2 bash 1: bash -e /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh # {'cwd': '/build/qtwebengine-everywhere-src-6.3.1/build', 'exe': '/nix/store/j9rc786ylq8cid4zgcn6idknilkbd4ax-bash-5.1-p16/bin/bash'} 20.0 18.1 9863 425 121 1 ninja 13856: ninja -j32 -l32 --tokenpool-master # {'exe': '/nix/store/k38101zidsncnv0qvcia82n1nmvkb57a-ninja-tokenpool-unstable-2022-02-13/bin/ninja'} 20.0 18.1 9858 378 120 40 ninja 17319: ninja -C /build/qtwebengine-everywhere-src-6.3.1/build/src/core/Release/x86_64 QtWebEngineCore # {'cwd': '/build/qtwebengine-everywhere-src-6.3.1/build/src/core/Release/x86_64', 'exe': '/nix/store/k38101zidsncnv0qvcia82n1nmvkb57a-ninja-tokenpool-unstable-2022-02-13/bin/ninja'} 1.0 1.1 956 229 2 1 python3 31547: python3 ../../../../../src/3rdparty/chromium/third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle --root_src_dir ../../../../../src/3rdparty/chromium/ --root_gen_dir gen --output_reldir core=third_party/blink/renderer/bindings/core/v8/ --output_reldir modules=third_party/blink/renderer/bindings/modules/v8/ --format_generated_files enumeration callback_function callback_interface dictionary interface namespace typedef union # {'exe': '/nix/store/i6svmmq24wl4i10dx0m8blrpl3m3fq4b-python3-3.10.5/bin/python3.10'} 1.0 1.0 790 228 1 0 python3 31861: python3 ../../../../../src/3rdparty/chromium/third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle --root_src_dir ../../../../../src/3rdparty/chromium/ --root_gen_dir gen --output_reldir core=third_party/blink/renderer/bindings/core/v8/ --output_reldir modules=third_party/blink/renderer/bindings/modules/v8/ --format_generated_files enumeration callback_function callback_interface dictionary interface namespace typedef union # {'exe': '/nix/store/i6svmmq24wl4i10dx0m8blrpl3m3fq4b-python3-3.10.5/bin/python3.10'} 0.0 0.0 132 225 3 1 python3 32929: python3 ../../../../../src/3rdparty/chromium/third_party/node/node.py ../../../../../src/3rdparty/chromium/third_party/devtools-frontend/src/node_modules/rollup/dist/bin/rollup --silent --config ../../../../../src/3rdparty/chromium/third_party/devtools-frontend/src/scripts/build/rollup.config.js --input gen/third_party/devtools-frontend/src/front_end/entrypoints/formatter_worker/FormatterActions.prebundle.js --file gen/third_party/devtools-frontend/src/front_end/entrypoints/formatter_worker/FormatterActions.js --configDCHECK # {'exe': '/nix/store/i6svmmq24wl4i10dx0m8blrpl3m3fq4b-python3-3.10.5/bin/python3.10'} 0.0 0.0 122 224 2 1 node 32930: node ../../../../../src/3rdparty/chromium/third_party/devtools-frontend/src/node_modules/rollup/dist/bin/rollup --config ../../../../../src/3rdparty/chromium/third_party/devtools-frontend/src/scripts/build/rollup.config.js --input gen/third_party/devtools-frontend/src/front_end/entrypoints/formatter_worker/FormatterActions.prebundle.js --file gen/third_party/devtools-frontend/src/front_end/entrypoints/formatter_worker/FormatterActions.js --configDCHECK # {'exe': '/nix/store/2ph85rmc3j8n5k679y4zp0qvqzrmwwkn-nodejs-16.15.0/bin/node'} 0.0 0.0 25 224 1 0 node 32995: node /build/qtwebengine-everywhere-src-6.3.1/src/3rdparty/chromium/third_party/devtools-frontend/src/node_modules/jest-worker/build/workers/processChild.js # {'exe': '/nix/store/2ph85rmc3j8n5k679y4zp0qvqzrmwwkn-nodejs-16.15.0/bin/node'} 0.0 0.0 149 223 3 1 python3 33114: python3 ../../../../../src/3rdparty/chromium/third_party/node/node.py ../../../../../src/3rdparty/chromium/third_party/devtools-frontend/src/node_modules/rollup/dist/bin/rollup --silent --config ../../../../../src/3rdparty/chromium/third_party/devtools-frontend/src/scripts/build/rollup.config.js --input gen/third_party/devtools-frontend/src/front_end/core/platform/platform.prebundle.js --file gen/third_party/devtools-frontend/src/front_end/core/platform/platform.js --configDCHECK # {'exe': '/nix/store/i6svmmq24wl4i10dx0m8blrpl3m3fq4b-python3-3.10.5/bin/python3.10'} 0.0 0.0 139 223 2 1 node 33136: node ../../../../../src/3rdparty/chromium/third_party/devtools-frontend/src/node_modules/rollup/dist/bin/rollup --config ../../../../../src/3rdparty/chromium/third_party/devtools-frontend/src/scripts/build/rollup.config.js --input gen/third_party/devtools-frontend/src/front_end/core/platform/platform.prebundle.js --file gen/third_party/devtools-frontend/src/front_end/core/platform/platform.js --configDCHECK # {'exe': '/nix/store/2ph85rmc3j8n5k679y4zp0qvqzrmwwkn-nodejs-16.15.0/bin/node'} 0.0 0.0 25 223 1 0 node 33512: node /build/qtwebengine-everywhere-src-6.3.1/src/3rdparty/chromium/third_party/devtools-frontend/src/node_modules/jest-worker/build/workers/processChild.js # {'exe': '/nix/store/2ph85rmc3j8n5k679y4zp0qvqzrmwwkn-nodejs-16.15.0/bin/node'} 0.0 0.0 136 223 3 1 python3 33246: python3 ../../../../../src/3rdparty/chromium/third_party/node/node.py ../../../../../src/3rdparty/chromium/third_party/devtools-frontend/src/node_modules/rollup/dist/bin/rollup --silent --config ../../../../../src/3rdparty/chromium/third_party/devtools-frontend/src/scripts/build/rollup.config.js --input gen/third_party/devtools-frontend/src/front_end/models/heap_snapshot_model/heap_snapshot_model.prebundle.js --file gen/third_party/devtools-frontend/src/front_end/models/heap_snapshot_model/heap_snapshot_model.js --configDCHECK # {'exe': '/nix/store/i6svmmq24wl4i10dx0m8blrpl3m3fq4b-python3-3.10.5/bin/python3.10'} 0.0 0.0 126 223 2 1 node 33267: node ../../../../../src/3rdparty/chromium/third_party/devtools-frontend/src/node_modules/rollup/dist/bin/rollup --config ../../../../../src/3rdparty/chromium/third_party/devtools-frontend/src/scripts/build/rollup.config.js --input gen/third_party/devtools-frontend/src/front_end/models/heap_snapshot_model/heap_snapshot_model.prebundle.js --file gen/third_party/devtools-frontend/src/front_end/models/heap_snapshot_model/heap_snapshot_model.js --configDCHECK # {'exe': '/nix/store/2ph85rmc3j8n5k679y4zp0qvqzrmwwkn-nodejs-16.15.0/bin/node'} 0.0 0.0 25 222 1 0 node 33645: node /build/qtwebengine-everywhere-src-6.3.1/src/3rdparty/chromium/third_party/devtools-frontend/src/node_modules/jest-worker/build/workers/processChild.js # {'exe': '/nix/store/2ph85rmc3j8n5k679y4zp0qvqzrmwwkn-nodejs-16.15.0/bin/node'}

note the time column. the workers hang since 200 seconds with average load zero (2nd column)

blocked by ... ninja-build/ninja#2164

muscaln · 2022-06-22T13:19:43Z

tdesktop and tg_owt bumped in #178421.

nixos-discourse · 2022-06-24T17:22:22Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-build-profiling-memory-and-cpu-usage/19934/1

pkgs/development/libraries/qt-6/srcs.nix

kevincox · 2022-07-01T13:41:20Z

Result of nixpkgs-review pr 178171 run on x86_64-linux 1

2 packages marked as broken and skipped:

libsForQt512.fcitx5-qt
libsForQt514.fcitx5-qt

41 packages built:

cutemaze
fcitx5-chinese-addons
fcitx5-configtool
fcitx5-unikey
fcitx5-with-addons
jami-client-qt
jami-libclient
libsForQt5.fcitx5-qt (libsForQt515.fcitx5-qt ,plasma5Packages.fcitx5-qt)
qt6.qt3d (qt6Packages.qt3d)
qt6.qt5compat (qt6Packages.qt5compat)
qt6.qtbase (qt6Packages.qtbase)
qt6.qtcharts (qt6Packages.qtcharts)
qt6.qtconnectivity (qt6Packages.qtconnectivity)
qt6.qtdatavis3d (qt6Packages.qtdatavis3d)
qt6.qtdeclarative (qt6Packages.qtdeclarative)
qt6.qtdoc (qt6Packages.qtdoc)
qt6.qtimageformats (qt6Packages.qtimageformats)
qt6.qtlanguageserver (qt6Packages.qtlanguageserver)
qt6.qtlottie (qt6Packages.qtlottie)
qt6.qtmultimedia (qt6Packages.qtmultimedia)
qt6.qtnetworkauth (qt6Packages.qtnetworkauth)
qt6.qtpositioning (qt6Packages.qtpositioning)
qt6.qtquick3d (qt6Packages.qtquick3d)
qt6.qtquicktimeline (qt6Packages.qtquicktimeline)
qt6.qtremoteobjects (qt6Packages.qtremoteobjects)
qt6.qtscxml (qt6Packages.qtscxml)
qt6.qtsensors (qt6Packages.qtsensors)
qt6.qtserialbus (qt6Packages.qtserialbus)
qt6.qtserialport (qt6Packages.qtserialport)
qt6.qtshadertools (qt6Packages.qtshadertools)
qt6.qtsvg (qt6Packages.qtsvg)
qt6.qttools (qt6Packages.qttools)
qt6.qttranslations (qt6Packages.qttranslations)
qt6.qtvirtualkeyboard (qt6Packages.qtvirtualkeyboard)
qt6.qtwayland (qt6Packages.qtwayland)
qt6.qtwebchannel (qt6Packages.qtwebchannel)
qt6.qtwebengine (qt6Packages.qtwebengine)
qt6.qtwebsockets (qt6Packages.qtwebsockets)
qt6.qtwebview (qt6Packages.qtwebview)
qt6Packages.quazip
tdesktop

kevincox · 2022-07-01T13:47:50Z

This appears to fix the broken qtwebengine build so I think we should merge it. We can make other improvements as we go. Maybe file tickets for the failure to respect the cores config?

kevincox · 2022-07-04T14:14:49Z

pkgs/development/libraries/qt-6/modules/qtwebengine.nix

Are there plans to upstream these patches? Do we want to maintain these fairly significant changes to the build process?

To be honest I'm not a huge fan of the significant changes and complicated patching.

i linked upstream issues in my first post

what is worse?

wait for upstream

overload nix build servers

to review my patches, you would have to review ...

300 lines in tokenpool.py

260 lines in tokenpool.js

tokenpool-gnu-make.cc etc ...
these are 99% copy-paste from the tokenpool-master fork via ninja-build/ninja#1139

650 lines of patches

$ ( cd pkgs/development/libraries/qt-6/modules/patches/qtwebengine && wc -l * ) 129 0001-task_queue.py-add-jobclient.patch 24 0002-build_inspector_overlay.py-fix-inherit-fds.patch 282 0003-gn-add-jobclient.patch 95 0004-node.py-add-debug.patch 121 0005-mojom_parser.py-add-jobclient.patch 651 total

Yeah, I agree. It isn't pretty either way.

I think my preferences would be:

Accept PR as-is, with patches going upstream so that we can update to whatever upstream does once it happens. This way we just have temporary maintenance burden.

Suck it up and live with the poor resource usage.

Maintain these patches out-of-tree forever.

But I'm not really sure how much of a problem the poor scheduling is. Also if you are happy to maintain these patches for a long time then worst case is that we rip them out if we don't want to maintain them in the future.

I would love to have more opinions here.

pkgs/development/python-modules/nix-build-profiler/default.nix

pkgs/top-level/all-packages.nix

nrdxp · 2022-07-05T00:28:40Z

Would the ninja fix solve the problems we are having building qtwebengine on hydra?

This appears to fix the broken qtwebengine build so I think we should merge it.

The existing package itself is not broken. It's a ulimit issue with Nix itself. See here:
https://discourse.nixos.org/t/why-is-qt6-qtwebengine-failing-ulimit-n/19539

Further, these build patches seem to cause the builder to stall out completely at random places during the build. Each time I try to build it, it stalls in a different place, so I cannot complete the build because it just stalls permanently. Considering that, I am not in favor of the build patches, as without them the build completes just fine, although it does seem to progress more slowly. I would take a reliable build over a fast one any day.

kevincox · 2022-07-05T00:58:08Z

I tried building e206f87 and it has stalled out as well. Definitely can't be merged with the stalling.

milahu · 2022-07-05T08:18:56Z

I am not in favor of the build patches, as without them the build completes just fine, although it does seem to progress more slowly

yes, e45d046 works but ignores NIX_BUILD_CORES
could be slower due to overloading and congestion

kevincox · 2022-07-05T11:35:24Z

Ok, what do you think about resetting this PR to e45d046? Then we can deal with the other improvements in a different PR to be considered in isolation?

milahu · 2022-07-05T11:51:25Z

yes. just give me a few more minutes, ideally i just messed up the rebase

kevincox · 2022-07-05T12:24:19Z

Result of nixpkgs-review pr 178171 run on x86_64-linux 1

2 packages marked as broken and skipped:

libsForQt512.fcitx5-qt
libsForQt514.fcitx5-qt

41 packages built:

cutemaze
fcitx5-chinese-addons
fcitx5-configtool
fcitx5-unikey
fcitx5-with-addons
jami-client-qt
jami-libclient
libsForQt5.fcitx5-qt
qt6.qt3d
qt6.qt5compat
qt6.qtbase
qt6.qtcharts
qt6.qtconnectivity
qt6.qtdatavis3d
qt6.qtdeclarative
qt6.qtdoc
qt6.qtimageformats
qt6.qtlanguageserver
qt6.qtlottie
qt6.qtmultimedia
qt6.qtnetworkauth
qt6.qtpositioning
qt6.qtquick3d
qt6.qtquicktimeline
qt6.qtremoteobjects
qt6.qtscxml
qt6.qtsensors
qt6.qtserialbus
qt6.qtserialport
qt6.qtshadertools
qt6.qtsvg
qt6.qttools
qt6.qttranslations
qt6.qtvirtualkeyboard
qt6.qtwayland
qt6.qtwebchannel
qt6.qtwebengine
qt6.qtwebsockets
qt6.qtwebview
qt6Packages.quazip
tdesktop

milahu · 2022-07-05T12:29:31Z

please try to build qt6-631-debug-qtwebengine-3
im 10% into the build and its running as expected

kevincox · 2022-07-05T12:33:08Z

I get an error on 8f48223

log

% nix-build -A qt6.qtwebengine
these 12 derivations will be built:
  /nix/store/40wc4c6y9gldy8vsx7swkgclih03p4hn-source.drv
  /nix/store/83sj2lb3lc73g1hpgjj89qh53dbsfps8-qtbase-6.3.1.drv
  /nix/store/5nk779ghm8dygif8fkyi6ibg88ba2krd-qtserialport-6.3.1.drv
  /nix/store/x4sawdx3xzwxrf52nx9lwsghfs6dhm10-qtshadertools-6.3.1.drv
  /nix/store/cidc9vdjgyq2zi5jsjpf29fjp9cgm8ng-qtdeclarative-6.3.1.drv
  /nix/store/6arridv0q268jrh87ddj5f1nkgn3k0fq-qtpositioning-6.3.1.drv
  /nix/store/b1yfbdw04nfih0z668jm696glaim78kj-qtwebsockets-6.3.1.drv
  /nix/store/lv9g4pr6pydbrp83ylxl3qj0z7clnxk0-python3.9-gnumake-tokenpool-0.0.1.drv
  /nix/store/lzi42xh5y2r9vi49b3fb1q5fcsij7gbs-python3.9-nix-build-profiler-0.0.1.drv
  /nix/store/rx5h1zrdqlhk73j66q3d1y81rds1jn7c-qtwebchannel-6.3.1.drv
  /nix/store/x3hz687a372718w9qwp8qxxi8wkfvl14-ninja-tokenpool-unstable-2022-02-13.drv
  /nix/store/kwcwnrhd9pw1cy46j3y9lzazk9d5sjrk-qtwebengine-6.3.1.drv
these 15 paths will be fetched (24.15 MiB download, 98.48 MiB unpacked):
  /nix/store/0p53nw75f5ixvwzwv5xynbmkd56f4xx1-stoken-0.92
  /nix/store/46acy2byia8xb99z4addfrlrz4fq7kbs-pipewire-0.3.51-lib
  /nix/store/4xp79p3zg45ffiw6xmrpn33rm328i569-ffmpeg-4.4.1-dev
  /nix/store/5179ywqpvqlbfa2hasr1as74rq0ld4n2-networkmanager-1.38.0
  /nix/store/53sb4zq7201bs5wg102yy03af26d67vd-ffmpeg-4.4.1
  /nix/store/b633fhyfxcw95hybskdq15zdy3jsyhi0-pipewire-0.3.51-dev
  /nix/store/cw1nbvyxva8gqiazmwdif91vkh0q9p0p-libproxy-0.4.17
  /nix/store/l34rlzmpy1criasx6l0vjb63dfqvvj63-ffmpeg-4.4.1-bin
  /nix/store/mkzxyh06njyq485gavmvjcg67lzdqaqv-ffmpeg-4.4.1
  /nix/store/npqzzphxiwfhz3sshhx3vh2zp1x13hjm-SDL2-2.0.22
  /nix/store/q1xladjz5x2rzs3icr1y43ymn384g29b-pipewire-0.3.51
  /nix/store/vlclik0ciwfhplj06y9iv9py6h5n9594-gtk+3-3.24.34-dev
  /nix/store/wyl82a2pqhr2awwavbklh1cz16rhab2q-openconnect-8.20
  /nix/store/x38hs5vn93l68p2mw0i5xb535myz2p3k-libproxy-0.4.17-dev
  /nix/store/z61p740r7wld3rljzqknlj4jc2wp25hs-cups-2.4.2-dev
copying path '/nix/store/z61p740r7wld3rljzqknlj4jc2wp25hs-cups-2.4.2-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/mkzxyh06njyq485gavmvjcg67lzdqaqv-ffmpeg-4.4.1' from 'https://cache.nixos.org'...
building '/nix/store/x3hz687a372718w9qwp8qxxi8wkfvl14-ninja-tokenpool-unstable-2022-02-13.drv'...
building '/nix/store/40wc4c6y9gldy8vsx7swkgclih03p4hn-source.drv'...
copying path '/nix/store/0p53nw75f5ixvwzwv5xynbmkd56f4xx1-stoken-0.92' from 'https://cache.nixos.org'...
unpacking sources
unpacking source archive /nix/store/wgyxh790z2sqgk4mwk85m5x10cgjnbjm-source
source root is source
patching sources
configuring
no configure script, doing nothing
building
bootstrapping ninja...
copying path '/nix/store/vlclik0ciwfhplj06y9iv9py6h5n9594-gtk+3-3.24.34-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/wyl82a2pqhr2awwavbklh1cz16rhab2q-openconnect-8.20' from 'https://cache.nixos.org'...

trying https://github.com/milahu/gnumake-tokenpool/archive/7eba965adfafa429333dd40d7d920085578185ba.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
copying path '/nix/store/5179ywqpvqlbfa2hasr1as74rq0ld4n2-networkmanager-1.38.0' from 'https://cache.nixos.org'...
100  105k  100  105k    0     0   321k      0 --:--:-- --:--:-- --:--:--  321k
unpacking source archive /build/7eba965adfafa429333dd40d7d920085578185ba.tar.gz
building '/nix/store/lv9g4pr6pydbrp83ylxl3qj0z7clnxk0-python3.9-gnumake-tokenpool-0.0.1.drv'...
Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing setuptools-build-hook
Using setuptoolsBuildPhase
Using setuptoolsShellHook
Sourcing pip-install-hook
Using pipInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
Sourcing setuptools-check-hook
Using setuptoolsCheckPhase
unpacking sources
unpacking source archive /nix/store/759126rs3vqw3abx68lsq3i9s5v1gm4s-source
do not know how to unpack source archive /nix/store/759126rs3vqw3abx68lsq3i9s5v1gm4s-source
error: builder for '/nix/store/lv9g4pr6pydbrp83ylxl3qj0z7clnxk0-python3.9-gnumake-tokenpool-0.0.1.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/lzi42xh5y2r9vi49b3fb1q5fcsij7gbs-python3.9-nix-build-profiler-0.0.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/kwcwnrhd9pw1cy46j3y9lzazk9d5sjrk-qtwebengine-6.3.1.drv' failed to build

milahu · 2022-07-05T12:41:14Z

youre hitting NixOS/nix#6572

workaround

nix-build -E '
let
  pkgs = import <nixpkgs> {};
in
  pkgs.fetchFromGitHub {
    owner = "milahu";
    repo = "gnumake-tokenpool";
    rev = "4eb559ae323bef153cbe1d0a5e3496b377fb7856";
    sha256 = "v3UqPi4fCnN86xmp4eOSz8IXNH70E8Kvvq6HVOiCiwk=";
  }
'

kevincox · 2022-07-05T12:49:12Z

Same issue:

log

8:48:15% nix-build -E ' let pkgs = import <nixpkgs> {}; in pkgs.fetchFromGitHub { owner = "milahu"; repo = "gnumake-tokenpool"; rev = "4eb559ae323bef153cbe1d0a5e3496b377fb7856"; sha256 = "v3UqPi4fCnN86xmp4eOSz8IXNH70E8Kvvq6HVOiCiwk="; } ' /nix/store/nqw44wdq34xd0klarc6li51lv8iyihiv-source 0 kevinryzen:~/p/nixpkgs#8f482236935 8:48:16% nix-build -A qt6.qtwebengine these 12 derivations will be built: /nix/store/40wc4c6y9gldy8vsx7swkgclih03p4hn-source.drv /nix/store/83sj2lb3lc73g1hpgjj89qh53dbsfps8-qtbase-6.3.1.drv /nix/store/5nk779ghm8dygif8fkyi6ibg88ba2krd-qtserialport-6.3.1.drv /nix/store/x4sawdx3xzwxrf52nx9lwsghfs6dhm10-qtshadertools-6.3.1.drv /nix/store/cidc9vdjgyq2zi5jsjpf29fjp9cgm8ng-qtdeclarative-6.3.1.drv /nix/store/6arridv0q268jrh87ddj5f1nkgn3k0fq-qtpositioning-6.3.1.drv /nix/store/b1yfbdw04nfih0z668jm696glaim78kj-qtwebsockets-6.3.1.drv /nix/store/lv9g4pr6pydbrp83ylxl3qj0z7clnxk0-python3.9-gnumake-tokenpool-0.0.1.drv /nix/store/lzi42xh5y2r9vi49b3fb1q5fcsij7gbs-python3.9-nix-build-profiler-0.0.1.drv /nix/store/rx5h1zrdqlhk73j66q3d1y81rds1jn7c-qtwebchannel-6.3.1.drv /nix/store/x3hz687a372718w9qwp8qxxi8wkfvl14-ninja-tokenpool-unstable-2022-02-13.drv /nix/store/kwcwnrhd9pw1cy46j3y9lzazk9d5sjrk-qtwebengine-6.3.1.drv these 3 paths will be fetched (8.32 MiB download, 26.03 MiB unpacked): /nix/store/4xp79p3zg45ffiw6xmrpn33rm328i569-ffmpeg-4.4.1-dev /nix/store/53sb4zq7201bs5wg102yy03af26d67vd-ffmpeg-4.4.1 /nix/store/l34rlzmpy1criasx6l0vjb63dfqvvj63-ffmpeg-4.4.1-bin copying path '/nix/store/53sb4zq7201bs5wg102yy03af26d67vd-ffmpeg-4.4.1' from 'https://cache.nixos.org'... building '/nix/store/x3hz687a372718w9qwp8qxxi8wkfvl14-ninja-tokenpool-unstable-2022-02-13.drv'... building '/nix/store/40wc4c6y9gldy8vsx7swkgclih03p4hn-source.drv'... building '/nix/store/83sj2lb3lc73g1hpgjj89qh53dbsfps8-qtbase-6.3.1.drv'... unpacking sources unpacking source archive /nix/store/wgyxh790z2sqgk4mwk85m5x10cgjnbjm-source source root is source patching sources configuring no configure script, doing nothing building bootstrapping ninja...

trying https://github.com/milahu/gnumake-tokenpool/archive/7eba965adfafa429333dd40d7d920085578185ba.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 105k 0 105k 0 0 328k 0 --:--:-- --:--:-- --:--:-- 328k
unpacking source archive /build/7eba965adfafa429333dd40d7d920085578185ba.tar.gz
unpacking sources
unpacking source archive /nix/store/bja9gvymwqjilwb2r90c54dg4nbqxgac-qtbase-everywhere-src-6.3.1.tar.xz
building '/nix/store/lv9g4pr6pydbrp83ylxl3qj0z7clnxk0-python3.9-gnumake-tokenpool-0.0.1.drv'...
Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing setuptools-build-hook
Using setuptoolsBuildPhase
Using setuptoolsShellHook
Sourcing pip-install-hook
Using pipInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
Sourcing setuptools-check-hook
Using setuptoolsCheckPhase
unpacking sources
unpacking source archive /nix/store/759126rs3vqw3abx68lsq3i9s5v1gm4s-source
do not know how to unpack source archive /nix/store/759126rs3vqw3abx68lsq3i9s5v1gm4s-source
error: builder for '/nix/store/lv9g4pr6pydbrp83ylxl3qj0z7clnxk0-python3.9-gnumake-tokenpool-0.0.1.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/lzi42xh5y2r9vi49b3fb1q5fcsij7gbs-python3.9-nix-build-profiler-0.0.1.drv' failed to build
error (ignored): error: cannot unlink '/tmp/nix-build-qtbase-6.3.1.drv-0/qtbase-everywhere-src-6.3.1': Directory not empty
error: 1 dependencies of derivation '/nix/store/kwcwnrhd9pw1cy46j3y9lzazk9d5sjrk-qtwebengine-6.3.1.drv' failed to build
1 kevinryzen:~/p/nixpkgs#8f482236935

milahu · 2022-07-05T13:27:40Z

sorry, i had the wrong version
must be same as in pkgs/development/python-modules/gnumake-tokenpool/default.nix

Details

nix-build -E '
let
  pkgs = import <nixpkgs> {};
in
  pkgs.fetchFromGitHub {
    owner = "milahu";
    repo = "gnumake-tokenpool";
    rev = "7eba965adfafa429333dd40d7d920085578185ba";
    sha256 = "PpQ4uNClPCMwVoRWDq7PJmGbRCoaTjwEkyJviYLDqMM=";
  }
'

kevincox · 2022-07-05T16:04:15Z

That built for me.

kevincox · 2022-07-05T16:20:18Z

Any objections to merging this PR as-is? We can then evaluate other improvements as independent followups?

milahu · 2022-07-05T16:24:46Z

fine with me. the fix for qtwebengine still needs some polish = time

kevincox · 2022-07-07T18:47:12Z

This is failing on nixos-unstable with:

/bin/sh: /build/qtwebengine-everywhere-src-6.3.1/build/linker_ulimit.sh: not found

https://hydra.nixos.org/build/182882913/nixlog/1/tail

kevincox · 2022-07-07T18:53:13Z

This appears to come from https://sourcegraph.com/github.com/qt/qtwebengine@126584f52e883a8caaacb79c34e9e0779260d61f/-/blob/cmake/Functions.cmake?L1226-1229&subtree=true which uses /bin/bash in the shebang which won't exist in a build.

kevincox · 2022-07-07T18:53:57Z

It's not clear to me why this worked before merging. Doesn't seem like it would be a merge conflict.

milahu · 2022-07-07T18:59:03Z

It's not clear to me why this worked before merging.

see the condition ulimitOutput LESS 4096

edit:

similar upstream issue https://bugreports.qt.io/browse/QTBUG-97568
my failed attempts to send patches to gerrit https://codereview.qt-project.org/c/qt/qtbase/+/378519

in this case, i guess upstream would accept a patch

kevincox · 2022-07-07T20:48:28Z

I'm still not sure why this was different before and after the merge though. I built on the same nix-daemon for both. Unless nixpkgs-review is setting funny options.

milahu · 2022-07-07T20:59:34Z

did you see ...?
https://discourse.nixos.org/t/why-is-qt6-qtwebengine-failing-ulimit-n/19539

kevincox · 2022-07-07T21:01:23Z

Yes, but unless I am missing something it shouldn't be the (only) problem here.

The thread seems to imply that the limit was raised already and it still failed yesterday.
It doesn't affect my workstation when I built before and after.

milahu · 2022-07-07T21:21:06Z

shrug ...

$ ulimit -n
4096

$ nix-build -E 'with import <nixpkgs> {}; stdenv.mkDerivation { name = "x"; unpackPhase = "ulimit -n; exit 1"; }' 
...
131072

edit: possibly also a symptom of cpu overloading

kevincox · 2022-07-07T22:15:28Z

% ulimit -n
1024
% nix-build -E 'with import <nixpkgs> {}; stdenv.mkDerivation { name = "x"; unpackPhase = "ulimit -n; exit 1"; }'

this derivation will be built:
  /nix/store/69i3v72clnj460bpw8csw87cgy4k68nr-x.drv
building '/nix/store/69i3v72clnj460bpw8csw87cgy4k68nr-x.drv'...
unpacking sources
1048576

But from the log on my machine.

-- Checking 'ulimit -n'
 -- Open files limit 1048576

The hydra log does have "Open files limit 1024" though.

kevincox · 2022-07-07T22:29:41Z

Hmm, It seems that the code attempts to extract the integer output from ulimitOutput into limit. But then the condition uses ulimitOutput.

function(check_for_ulimit)
    message("-- Checking 'ulimit -n'")
    execute_process(COMMAND bash -c "ulimit -n"
        OUTPUT_VARIABLE ulimitOutput
    )
    string(REGEX MATCHALL "[0-9]+" limit "${ulimitOutput}")
    message(" -- Open files limit ${limit}")
    if(NOT (QT_FEATURE_use_gold_linker OR QT_FEATURE_use_lld_linker) AND ulimitOutput LESS 4096)
       # ... snip ...
    endif()
endfunction()

But the output of ulimit -n should be just an integer anyways. So that shouldn't matter.

kevincox · 2022-07-07T22:44:46Z

Ah, I found the problem. nixos-rebuild doesn't appear to use the nix-daemon so it was using a different ulimit. Building with nix-build worked. This explains why it worked in nixpkgs-review and why it doesn't build for nixos-rebuild and on hydra. (Other than apparently hydra did bump their limit).

ofborg bot requested a review from oxalica June 18, 2022 18:11

ofborg bot added 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. labels Jun 18, 2022

SuperSandro2000 reviewed Jun 18, 2022

View reviewed changes

pkgs/development/libraries/qt-6/modules/qtwebengine.nix Outdated Show resolved Hide resolved

pkgs/applications/networking/instant-messengers/telegram/tdesktop/tg_owt.nix Outdated Show resolved Hide resolved

NickCao mentioned this pull request Jun 19, 2022

qt6: 6.3.0 -> 6.3.1 #178003

Closed

13 tasks

This comment was marked as spam.

Sign in to view

SuperSandro2000 reviewed Jun 22, 2022

View reviewed changes

milahu commented Jun 28, 2022

View reviewed changes

pkgs/development/libraries/qt-6/srcs.nix Outdated Show resolved Hide resolved

kevincox approved these changes Jul 1, 2022

View reviewed changes

qt6: 6.3.0 -> 6.3.1

e45d046

github-actions bot added the 6.topic: python Python is a high-level, general-purpose programming language. label Jul 4, 2022

ofborg bot added 8.has: package (new) This PR adds a new package 11.by: package-maintainer This PR was created by a maintainer of all the package it changes. labels Jul 4, 2022

kevincox reviewed Jul 4, 2022

View reviewed changes

pkgs/top-level/all-packages.nix Outdated Show resolved Hide resolved

github-actions bot removed the 6.topic: python Python is a high-level, general-purpose programming language. label Jul 4, 2022

kevincox merged commit 051b5b6 into NixOS:master Jul 5, 2022

Uh oh!

Conversation

milahu commented Jun 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as spam.

milahu commented Jun 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

milahu commented Jun 20, 2022

Uh oh!

NickCao commented Jun 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

milahu Jun 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

milahu Jul 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

muscaln commented Jun 22, 2022

Uh oh!

nixos-discourse commented Jun 24, 2022

Uh oh!

Uh oh!

kevincox commented Jul 1, 2022

Uh oh!

kevincox commented Jul 1, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

milahu Jul 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

nrdxp commented Jul 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevincox commented Jul 5, 2022

Uh oh!

milahu commented Jul 5, 2022

Uh oh!

kevincox commented Jul 5, 2022

Uh oh!

milahu commented Jul 5, 2022

Uh oh!

kevincox commented Jul 5, 2022

Uh oh!

milahu commented Jul 5, 2022

Uh oh!

kevincox commented Jul 5, 2022

Uh oh!

milahu commented Jul 5, 2022

Uh oh!

kevincox commented Jul 5, 2022

Uh oh!

milahu commented Jul 5, 2022

Uh oh!

milahu commented Jun 18, 2022 •

edited

Loading

milahu commented Jun 19, 2022 •

edited

Loading

NickCao commented Jun 22, 2022 •

edited

Loading

milahu Jun 25, 2022 •

edited

Loading

milahu Jul 4, 2022 •

edited

Loading

milahu Jul 4, 2022 •

edited

Loading

nrdxp commented Jul 5, 2022 •

edited

Loading

kevincox commented Jul 5, 2022 •

edited

Loading

milahu commented Jul 7, 2022 •

edited

Loading

milahu commented Jul 7, 2022 •

edited

Loading