depends: Remove Qt build-time dependencies #29923

laanwj · 2024-04-20T15:40:35Z

In the spirit of the halving, let's (approximately) halve the number of packages in depends. Remove the following packages:

  bdb.mk
  boost.mk
  capnp.mk
- expat.mk
- fontconfig.mk
- freetype.mk
  libevent.mk
  libmultiprocess.mk
  libnatpmp.mk
- libXau.mk
- libxcb.mk
- libxcb_util.mk
- libxcb_util_image.mk
- libxcb_util_keysyms.mk
- libxcb_util_render.mk
- libxcb_util_wm.mk
- libxkbcommon.mk
  miniupnpc.mk
  native_capnp.mk
  native_cctools.mk
  native_libmultiprocess.mk
  native_libtapi.mk
  native_llvm.mk
  packages.mk
  qrencode.mk
  qt.mk
+ qtsowrap.mk
  sqlite.mk
  systemtap.mk
- xcb_proto.mk
- xproto.mk
  zeromq.mk

This is done by loading all of Qt's system dependencies at run-time, replacing them with an auto-generated wrapper library qtsowrap:

See https://github.com/laanwj/qtsowrap for details.

After this, bitcoin-qt has the same direct linking dependencies as bitcoind and other binaries:

$ ldd build/src/qt/bitcoin-qt
        linux-vdso.so.1 (0x00007ffc01d83000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f76dbf1f000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f76dbc00000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f76de7a7000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f76dba1f000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f76de7d7000)

(libdl.so.X needed too depending on the libc version)

For comparison, bitcoin-qt from release 27.0 has the following:

$ ldd ~/workdir/x86_64-linux-gnu/bitcoin-27.0/bin/bitcoin-qt 
        linux-vdso.so.1 (0x00007ffde7f73000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa098e37000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa098e2f000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa098d4f000)
        libfontconfig.so.1 => /lib/x86_64-linux-gnu/libfontconfig.so.1 (0x00007fa098cff000)
        libfreetype.so.6 => /lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007fa09632f000)
        libxcb-icccm.so.4 => /lib/x86_64-linux-gnu/libxcb-icccm.so.4 (0x00007fa098cf7000)
        libxcb-image.so.0 => /lib/x86_64-linux-gnu/libxcb-image.so.0 (0x00007fa098cef000)
        libxcb-shm.so.0 => /lib/x86_64-linux-gnu/libxcb-shm.so.0 (0x00007fa098ce7000)
        libxcb-keysyms.so.1 => /lib/x86_64-linux-gnu/libxcb-keysyms.so.1 (0x00007fa096000000)
        libxcb-randr.so.0 => /lib/x86_64-linux-gnu/libxcb-randr.so.0 (0x00007fa098ccf000)
        libxcb-render-util.so.0 => /lib/x86_64-linux-gnu/libxcb-render-util.so.0 (0x00007fa095c00000)
        libxcb-render.so.0 => /lib/x86_64-linux-gnu/libxcb-render.so.0 (0x00007fa098cbf000)
        libxcb-shape.so.0 => /lib/x86_64-linux-gnu/libxcb-shape.so.0 (0x00007fa098cb7000)
        libxcb-sync.so.1 => /lib/x86_64-linux-gnu/libxcb-sync.so.1 (0x00007fa098caf000)
        libxcb-xfixes.so.0 => /lib/x86_64-linux-gnu/libxcb-xfixes.so.0 (0x00007fa098c9f000)
        libxcb-xinerama.so.0 => /lib/x86_64-linux-gnu/libxcb-xinerama.so.0 (0x00007fa096327000)
        libxcb-xkb.so.1 => /lib/x86_64-linux-gnu/libxcb-xkb.so.1 (0x00007fa096307000)
        libxcb.so.1 => /lib/x86_64-linux-gnu/libxcb.so.1 (0x00007fa0962d7000)
        libxkbcommon-x11.so.0 => /lib/x86_64-linux-gnu/libxkbcommon-x11.so.0 (0x00007fa0962c7000)
        libxkbcommon.so.0 => /lib/x86_64-linux-gnu/libxkbcommon.so.0 (0x00007fa09627f000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fa09625f000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa095e1f000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fa098e4f000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007fa09622f000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fa09620f000)
        libpng16.so.16 => /lib/x86_64-linux-gnu/libpng16.so.16 (0x00007fa095bc7000)
        libbrotlidec.so.1 => /lib/x86_64-linux-gnu/libbrotlidec.so.1 (0x00007fa095e0f000)
        libxcb-util.so.1 => /lib/x86_64-linux-gnu/libxcb-util.so.1 (0x00007fa095bbf000)
        libXau.so.6 => /lib/x86_64-linux-gnu/libXau.so.6 (0x00007fa096207000)
        libXdmcp.so.6 => /lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007fa095800000)
        libbrotlicommon.so.1 => /lib/x86_64-linux-gnu/libbrotlicommon.so.1 (0x00007fa095b97000)
        libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007fa095b7f000)
        libmd.so.0 => /lib/x86_64-linux-gnu/libmd.so.0 (0x00007fa095b6f000)

This does not affect Windows or MacOS, which already do not have this problem of having to build half the OS in the dependency chain.

This will make it possible to add Wayland support to the release binaries without requiring the Wayland (or X) libraries to be installed on every system that runs them.

Reviewing/Testing

Three relatively small Python scripts that generate everything:
One important thing to test is that the resulting bitcoin-qt binaries work on a range of different Linux distributions (also older ones that people still care about).
i've added some review notes for the Qt patch here: https://github.com/laanwj/qtsowrap?tab=readme-ov-file#patching-qt . What is important to ensure is that initialize_ happens before any calls to a library, as calling early will cause a crash. Forgetting to wrap a header will be found during linking, as the symbol will be wrong, so it's not a review concern.

TODOs (all done)

(before merge, probably)

DrahtBot · 2024-04-20T15:40:38Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type	Reviewers
Concept ACK	jonatack

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#29880 (depends: build FreeType with CMake by fanquake)
#29878 (depends: build expat with CMake by fanquake)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

luke-jr · 2024-04-21T03:10:03Z

Instead of Qt's dependencies, maybe we should wrap Qt itself this way? Or do C++ templates break that too much?

laanwj · 2024-04-21T06:55:22Z

Instead of Qt's dependencies, maybe we should wrap Qt itself this way? Or do C++ templates break that too much?

This is basically not possible with a C++ library. It is thanks to C's lack of name mangling and its straightforward linkage model that makes this possible. Also Qt has a ton of symbols compared to these libraries. And even if you narrow it down to only the used symbols, it's much harder to keep track of.

Conceptually, a few low-level X11 (or in the future, Wayland) and font libraries are much more likely to be on the user's system than a compatible version of Qt. That will only become worse with Qt 5 versus 6. This PR, as is, doesn't change the needed libraries, only how they're imported.

DrahtBot · 2024-04-22T05:04:43Z

Guix builds (on x86_64)

File	commit `67c0d93` (master)	commit eed10dd6f6d9a7cdc3e8379cf8b12a5a2995d575 (master and this pull)
SHA256SUMS.part	`0744b1285352a368...`	`1a98bfe3c1beccf7...`
*-aarch64-linux-gnu-debug.tar.gz	`16b414f82f4f07bd...`	`67822a1b1b1ba79a...`
*-aarch64-linux-gnu.tar.gz	`117b35bd779ed725...`	`c62e79c489bb0cb6...`
*-arm-linux-gnueabihf-debug.tar.gz	`07e864c7389441c9...`	`29c2ee47a1176010...`
*-arm-linux-gnueabihf.tar.gz	`db135d89aa80ca9a...`	`1c0f91b81c0cefdf...`
*-arm64-apple-darwin-unsigned.tar.gz	`98e4bf8a803c8a43...`	`fd94966e3845c4db...`
*-arm64-apple-darwin-unsigned.zip	`6aecc3420e8db5f6...`	`d3febce3a26e5eed...`
*-arm64-apple-darwin.tar.gz	`0f34ec3767a21c30...`	`05cb2476b672acf9...`
*-powerpc64-linux-gnu-debug.tar.gz	`ffee35ca9046fcfe...`	`566d5b48f75755bf...`
*-powerpc64-linux-gnu.tar.gz	`5d7b88059c977045...`	`eb009b208f979a01...`
*-riscv64-linux-gnu-debug.tar.gz	`40554584a6519644...`	`e3659a5086fe1b79...`
*-riscv64-linux-gnu.tar.gz	`0e28036c9438f9d3...`	`5fd8ac0c3f1e39a4...`
*-x86_64-apple-darwin-unsigned.tar.gz	`c9f7d339c48b8c3a...`	`ff199bd2a879b198...`
*-x86_64-apple-darwin-unsigned.zip	`6bb037c1a1e0003c...`	`cd284f0531fb033d...`
*-x86_64-apple-darwin.tar.gz	`02856cf44b673e4d...`	`4d4ffc4a383a8038...`
*-x86_64-linux-gnu-debug.tar.gz	`8938c00d38015b39...`	`e539b47da948ddc1...`
*-x86_64-linux-gnu.tar.gz	`d6ed9bb233a75b51...`	`361a416eb9b6c192...`
*.tar.gz	`8a85911c482de010...`	`1a83436d12c8d025...`
guix_build.log	`1cef86b1f8b71fe8...`	`91d6890f319ecbbd...`
guix_build.log.diff		`97e151810f253905...`

laanwj · 2024-04-22T06:29:09Z

One important thing to test here is that the resulting bitcoin-qt binaries work on a range of different Linux distributions (also older ones that people still care about).

There's no inherent reason to assume this will make compatibility worse, but it currently loads all symbols from the headers for the respective libraries (https://github.com/laanwj/qtsowrap?tab=readme-ov-file#versions) which might enforce the lower bound on version stronger than used to be done.

Might consider downgrading the headers, if this turns out to be a problem. Or add a list in the descriptor per library of the functions actually used. But this is more work to maintain than a list of version numbers.

laanwj · 2024-04-22T10:04:24Z

~~Example: failure on Ubuntu 22.04:~~

$ ./bitcoin-qt
/lib/x86_64-linux-gnu/libxcb-xfixes.so.0: undefined symbol: xcb_xfixes_set_client_disconnect_mode_checked
qt.qpa.xcb: Unable to load required libraries, skipping initialization of XCB backend.
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: xcb.

Aborted (core dumped)

Ubuntu 22.04 has the following version of libxcb-xfixes:

$ apt-cache show libxcb-xfixes0
Package: libxcb-xfixes0
Architecture: amd64
Version: 1.14-3ubuntu3

1.14. That's kind of strange! That's the same one we have.
It may be due to a difference in xcb_proto (which supplies the protocols that the header files are generated from) instead.
Ours is 1.15.2 (as of 7cb88c8), ubuntu 22.04 has 1.14. So i'll downgrade to that.

Edit: that was it! Ubuntu 22.04 now works.

DrahtBot · 2024-04-22T21:49:02Z

Guix builds (on x86_64)

File	commit `67c0d93` (master)	commit 510237f44755ef16db00758f1a9639350638c16e (master and this pull)
SHA256SUMS.part	`0744b1285352a368...`	`3ae06a46ca780c3c...`
*-aarch64-linux-gnu-debug.tar.gz	`16b414f82f4f07bd...`	`d0129eb64e24dc3f...`
*-aarch64-linux-gnu.tar.gz	`117b35bd779ed725...`	`cb011e21503ded6f...`
*-arm-linux-gnueabihf-debug.tar.gz	`07e864c7389441c9...`	`5d6a6d11e3e615ae...`
*-arm-linux-gnueabihf.tar.gz	`db135d89aa80ca9a...`	`ca915b6fc079495b...`
*-arm64-apple-darwin-unsigned.tar.gz	`98e4bf8a803c8a43...`	`025f3135577269af...`
*-arm64-apple-darwin-unsigned.zip	`6aecc3420e8db5f6...`	`9698d6f66962aabc...`
*-arm64-apple-darwin.tar.gz	`0f34ec3767a21c30...`	`853bbf572d24f213...`
*-powerpc64-linux-gnu-debug.tar.gz	`ffee35ca9046fcfe...`	`1cd5511bf725a041...`
*-powerpc64-linux-gnu.tar.gz	`5d7b88059c977045...`	`8a3d01afae413c2a...`
*-riscv64-linux-gnu-debug.tar.gz	`40554584a6519644...`	`a4471ab56fa86368...`
*-riscv64-linux-gnu.tar.gz	`0e28036c9438f9d3...`	`a80860a866930eeb...`
*-x86_64-apple-darwin-unsigned.tar.gz	`c9f7d339c48b8c3a...`	`12d8bf9080c88e99...`
*-x86_64-apple-darwin-unsigned.zip	`6bb037c1a1e0003c...`	`a245b1f75d50d4b0...`
*-x86_64-apple-darwin.tar.gz	`02856cf44b673e4d...`	`45496c3276b269d0...`
*-x86_64-linux-gnu-debug.tar.gz	`8938c00d38015b39...`	`446391abd341af1f...`
*-x86_64-linux-gnu.tar.gz	`d6ed9bb233a75b51...`	`aa5d8a27ec441c31...`
*.tar.gz	`8a85911c482de010...`	`28b5b41a3d16ba53...`
guix_build.log	`682b861e333169fc...`	`d817b205d0a593da...`
guix_build.log.diff		`bb1c42ae597e504e...`

laanwj · 2024-04-23T10:23:53Z

i've added some review notes for the Qt patch here: https://github.com/laanwj/qtsowrap?tab=readme-ov-file#patching-qt

What is important to ensure is that initialize_ happens before any calls to a library, as calling early will cause a crash. Forgetting to wrap a header will be found during linking as the symbol will be wrong.

maflcko · 2024-05-28T19:27:00Z

This PR doesn't change the libraries required to be installed on the user's system

Correct. I tested this in three different boxes and it worked.

laanwj · 2024-05-30T08:48:40Z

@maflcko Thank you for testing!

Force-push: trivial rebase for Android removal from build system

theuni · 2024-06-06T13:58:17Z

Closed on purpose?

laanwj · 2024-06-06T14:40:45Z

i was a bit tired of seeing "you need to rebase" messages, will get around to this 😄

This was required by xkbcommon build, which is no longer a dependency for build.

laanwj · 2024-06-17T17:57:24Z

Force-push: rebased to master for qt version bump. Trivial changes to depends, didn't need to change qt patch.

DrahtBot · 2024-06-17T20:07:28Z

🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the
documentation.

Possibly this is due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.

Leave a comment here, if you need help tracking down a confusing failure.

_{Debug: https://github.com/bitcoin/bitcoin/runs/26328929494}

These are indirect dependencies of Qt and don't necessarily need to be mentioned here. Fix "target filename not found" md linter error.

DrahtBot · 2024-07-19T09:55:46Z

🐙 This pull request conflicts with the target branch and needs rebase.

laanwj · 2024-08-27T13:39:03Z

closing due to lack of interest

laanwj added Linux/Unix Build system labels Apr 20, 2024

bitcoin deleted a comment Apr 20, 2024

DrahtBot added the CI failed label Apr 20, 2024

laanwj force-pushed the 2024-04-qtsowrap branch 3 times, most recently from 534b230 to 6cd5e8a Compare April 20, 2024 18:48

This was referenced Apr 21, 2024

guix: remove bzip2 from deps #29895

Merged

depends: build FreeType with CMake #29880

Merged

depends: build expat with CMake #29878

Merged

laanwj mentioned this pull request Apr 21, 2024

[Linux] Add wayland support #19950

Open

maflcko added the DrahtBot Guix build requested label Apr 21, 2024

DrahtBot removed the DrahtBot Guix build requested label Apr 22, 2024

DrahtBot removed the CI failed label Apr 22, 2024

maflcko added the DrahtBot Guix build requested label Apr 22, 2024

DrahtBot added the CI failed label Apr 22, 2024

laanwj mentioned this pull request Apr 22, 2024

RFC: In guix compile the GUI sequentially from everything else? #29914

Open

DrahtBot removed DrahtBot Guix build requested CI failed labels Apr 22, 2024

DrahtBot added the CI failed label Apr 23, 2024

laanwj force-pushed the 2024-04-qtsowrap branch from 39e891d to eb60c77 Compare April 23, 2024 11:58

DrahtBot removed the CI failed label Apr 23, 2024

DrahtBot added the Needs rebase label May 30, 2024

laanwj force-pushed the 2024-04-qtsowrap branch from 2d8362c to 23666aa Compare May 30, 2024 09:12

DrahtBot removed the Needs rebase label May 30, 2024

This was referenced May 31, 2024

depends: consolidate dependency docs #30204

Merged

depends: remove FORCE_USE_SYSTEM_CLANG #30201

Merged

depends: qt 5.15.14 and fix macOS build with Clang 18 #30198

Merged

DrahtBot added the Needs rebase label May 31, 2024

laanwj closed this Jun 4, 2024

laanwj added 5 commits June 17, 2024 14:58

depends: Add qtsowrap, make Qt build depend on it

a26b8af

depends: Add qtsowrap patch for Qt

97e5c3e

contrib: Remove qt dependency libraries from symbol check

75f2b57

depends: Actually remove Qt dependencies packages

16db854

guix, depends, ci: Remove bison dependency

3ce0fb1

This was required by xkbcommon build, which is no longer a dependency for build.

laanwj reopened this Jun 17, 2024

laanwj force-pushed the 2024-04-qtsowrap branch from 23666aa to 3ce0fb1 Compare June 17, 2024 17:55

DrahtBot added CI failed and removed Needs rebase labels Jun 17, 2024

doc: Remove fontconfig and freetype from dependencies.md

7acb6c0

These are indirect dependencies of Qt and don't necessarily need to be mentioned here. Fix "target filename not found" md linter error.

DrahtBot removed the CI failed label Jun 18, 2024

DrahtBot added the Needs rebase label Jul 19, 2024

laanwj closed this Aug 27, 2024

bitcoin locked and limited conversation to collaborators Aug 27, 2025

depends: Remove Qt build-time dependencies #29923

depends: Remove Qt build-time dependencies #29923

Uh oh!

Conversation

laanwj commented Apr 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviewing/Testing

TODOs (all done)

Uh oh!

DrahtBot commented Apr 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage

Reviews

Conflicts

Uh oh!

luke-jr commented Apr 21, 2024

Uh oh!

laanwj commented Apr 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DrahtBot commented Apr 22, 2024

Guix builds (on x86_64)

Uh oh!

laanwj commented Apr 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laanwj commented Apr 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DrahtBot commented Apr 22, 2024

Guix builds (on x86_64)

Uh oh!

laanwj commented Apr 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maflcko commented May 28, 2024

Uh oh!

laanwj commented May 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

theuni commented Jun 6, 2024

Uh oh!

laanwj commented Jun 6, 2024

Uh oh!

laanwj commented Jun 17, 2024

Uh oh!

DrahtBot commented Jun 17, 2024

Uh oh!

DrahtBot commented Jul 19, 2024

Uh oh!

laanwj commented Aug 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

laanwj commented Apr 20, 2024 •

edited

Loading

DrahtBot commented Apr 20, 2024 •

edited

Loading

laanwj commented Apr 21, 2024 •

edited

Loading

laanwj commented Apr 22, 2024 •

edited

Loading

laanwj commented Apr 22, 2024 •

edited

Loading

laanwj commented Apr 23, 2024 •

edited

Loading

laanwj commented May 30, 2024 •

edited

Loading