cpython: have powerpc64le use "ppc64le" to follow PEP600#168083
Conversation
|
I'm still not confident that Python itself expects to be called Like I said in the previous issue, I think the right solution might be to just not build manylinux wheels by default. |
That's why I checked the behavior on Debian, which is definitely the distribution with the best support for ppc64le. Debian puts ppc64le in the wheel name. See the
Frankly I wouldn't know who's in charge of that in order to ask, or how to ask. Do you know? They'll probably look at me strangely and ask what this weird "nix" stuff is all about and tell me to just run I'm not a python developer, I just really like qutebrowser.
Unfortunately PyQt5 has needed a manylinux wheel since February: #163589 |
|
So the problem that leads to that error building PyQt5 and stops you from using qutebrowser is that sip generates a wheel name for PyQt5 that pip does not believe is correct. The ultimate problem on PowerPC seems to be that in some situations the architecture is called
What do NixOS and Debian (and Fedora, if you can test it easily) print for
I understand the sentiment, but I'm pretty sure this will give you a Python which calls the architecture It turns out further in that thread that CentOS/RHEL/Fedora patches their Python to call the architecture That issue is also where I would post a comment to ask for upstream's opinion, and what the planned future names are for both the architecture as returned by
The reason this issue originally occurred on aarch64 is that sip was updated past 5.2.0, which started to build manylinux wheels by default, with an invalid name. However, sip has an option to turn this off. Manylinux wheels are pretty useless with NixOS because a) they refer to libraries by paths only present on a NixOS system, instead of generic paths that would be on many distros, and b) you can't actually do anything with them anyway, they only exist for a few seconds between the package build completing and the wheel being installed in the next phase. Therefore, I propose that the ultimate solution is to patch sip to not build manylinux wheels by default (or perhaps to forbid building them entirely), and instead build a wheel for the specific system on which it will be installed like it did before, thus sidestepping this problem of the architecture name entirely. It's not clear to me if this works properly under cross-compilation, or if cross-compilation even works properly now (I suspect not). I can look into creating this patch myself (and will be sure to test it on aarch64 if you create it), but it should be easy for you to check if simply patching |
Yes, unfortunately that problem exists because so many software packages and linux distributions cook up their own naming scheme for architectures instead of simply using gcc/autoconf tuples. You'll see Unfortunately Python does not follow the gcc/autoconf taxonomy.
I don't use NixOS, but I can answer for nixpkgs instead. sysconfig.get_platform()My Debian (bullseye,
Nope. EXT_SUFFIXI'm not sure what the relevance of this is;
Anyways: Debian: Again, a match. It would be great if Python used autoconf tuples everywhere, but it clearly doesn't. This is definitely not the only situation where Future Hypotheticals
I'm testing this now, but the build will need to run overnight. If this works, I'm perfectly happy with it being merged instead of my PR. However if it doesn't work, and this PR is the only known fix that gets PyQt (and everything downstream of it) building again, I don't think it's appropriate to blockade it and leave powerpc (only) broken while we wait some indeterminate amount of time for some better solution to materialize. |
|
Thank you for taking the time to look into and explain this further. It looks like I misunderstood how
This is not correct, Fedora did in fact have a patch to change it to Since Debian's Python gives However, it would be nice if you could still wait to find out if my idea to disable manylinux builds works on your system. If they do, then we know we can safely do it if this issue pops up a third time somehow... |
There was a problem hiding this comment.
Logically I would expect this to be ppcle?
There was a problem hiding this comment.
Yes indeed, how embarrassing. I've pushed the fix.
It looks like this worked! I've written it up as #168115; if you'd prefer that one instead it's all the same to me. #168115 is marked as draft right now because I need to do one last rebuild with it to make sure nothing broke; assuming it doesn't I'll un-draft it in the morning. |
Yikes, changing things like this back and forth must have been unpleasant for Fedora users.
I don't think there's really a right or wrong here. Like all other distributions nixpkgs gets to pick its "Python name" for each platform. I was under the mistaken impression that something in
Thank you for pointing this out! It does work! In this case it really doesn't matter to me which name -- This is great! Thanks for pointing this out, I never would've guessed it would work. Assuming the build for #168115 completes without problems (it should) I'll un-draft that PR in the morning and mark this one as draft to indicate that your approach is preferred. |
|
I really think this is the better fix now. It seems weird that the distro gets to pick what I suppose now we can wait for the Nixpkgs Python maintainers to weigh in and see which approach they think is better. But I believe this fixes a genuine bug in Nixpkgs as opposed to simply working around the problem like #168115. |
Okay, sounds good; unmarked as draft. |
|
Squashed to a single commit, no other changes |
The PEP600 standard gives Python's naming scheme for various architectures; it follows the convention which was in use by Fedora in 2014. According to PEP600, the architecture name for Power PC is `ppc64le`, not `powerpc64le`. This is also how python3 declares its "supported wheels" under Debian on PowerPC, as checked with `pip debug --verbose` $ pip debug --verbose | grep powerpc $ pip debug --verbose | grep ppc | head cp39-cp39-manylinux_2_31_ppc64le cp39-cp39-manylinux_2_30_ppc64le cp39-cp39-manylinux_2_29_ppc64le cp39-cp39-manylinux_2_28_ppc64le cp39-cp39-manylinux_2_27_ppc64le cp39-cp39-manylinux_2_26_ppc64le cp39-cp39-manylinux_2_25_ppc64le cp39-cp39-manylinux_2_24_ppc64le cp39-cp39-manylinux_2_23_ppc64le Let's adjust the `pythonHostPlatform` expression in cpython/default.nix to pass the architecture using the naming scheme Python expects. Verified on a Raptor Computing Systems Talos II. Without this commit, PyQt5 fails to build, failing with "unsupported wheel". With this commit, it builds successfully.
SuperSandro2000
left a comment
There was a problem hiding this comment.
@jonringer @FRidh do you agree with this?
|
Ping |
|
@FRidh @jonringer any comment on this? Otherwise I would go ahead and merge it since it can't cause regressions for our main platforms. |
|
Should we also change _PYTHON_SYSCONFIGDATA_NAME? It looks like that still contains "powerpc64le". |
Unfotunately Python uses its own taxonomy ("ppc64le") in some places and the autoconf/multiarch taxonomy ("powerpc64le") in other places. It looks like Wheel names definitely use the python taxonomy, however. I think this whole two-taxonomies situation is the reason why the build script for python has both |
Description of changes
The PEP600 standard gives Python's naming scheme for various architectures; it follows the convention which was in use by Fedora in 2014. According to PEP600, the architecture name for Power PC is
ppc64le, notpowerpc64le. This is also how python3 declares its "supported wheels" under Debian on PowerPC, as checked withpip debug --verboseLet's adjust the
pythonHostPlatformexpression in cpython/default.nix to pass the architecture using the naming scheme Python expects.Verified on a Raptor Computing Systems Talos II.
Without this commit, PyQt5 fails to build, failing with "unsupported wheel". With this commit, it builds successfully. Currently building qutebrowser for additional verification.
CC: @r-burns @CrystalGamma
Things done
Built PyQt5 successfully with this. Currently building qutebrowser.
nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)