Reviewer's Guide by Sourcery

This pull request adds support for relocatable Python installations by introducing a mechanism to detect them and setting the rpath accordingly. It introduces a new isRelocatable function and a relocatable Scons option. It also replaces calls to isUvPython with isPythonBuildStandalonePython.

Sequence diagram for relocatable Python detection

sequenceDiagram
    participant Nuitka
    participant PythonFlavors
    participant sysconfig

    Nuitka->>PythonFlavors: isRelocatable()
    PythonFlavors->>sysconfig: get_config_var("PYTHON_BUILD_STANDALONE")
    sysconfig-->>PythonFlavors: Returns 1 or None
    alt PYTHON_BUILD_STANDALONE == 1
        PythonFlavors-->>Nuitka: True
    else isSelfCompiledPythonUninstalled()
        PythonFlavors->>PythonFlavors: isSelfCompiledPythonUninstalled()
        PythonFlavors-->>Nuitka: Result of isSelfCompiledPythonUninstalled()
    else
        PythonFlavors-->>Nuitka: False
    end

Updated class diagram for Python flavors

classDiagram
    class PythonFlavors {
        <<module>>
        +isRyePython()
        +isPythonBuildStandalonePython()
        +isPyenvPython()
        +isUninstalledPython()
        +isRelocatable()
        +getPythonFlavorName()
    }
    note for PythonFlavors.isRelocatable "New method to detect relocatable Python installations"
    note for PythonFlavors.isPythonBuildStandalonePython "Replaces isUvPython"

File-Level Changes

Assessment against linked issues

Possibly linked issues

• Binaries compiled with a Python installed by uv fails on Ubuntu: libpython3.13.so.1.0 not found #3325: This PR fixes the issue by adding rpath for relocatable Python installations like rye, resolving the missing libpython error.
• Binaries compiled with a Python installed by uv fails on Ubuntu: libpython3.13.so.1.0 not found #3325: The PR fixes the broader problem of using relocatable Python installations, which includes cases like the issue where Python.h is in a non-standard location.

You didn't respond to my ideas about doing it always, and I tend to take that as agreement. I think we understand that for your Python flavor, it is needed for sure, and as such I am now going to rebase it, but making the rpath non-optional, and to only disable it where the toolchain doesn't support that feature at most.

So, this rebased to 2.6.6 without any issues.

And I implemented my idea, renaming the flavor, please let me know if you like it, and inviting you to test it, I didn't quite check it yet. Doing it always should not break UV-Python's use of Standalone Build Python. Adding it as a flavor will make uses outside of uv supported as well I think.

It seems, somehow the rpath added survives on Linux now, where of course it should be removed for standalone to be working properly, I need to check that out.

So, it was only removing the rpath for the main executable on macOS, which may have been too little anyway, need to do it for all platforms now, as it now gets added always, that should repair general Linux standalone. Glad that Github Actions Python detected that with it not using static libpython, in my setups it didn't show.

Oh the joy of it, macOS (maybe Clang?) won't accept -Wl,-rpath while others won't accept -rpath it seems, making a difference there now.

I rebased to current develop and fixed macOS, which has no rpath at all, me being ignorant there.

Rebased it to the current develop branch after the other PRs merge. It's very disturbing that this still fails, despite it claiming to set the rpath properly on CtypesUsing.bin to $ORIGIN and somehow the strace output done on the running program is not complete, since it never mentions said DLL, but the libffi one from hostedtools cache, I will try and reproduce this locally to see where it goes wrong.

So, my suspect is that DT_RPATH is used by GitHub Actions Python, but no say my Debian Python, where i force it to use --static-libpython=no to reproduce it, so adding --remove-rpath according to patchelf man page should remove both and then setting it doesn't have us end up with both, with apparently the DT_RPATH then taking precedence. If that holds true, Nuitka would actually already have been bad at removing these kind of rpaths, which would be an issue by itself.

I am at a complete loss there, it seems the ELF output is perfect:

: ELF listing of main binary:

Dynamic section at offset 0x38a000 contains 30 entries:
  Tag        Type                         Name/Value
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpython3.8.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0xb000
 0x000000000000000d (FINI)               0xb5964
 0x0000000000000019 (INIT_ARRAY)         0x383618
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x383620
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x39b760
 0x0000000000000005 (STRTAB)             0x39a230
 0x0000000000000006 (SYMTAB)             0x660
 0x000000000000000a (STRSZ)              5421 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x383850
 0x0000000000000002 (PLTRELSZ)           5688 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x9600
 0x0000000000000007 (RELA)               0x3d80
 0x0000000000000008 (RELASZ)             22656 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x0000000000000018 (BIND_NOW)           
 0x000000006ffffffb (FLAGS_1)            Flags: NOW PIE
 0x000000006ffffffe (VERNEED)            0x3cc0
 0x000000006fffffff (VERNEEDNUM)         2
 0x000000006ffffff0 (VERSYM)             0x3a2e
 0x000000006ffffff9 (RELACOUNT)          863
 0x0000000000000000 (NULL)               0x0

Yet, the runtime traces taking complains the GitHub Action Python libpython is being loaded, for which there wouldn't be a reason why. And esp. if this wasn't sufficient, I don't see why the old code didn't show that happen, if somehow it was forced through environment or something. In the old code, it just didn't have an rpath, how would that make it check the local directory first, maybe $ORIGIN works differently from what I think it should.

So, this comes down to LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.18/x64/lib with which the binary is getting executed, and according to my testing, this forces the run resolution to not use the DLLs at $ORIGIN but to prefer anything found there, this is of course nothing new. I am deeply confused what causes this to be visible now and I need to investigate it carefully, it's alsmost as if we stopped using static libpython for GitHub Actions Python somehow, or something like that.

So, I think, I am making the executive decision, to minimize the differences of this PR to develop, and skip this for the next hotfix, unless I figure out what's going on through that. Failing tests on Github Actions is not acceptable. Obviously unsetting LD_LIBRARY_PATH before any execution of a standalone binary is kind of a must, for the strace to hold any water, but I would really like to understand why I haven't noticed this issue previously, since it's absolutely presenct on current hotfix branch as well, but strace doesn't load it from there. I tried manually removing the RPATH, to see if that's what causes it, but on my Debian that didn't do anything particular, so I am at a loss what is bug hiding there, or how the strace analysis is not effective previously.

So, I rebased this to factory branch now, to which I picked the changes that are in here, but could be separate, flavor rename, lib_Hacl2_SHA2 issue, and the minor test improvement to display the resulting file.

Now GitHub is just giving me a hard time for the merge of that other PR that now conflicts, I of course put that for the hotfix, so really, I think the best way ahead is to make the hotfix release now and rebase onto the then develop. The diff is really small now, but it seems centered around touching the linker rpath. I am contemplating to add to develop a change where for one standalone binary, it does the output of readelf anyway, so I can see what would be the change, maybe DT_RPATH vs. DT_RUNPATH or something stupid like that, although forcing to to DT_RPATH didn't impact my Debian now, and it would of course never be the real solution, since it wouldn't work everywhere.

So, with this rebased and minimal, still fails. Maybe an explanation why I don't just fix the LD_LIBRARY_PATH issue, but want to know why it's not been observed previously. In ATC I have learned that the most effective way of doing things, is not to fix the symptom you are seeing, but all things that have contributed to seeing it and to not seeing it earlier. So for me, it's super important to be able to trust these standalone tests that try to assert no outside DLLs are used. Something wasn't working there right, or some assumption I am holding is wrong. Current develop does have to fail I think, but it does not.

So, what I will probably add, is like a standalone compilation with artifacts like a report, and the readelf output, and the strace captured on execution, such that I have a reference point to easily compare the situation. We are down to just adding the rpath early, during compilation, and that causing the test to fail, and it being added later, to not do that. But also, it's being removed and replaced entirely, and yet there is a difference. I would like to get to the bottom of that, and Github Actions Python is obviously a relatively difficult environment for that.

So, I was distracted privately with real life Nuitka being in hospital for close to 2 weeks, but I am finally getting to this, and have updated factory with the recording of reports.

For a simple standalone this is what ldd gives without the PR code:

	linux-vdso.so.1 (0x00007ffe96908000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f4549719000)
	libpython3.8.so.1.0 => /home/runner/work/Nuitka/Nuitka/EmptyModuleTest.dist/libpython3.8.so.1.0 (0x00007f4549200000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4548e00000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f4549c25000)

And this is what readelf says:

Dynamic section at offset 0x359630 contains 30 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpython3.8.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN]
 0x000000000000000c (INIT)               0xb000
 0x000000000000000d (FINI)               0x9fe84
 0x0000000000000019 (INIT_ARRAY)         0x35a618
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x35a620
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x3b0
 0x0000000000000005 (STRTAB)             0x24e8
 0x0000000000000006 (SYMTAB)             0x658
 0x000000000000000a (STRSZ)              5316 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x35a850
 0x0000000000000002 (PLTRELSZ)           5688 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x94a0
 0x0000000000000007 (RELA)               0x3cf8
 0x0000000000000008 (RELASZ)             22440 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x0000000000000018 (BIND_NOW)           
 0x000000006ffffffb (FLAGS_1)            Flags: NOW PIE
 0x000000006ffffffe (VERNEED)            0x3c38
 0x000000006fffffff (VERNEEDNUM)         2
 0x000000006ffffff0 (VERSYM)             0x39ac
 0x000000006ffffff9 (RELACOUNT)          855
 0x0000000000000000 (NULL)               0x0

Both are good, but supposedly the PR breaks that, and I don't know how, but lets first confirm that by me rebasing to current factory branch.

So, indeed the regression is this PR branch, and not an old issue, noticed only now, the readelf has a RUNPATH which apparently must be weaker in terms of forcing the directory usage.

 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpython3.8.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

The ldd output then gives this:

	linux-vdso.so.1 (0x00007ffd71cfc000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9dd5ddd000)
	libpython3.8.so.1.0 => /opt/hostedtoolcache/Python/3.8.18/x64/lib/libpython3.8.so.1.0 (0x00007f9dd5600000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9dd5200000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f9dd5ed2000)

And while this affects libpython here, I guess it could affect practially all other DLLs contained.

I thought I had made sure it's a RPATH vs. RUNPATH, but somehow unsuccessfully maybe.

Pretty sure I tried patchelf with --force-rpath in some capacity, however, on my Debian just now it switches it over to be a RPATH like it used to be. Previously, the RPATH nature wasn't enforced, but it seems it's better for isolation by far, and as such is a good finding.

Of course, "Standalone Build Python" ought to not need a dynamic libpython, but rather libpython is not off the table, but I guess that's still a very good improvement to Nuitka. I wasn't aware of there being such a difference between RPATH and RUNPATH with LD_LIBRARY_PATH.

Rebased it back to develop and adapted the commit message, a bunch of fixes has already been evacuated from this PR before, leaving only the always-rpath change in.

I merged this but will put it into 2.6.8 hotfix directly if that works.

The newly released Nuitka 2.6.8 fixed the issue for me! 🎉

See: #3325 (comment)

Part of the 2.7 release