We're currently facing a bug with users having newer installations of Python (3.9).

We are depending on the reproducibility of Container Images and Digests, and image builds on different OS & Platforms should always result in the same image digest (note that we're always building images for linux / amd64).

However, we started noticing an issue with people that upgraded their Python 3 version to 3.9.
The digests seem to differ to older versions.

We noticed this on two different systems:

On MacOS, the system installations of python (/usr/bin) are old enough, for example on Big Sur, python3 is 3.8.2. However, the user may install newer Python versions with brew, which makes the python binary to be 3.9 instead of the system's one.
In this case, /usr/bin/python3 is the system-installation, but python3 selected from the user's $PATH is / may be brew's installation of python 3.9.

On Linux, where the user just had upgraded their python version.

We mitigated this issue, at least for MacOS, by registering custom toolchains:

py_runtime(
    name = "python3",
    interpreter_path = "/usr/bin/python3",
    python_version = "PY3",
)

py_runtime_pair(
    name = "python_pair",
    # we're not registering a python2 installation as:
    # old installations do not have `python2` as a binary
    # new installations have `python` linked to python 3.
    # So we never know which one works.
    #py2_runtime = "",
    py3_runtime = ":python3",
)

On Linux, we require the user to install a python version pre-3.9 and link it to /usr/bin/python3.

There is an issue in rules_python to register the python interpreter similarly to how go_register_toolchain downloads the correct version of the Go compiler; making the build independent of the Host. While this would fix the issue at hand, it would still only be a workaround too.

Is this a bug in rules_docker that the image digests differ with newer versions of python, or is this just an issue from Python itself?