[improve][build] pip with --prefer-binary to reduce image build time #22613
Conversation
docker/pulsar/Dockerfile
Outdated
| pulsar-client[all]==${PULSAR_CLIENT_PYTHON_VERSION} | ||
|
|
||
| pulsar-client[all]==${PULSAR_CLIENT_PYTHON_VERSION} \ | ||
| --prefer-binary --find-links https://wheel-index.linuxserver.io/alpine-3.19/ |
There was a problem hiding this comment.
do we want to trust linuxserver.io for binaries?
There was a problem hiding this comment.
Sure, it provides the grpcio package.
There was a problem hiding this comment.
We also download this wheel file to local.
There was a problem hiding this comment.
We also download this wheel file to local.
where and why? I couldn't find any existing references of linuxserver.io .
There was a problem hiding this comment.
From https://wheels.linuxserver.io/alpine-3.19/grpcio-1.62.2-cp311-cp311-linux_aarch64.whl to the pulsar repo.
There was a problem hiding this comment.
From https://wheels.linuxserver.io/alpine-3.19/grpcio-1.62.2-cp311-cp311-linux_aarch64.whl to the pulsar repo.
how does that get referenced? does the Alpine image already reference that?
There was a problem hiding this comment.
This solution comes from grpc/grpc#27512 (comment)
Non Alpine official solution.
There was a problem hiding this comment.
This solution comes from grpc/grpc#27512 (comment)
Non Alpine official solution.
What makes this solution "official"?
I think a better solution is to lower the required grpc version in pulsar-client-python to 1.59.3 so that no external solution wouldn't be required at all.
The grpc version should match apache-bookkeeper-client's grpc version to ensure compatibility of generated stubs.
| musl-dev \ | ||
| libffi-dev \ | ||
| py3-pip \ | ||
| py3-grpcio \ |
There was a problem hiding this comment.
since py3-grpcio is already installed, why isn't that used? isn't there a way to prevent upgrading of the grpcio package?
There was a problem hiding this comment.
I'm not good at pip, and as far as I know there's no way. The pip will uninstall the installed, and then to install a new grpcio package by the pulsar-client.
There was a problem hiding this comment.
It looks like this is causing the problem: apache/pulsar-client-python@162afd5 .
In Alpine, the grpcio version is 1.59.3 , https://pkgs.alpinelinux.org/package/v3.19/community/aarch64/py3-grpcio
@BewareMyPower Is there a way to allow using grpcio version 1.59.3 ?
There was a problem hiding this comment.
The operating system only requires the basic package, and we can use pip to install other dependencies, which are always the latest.
apk sometimes provides outdated packages.
The pip will uninstall the installed, and then to install a new grpcio package by the pulsar-client.
You can use the RUN pip3 install --break-system-packages pulsar-client[all]==${PULSAR_CLIENT_PYTHON_VERSION} --verbose --prefer-binary to reproduce this issue in the Dockerfile.
There was a problem hiding this comment.
apk sometimes provides outdated packages.
I explained the root cause in the previous comment: #22613 (comment) .
Alpine comes with grpc 1.59.3 package. I don't see a reason why that couldn't be supported.
lhotari
left a comment
There was a problem hiding this comment.
I have created apache/pulsar-client-python#211 as a way to mitigate the root cause.
02a351d to
93afd89
Compare
|
Forcing push resulted in this PR being closed. Once apache/pulsar-client-python#211 is merged, we can use the One issue is that PyPI doesn't provide the ratelimit wheel file, we can build this wheel, and upload it to the pulsar repo, this size is 6kb. |
Motivation
When building the ARM image, pip3 will take a long time to build the grpcio package, because the PyPI doesn't provide the grpcio aarch64 wheel file, this is unfriendly.
Build script:
Log(Improved):
And then we can add a job to build the ARM image.
Modifications
py3-grpcioby thepulsar-client--prefer-binaryand wheel mirror to speed up installationDocumentation
docdoc-requireddoc-not-neededdoc-complete