Error using transfer manager: cannot pickle '_cffi_backend.FFI' object #1116
Description
Introduction
I ran into this issue which it had should been fixed in google-cloud-storage version 2.9.0, however it is still happening in 2.10.0.
Check this issue for reference: #1044
The error is gone if I remove the cryptography dependency, but unfortunately it is present in my target environment, and even if I update it to the latest version the error is still present: also mentioning @andrewsg since it seemed that you wanted more info about this issue.
The error was happening in my target environment and I reproduced it using the attached Dockerfile.
Environment details
- OS type and version: Debian GNU/Linux 12 (bookworm)
- Python version: 3.10.12
- pip version: 23.2.1
- google-cloud-storage version: 2.10.0
Steps to reproduce
- Write the contents of this snippet to a file called
Dockerfile
FROM python:3.10.12
WORKDIR /app
RUN python -m pip install pip==23.2.1
RUN pip install \
google-cloud-storage==2.10.0 \
cryptography==3.4.8
RUN openssl genrsa -out key.pem 2048
RUN apt-get update && apt-get install -y jq
RUN jq -n \
--arg type "service_account" \
--arg project_id "ALL_THIS_DATA_IS_FAKE" \
--arg private_key_id "0123456789abcdef0123456789abcdef01234567" \
--arg private_key "$(cat key.pem)" \
--arg client_email "foo@example.iam.gserviceaccount.com" \
--arg client_id "012345678901234567890" \
--arg auth_uri "https://accounts.google.com/o/oauth2/auth" \
--arg token_uri "https://oauth2.googleapis.com/token" \
--arg auth_provider_x509_cert_url "https://www.googleapis.com/oauth2/v1/certs" \
--arg client_x509_cert_url "https://www.googleapis.com/robot/v1/metadata/x509/foo%40example.iam.gserviceaccount.com" \
--arg universe_domain "googleapis.com" \
'$ARGS.named' > credentials.json
RUN echo ' \n\
from google.oauth2 import service_account \n\
from google.cloud import storage \n\
from google.cloud.storage import transfer_manager \n\
credentials = service_account.Credentials.from_service_account_file("credentials.json") \n\
storage_client = storage.Client(credentials=credentials) \n\
bucket = storage_client.bucket("example") \n\
results = transfer_manager.download_many_to_path(bucket, blob_names="file.txt", destination_directory="/tmp", max_workers=8) \n\
' > script.py
CMD ["python", "/app/script.py" ]
- Run
docker run -it $(docker build -q .)from the same directory
Stack trace
/usr/local/lib/python3.10/site-packages/google/cloud/storage/transfer_manager.py:30: UserWarning: The module `transfer_manager` is a preview feature. Functionality and API may change. This warning will be removed in a future release.
warnings.warn(
Traceback (most recent call last):
File "/app/script.py", line 8, in <module>
results = transfer_manager.download_many_to_path(bucket, blob_names="file.txt", destination_directory="/tmp", max_workers=8)
File "/usr/local/lib/python3.10/site-packages/google/cloud/storage/transfer_manager.py", line 69, in convert_threads_or_raise
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/google/cloud/storage/transfer_manager.py", line 703, in download_many_to_path
return download_many(
File "/usr/local/lib/python3.10/site-packages/google/cloud/storage/transfer_manager.py", line 69, in convert_threads_or_raise
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/google/cloud/storage/transfer_manager.py", line 346, in download_many
_pickle_blob(blob) if needs_pickling else blob,
File "/usr/local/lib/python3.10/site-packages/google/cloud/storage/transfer_manager.py", line 910, in _pickle_blob
p.dump(blob)
TypeError: cannot pickle '_cffi_backend.FFI' object
Let me know if you need more info! Thanks in advance :)