This repository was archived by the owner on Mar 6, 2026. It is now read-only.
This repository was archived by the owner on Mar 6, 2026. It is now read-only.
Breaking change when upgraded from 2.23.3 to 2.27.0 #1478
Description
Error description
We use luigi in our project and after recent update of our python dependencies we started to receive following errors.
/usr/local/lib/python3.10/site-packages/luigi/worker.py 701 Luigi unexpected framework error while scheduling MyTask()
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/luigi/worker.py", line 796, in add
current = queue.get()
File "<string>", line 2, in get
File "/usr/local/lib/python3.10/multiprocessing/managers.py", line 833, in _callmethod
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Unserializable message: Traceback (most recent call last):
File "/usr/local/lib/python3.10/multiprocessing/managers.py", line 308, in serve_client
send(msg)
File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/local/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
TypeError: cannot pickle '_thread.lock' object
Workaround
I found out that dependency that caused it was google-auth. Bringing it down from 2.27.0 to 2.23.2 while keeping other dependencies updated solved the issue. It looks like this PR may be related to this behaviour.
Background
We use luigi with parallel_scheduling=true option which means multiprocessing.Queue is used underneath and it requires every object in it to be pickle-able.
Environment details
- OS: Debian 12 / Ubuntu 20.04 (same issue on both)
- Python version: 3.10
- pip version: 23.3.1
google-authversion: 2.27.0luigiversion: 3.5.0