-
Notifications
You must be signed in to change notification settings - Fork 7.4k
Cloudpickle 1.6 doesn't work with Pydantic #10459
Copy link
Copy link
Closed
Labels
P2Important issue, but not time-criticalImportant issue, but not time-criticalbugSomething that is supposed to be working; but isn'tSomething that is supposed to be working; but isn't
Description
What is the problem?
Ray version and other system information (Python version, TensorFlow version, OS): Python 3.6
Newest version of cloudpickle doesn't work with Pydantic and there is no clear workaround. (I tried implementing __reduce__ but that still failed)
Pydantic is becoming a core dependencies for Serve and Dashboard.
Reproduction (REQUIRED)
Script:
from pydantic import BaseModel
import ray
class MyModel(BaseModel):
a: int
# implementing reduce doesn't help
# def __reduce__(self):
# return MyModel.parse_raw, (self.json(),)
@ray.remote
class Actor:
def act(self, model: MyModel):
return model.a
ray.init()
a = Actor.remote()
model = MyModel(a=42)
ray.get(a.act.remote(model))
Output:
2020-08-31 18:04:58,139 INFO resource_spec.py:250 -- Starting Ray with 21.63 GiB memory available for workers and up to 10.82 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
2020-08-31 18:04:58,628 INFO services.py:1201 -- View the Ray dashboard at 127.0.0.1:8265
Traceback (most recent call last):
File "a.py", line 14, in <module>
a = Actor.remote()
File "/Users/simonmo/Desktop/ray/ray/python/ray/actor.py", line 378, in remote
return self._remote(args=args, kwargs=kwargs)
File "/Users/simonmo/Desktop/ray/ray/python/ray/actor.py", line 548, in _remote
meta.method_meta.methods.keys())
File "/Users/simonmo/Desktop/ray/ray/python/ray/function_manager.py", line 358, in export_actor_class
"class": pickle.dumps(Class),
File "/Users/simonmo/Desktop/ray/ray/python/ray/cloudpickle/cloudpickle_fast.py", line 73, in dumps
cp.dump(obj)
File "/Users/simonmo/Desktop/ray/ray/python/ray/cloudpickle/cloudpickle_fast.py", line 563, in dump
return Pickler.dump(self, obj)
File "/Users/simonmo/Desktop/ray/ray/python/ray/cloudpickle/cloudpickle_fast.py", line 642, in reducer_override
parametrized_type_hint_getinitargs(obj)
File "/Users/simonmo/Desktop/ray/ray/python/ray/cloudpickle/cloudpickle.py", line 529, in parametrized_type_hint_getinitargs
"Cloudpickle Error: Unknown type {}".format(type(obj))
_pickle.PicklingError: Cloudpickle Error: Unknown type <class 'pydantic.main.ModelMetaclass'>
- I have verified my script runs in a clean environment and reproduces the issue.
- I have verified the issue also occurs with the latest wheels.
Reactions are currently unavailable
Metadata
Metadata
Labels
P2Important issue, but not time-criticalImportant issue, but not time-criticalbugSomething that is supposed to be working; but isn'tSomething that is supposed to be working; but isn't