Skip to content

Cloudpickle 1.6 doesn't work with Pydantic #10459

@simon-mo

Description

@simon-mo

What is the problem?

Ray version and other system information (Python version, TensorFlow version, OS): Python 3.6

Newest version of cloudpickle doesn't work with Pydantic and there is no clear workaround. (I tried implementing __reduce__ but that still failed)

Pydantic is becoming a core dependencies for Serve and Dashboard.

Reproduction (REQUIRED)

Script:

from pydantic import BaseModel
import ray

class MyModel(BaseModel):
    a: int

# implementing reduce doesn't help
#    def __reduce__(self):
#        return MyModel.parse_raw, (self.json(),)


@ray.remote
class Actor:
    def act(self, model: MyModel):
        return model.a

ray.init()

a = Actor.remote()
model = MyModel(a=42)
ray.get(a.act.remote(model))

Output:

2020-08-31 18:04:58,139	INFO resource_spec.py:250 -- Starting Ray with 21.63 GiB memory available for workers and up to 10.82 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
2020-08-31 18:04:58,628	INFO services.py:1201 -- View the Ray dashboard at 127.0.0.1:8265
Traceback (most recent call last):
  File "a.py", line 14, in <module>
    a = Actor.remote()
  File "/Users/simonmo/Desktop/ray/ray/python/ray/actor.py", line 378, in remote
    return self._remote(args=args, kwargs=kwargs)
  File "/Users/simonmo/Desktop/ray/ray/python/ray/actor.py", line 548, in _remote
    meta.method_meta.methods.keys())
  File "/Users/simonmo/Desktop/ray/ray/python/ray/function_manager.py", line 358, in export_actor_class
    "class": pickle.dumps(Class),
  File "/Users/simonmo/Desktop/ray/ray/python/ray/cloudpickle/cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "/Users/simonmo/Desktop/ray/ray/python/ray/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
  File "/Users/simonmo/Desktop/ray/ray/python/ray/cloudpickle/cloudpickle_fast.py", line 642, in reducer_override
    parametrized_type_hint_getinitargs(obj)
  File "/Users/simonmo/Desktop/ray/ray/python/ray/cloudpickle/cloudpickle.py", line 529, in parametrized_type_hint_getinitargs
    "Cloudpickle Error: Unknown type {}".format(type(obj))
_pickle.PicklingError: Cloudpickle Error: Unknown type <class 'pydantic.main.ModelMetaclass'>
  • I have verified my script runs in a clean environment and reproduces the issue.
  • I have verified the issue also occurs with the latest wheels.

Metadata

Metadata

Labels

P2Important issue, but not time-criticalbugSomething that is supposed to be working; but isn't

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions