Skip to content

Serialization of ufuncs breaks depending on import order #392

@renatolfc

Description

@renatolfc

While investigating why I was unable to load_session when I had some ufuncs from scipy.special, I noticed that dill fails to determine the module the function belongs to depending on the order of the imports:

On a fresh interpreter session (Python 3.8.6), if you load dill prior to loading scipy.special, you get:

>>> import dill
>>> from scipy.special import gdtrix
>>> dill.dumps(gdtrix)
b'\x80\x04\x95\x1a\x00\x00\x00\x00\x00\x00\x00\x8c\x0b__mp_main__\x94\x8c\x06gdtrix\x94\x93\x94.'

Notice the incorrect __mp_main__.

Now, if you invert the ordering of the imports:

>>> from scipy.special import gdtrix
>>> import dill
>>> dill.dumps(gdtrix)
b'\x80\x04\x95$\x00\x00\x00\x00\x00\x00\x00\x8c\x15scipy.special._ufuncs\x94\x8c\x06gdtrix\x94\x93\x94.'

Oddly enough, if we call pickle.whichmodule, it finds the correct module irrespective of import order:

>>> import pickle
>>> from scipy.special import gdtrix
>>> pickle.whichmodule(gdtrix, gdtrix.__name__)
'scipy.special._ufuncs'

Or:

>>> from scipy.special import gdtrix
>>> import pickle
>>> pickle.whichmodule(gdtrix, gdtrix.__name__)
'scipy.special._ufuncs'

The only way I can think of getting an incorrect module is to pass name=None to whichmodule:

>>> pickle.whichmodule(gdtrix, None)
'__main__'

Which might indicate that some cache is initialized when the module is loaded and it somehow fails to update for some types?

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions