Skip to content

AttributeError: Adam object has no attribute param_groups with Python 3.11 #116707

@gcroci2

Description

@gcroci2

🐛 Describe the bug

Suddenly, some of my package's tests fail on GitHub Actions, but only with Python version 3.11 (not 3.10), and only on Linux (not on Mac OS M1). In particular, they fail when I try to load a state dict pickled with torch.save:

data = torch.load('<saved_dict.pth.tar>', map_location=torch.device('cpu'))

The two problematic files are here: https://github.com/DeepRank/deeprank2/tree/dev/tests/data/pretrained
Here is the failed action: https://github.com/DeepRank/deeprank2/actions/runs/7400017804/job/20132736281?pr=515, from which you can also see how the same tests do not fail with Python 3.10.

Note that it may be something similar to issue #80831

Error message:

tests/test_trainer.py:686: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/share/miniconda/envs/__setup_conda/lib/python3.11/site-packages/deeprank2/dataset.py:787: in __init__
    self._check_and_inherit_train(GraphDataset, self.inherited_params)
/usr/share/miniconda/envs/__setup_conda/lib/python3.11/site-packages/deeprank2/dataset.py:93: in _check_and_inherit_train
    data = torch.load(self.train_source, map_location=torch.device('cpu'))
/usr/share/miniconda/envs/__setup_conda/lib/python3.11/site-packages/torch/serialization.py:809: in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

zip_file = <torch.PyTorchFileReader object at 0x7f80062597b0>
map_location = device(type='cpu')
pickle_module = <module 'pickle' from '/usr/share/miniconda/envs/__setup_conda/lib/python3.11/pickle.py'>
pickle_file = 'data.pkl', pickle_load_args = {'encoding': 'utf-8'}
persistent_load = <function _load.<locals>.persistent_load at 0x7f801ddf2fc0>
UnpicklerWrapper = <class 'torch.serialization._load.<locals>.UnpicklerWrapper'>
data_file = <_io.BytesIO object at 0x7f800510e0c0>
unpickler = <torch.serialization._load.<locals>.UnpicklerWrapper object at 0x7f80063e07d0>

    def _load(zip_file, map_location, pickle_module, pickle_file='data.pkl', **pickle_load_args):
        restore_location = _get_restore_location(map_location)
    
        loaded_storages = {}
    
        def load_tensor(dtype, numel, key, location):
            name = f'data/{key}'
    
            storage = zip_file.get_storage_from_record(name, numel, torch.UntypedStorage)._typed_storage()._untyped_storage
            # TODO: Once we decide to break serialization FC, we can
            # stop wrapping with TypedStorage
            typed_storage = torch.storage.TypedStorage(
                wrap_storage=restore_location(storage, location),
                dtype=dtype,
                _internal=True)
    
            if typed_storage._data_ptr() != 0:
                loaded_storages[key] = typed_storage
    
            return typed_storage
    
        def persistent_load(saved_id):
            assert isinstance(saved_id, tuple)
            typename = _maybe_decode_ascii(saved_id[0])
            data = saved_id[1:]
    
            assert typename == 'storage', \
                f"Unknown typename for persistent_load, expected 'storage' but got '{typename}'"
            storage_type, key, location, numel = data
            if storage_type is torch.UntypedStorage:
                dtype = torch.uint8
            else:
                dtype = storage_type.dtype
    
            if key in loaded_storages:
                typed_storage = loaded_storages[key]
            else:
                nbytes = numel * torch._utils._element_size(dtype)
                typed_storage = load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
    
            return typed_storage
    
        load_module_mapping: Dict[str, str] = {
            # See https://github.com/pytorch/pytorch/pull/51633
            'torch.tensor': 'torch._tensor'
        }
    
        # Need to subclass Unpickler instead of directly monkey-patching the find_class method
        # because it's marked readonly in pickle.
        # The type: ignore is because mypy can't statically determine the type of this class.
        class UnpicklerWrapper(pickle_module.Unpickler):  # type: ignore[name-defined]
            # from https://stackoverflow.com/questions/133[984](https://github.com/DeepRank/deeprank2/actions/runs/7400017804/job/20132736281?pr=515#step:4:985)62/unpickling-python-objects-with-a-changed-module-path/13405732
            # Lets us override the imports that pickle uses when unpickling an object.
            # This is useful for maintaining BC if we change a module path that tensor instantiation relies on.
            def find_class(self, mod_name, name):
                if type(name) is str and 'Storage' in name:
                    try:
                        return StorageType(name)
                    except KeyError:
                        pass
                mod_name = load_module_mapping.get(mod_name, mod_name)
                return super().find_class(mod_name, name)
    
        # Load the data (which may in turn use `persistent_load` to load tensors)
        data_file = io.BytesIO(zip_file.get_record(pickle_file))
    
        unpickler = UnpicklerWrapper(data_file, **pickle_load_args)
        unpickler.persistent_load = persistent_load
>       result = unpickler.load()
E       AttributeError: 'Adam' object has no attribute 'param_groups'

/usr/share/miniconda/envs/__setup_conda/lib/python3.11/site-packages/torch/serialization.py:1172: AttributeError

Versions

PyTorch version: 2.0.1
Is debug build: False

OS: Ubuntu 22.04.3, LTS
Image version: 20231217.2.0
Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20231217.2/images/ubuntu/Ubuntu2204-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20231217.2

Conda version: 23.11.0
Python version: 3.11.5
Is CUDA available: False

Versions of relevant libraries:
[conda] pytorch 2.0.1 py3.11_cpu_0 pytorch
[conda] pytorch-mutex 1.0 cpu pytorch
[conda] torchaudio 2.0.2 py311_cpu pytorch
[conda] torchvision 0.15.2 py311_cpu pytorch

cc @vincentqb @jbschlosser @albanD @janeyx99 @crcrpar @mruberry @mikaylagawarecki

Metadata

Metadata

Assignees

No one assigned

    Labels

    module: optimizerRelated to torch.optimmodule: serializationIssues related to serialization (e.g., via pickle, or otherwise) of PyTorch objectstriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions