Skip to content

Fix hooks handling for unpickled nnmodule#96224

Closed
wconstab wants to merge 1 commit intogh/wconstab/125/basefrom
gh/wconstab/125/head
Closed

Fix hooks handling for unpickled nnmodule#96224
wconstab wants to merge 1 commit intogh/wconstab/125/basefrom
gh/wconstab/125/head

Conversation

@wconstab
Copy link
Copy Markdown
Contributor

@wconstab wconstab commented Mar 7, 2023

wconstab added a commit that referenced this pull request Mar 7, 2023
ghstack-source-id: f0700a3
Pull Request resolved: #96224
@pytorch-bot
Copy link
Copy Markdown

pytorch-bot bot commented Mar 7, 2023

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/96224

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit c66a5b4:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Copy link
Copy Markdown
Collaborator

@albanD albanD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@wconstab
Copy link
Copy Markdown
Contributor Author

wconstab commented Mar 8, 2023

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Mar 8, 2023
@wconstab
Copy link
Copy Markdown
Contributor Author

wconstab commented Mar 8, 2023

@albanD we can continue the discussion on the other PR (revert: #96246) but i figured we might as well get this landed asap just in case

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

Merge failed

Reason: This PR needs a label
If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Details for Dev Infra team Raised by workflow job

@wconstab wconstab added the topic: not user facing topic category label Mar 8, 2023
@wconstab
Copy link
Copy Markdown
Contributor Author

wconstab commented Mar 8, 2023

@pytorchbot merge

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@ezyang
Copy link
Copy Markdown
Contributor

ezyang commented Mar 8, 2023

This appears to have regressed opacus_cifar

2023-03-08T06:16:41.9985721Z ERROR:common:backend='inductor' raised:
2023-03-08T06:16:41.9986248Z AttributeError: 'Conv2d' object has no attribute '_backward_hooks'
2023-03-08T06:16:41.9986607Z 
2023-03-08T06:16:41.9986615Z 
2023-03-08T06:16:41.9986874Z You can suppress this exception and fall back to eager by setting:
2023-03-08T06:16:41.9987253Z     torch._dynamo.config.suppress_errors = True
2023-03-08T06:16:41.9987479Z Traceback (most recent call last):
2023-03-08T06:16:41.9987768Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1321, in check_accuracy
2023-03-08T06:16:41.9988090Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-03-08T06:16:41.9988673Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 254, in _fn
2023-03-08T06:16:41.9989133Z     return fn(*args, **kwargs)
2023-03-08T06:16:41.9989593Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1187, in run_n_iterations
2023-03-08T06:16:41.9990077Z     self.model_iter_fn(mod, inputs, collect_outputs=False)
2023-03-08T06:16:41.9990465Z   File "benchmarks/dynamo/torchbench.py", line 364, in forward_pass
2023-03-08T06:16:41.9990874Z     return mod(*inputs)
2023-03-08T06:16:41.9991479Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl
2023-03-08T06:16:41.9991759Z     return forward_call(*args, **kwargs)
2023-03-08T06:16:41.9992151Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 391, in catch_errors
2023-03-08T06:16:41.9992454Z     return callback(frame, cache_size, hooks)
2023-03-08T06:16:41.9992859Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 406, in _convert_frame
2023-03-08T06:16:41.9993166Z     result = inner_convert(frame, cache_size, hooks)
2023-03-08T06:16:41.9993662Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 105, in _fn
2023-03-08T06:16:41.9993937Z     return fn(*args, **kwargs)
2023-03-08T06:16:41.9994319Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 263, in _convert_frame_assert
2023-03-08T06:16:41.9994603Z     return _compile(
2023-03-08T06:16:41.9994963Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper
2023-03-08T06:16:41.9995232Z     r = func(*args, **kwargs)
2023-03-08T06:16:41.9995656Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 326, in _compile
2023-03-08T06:16:41.9995950Z     out_code = transform_code_object(code, transform)
2023-03-08T06:16:41.9996399Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 530, in transform_code_object
2023-03-08T06:16:41.9996881Z     transformations(instructions, code_options)
2023-03-08T06:16:41.9997275Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 313, in transform
2023-03-08T06:16:41.9997545Z     tracer.run()
2023-03-08T06:16:41.9997911Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1840, in run
2023-03-08T06:16:41.9998177Z     super().run()
2023-03-08T06:16:41.9998530Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 597, in run
2023-03-08T06:16:41.9998801Z     and self.step()
2023-03-08T06:16:41.9999171Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 560, in step
2023-03-08T06:16:41.9999446Z     getattr(self, inst.opname)(inst)
2023-03-08T06:16:41.9999974Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1919, in RETURN_VALUE
2023-03-08T06:16:42.0000279Z     self.output.compile_subgraph(
2023-03-08T06:16:42.0000821Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 545, in compile_subgraph
2023-03-08T06:16:42.0001153Z     self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
2023-03-08T06:16:42.0001678Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 615, in compile_and_call_fx_graph
2023-03-08T06:16:42.0001995Z     compiled_fn = self.call_user_compiler(gm)
2023-03-08T06:16:42.0002369Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 164, in time_wrapper
2023-03-08T06:16:42.0002641Z     r = func(*args, **kwargs)
2023-03-08T06:16:42.0003030Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 701, in call_user_compiler
2023-03-08T06:16:42.0003377Z     raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
2023-03-08T06:16:42.0003813Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 697, in call_user_compiler
2023-03-08T06:16:42.0004143Z     compiled_fn = compiler_fn(gm, self.fake_example_inputs())
2023-03-08T06:16:42.0004561Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/debug_utils.py", line 1064, in debug_wrapper
2023-03-08T06:16:42.0004853Z     compiled_gm = compiler_fn(gm, example_inputs)
2023-03-08T06:16:42.0005261Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor
2023-03-08T06:16:42.0005551Z     return compile_fx(*args, **kwargs)
2023-03-08T06:16:42.0005942Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_inductor/compile_fx.py", line 429, in compile_fx
2023-03-08T06:16:42.0006243Z     model_ = overrides.fuse_fx(model_, example_inputs_)
2023-03-08T06:16:42.0006647Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_inductor/overrides.py", line 97, in fuse_fx
2023-03-08T06:16:42.0006944Z     gm = mkldnn_fuse_fx(gm, example_inputs)
2023-03-08T06:16:42.0007325Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_inductor/mkldnn.py", line 516, in mkldnn_fuse_fx
2023-03-08T06:16:42.0007601Z     gm = pack_module(gm)
2023-03-08T06:16:42.0007972Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_inductor/mkldnn.py", line 743, in pack_module
2023-03-08T06:16:42.0008291Z     new_module = computation_op_packed_map[type(cur_module)](
2023-03-08T06:16:42.0008694Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_inductor/mkldnn.py", line 415, in packed_conv_eval
2023-03-08T06:16:42.0008971Z     return ConvUnary2d(
2023-03-08T06:16:42.0009338Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_inductor/mkldnn.py", line 116, in __init__
2023-03-08T06:16:42.0009633Z     self._update_module_params(conv, unary, input_size)
2023-03-08T06:16:42.0010054Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_inductor/mkldnn.py", line 119, in _update_module_params
2023-03-08T06:16:42.0010435Z     self.__dict__ = copy.deepcopy(conv.__dict__)
2023-03-08T06:16:42.0010707Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 146, in deepcopy
2023-03-08T06:16:42.0010930Z     y = copier(x, memo)
2023-03-08T06:16:42.0011182Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 230, in _deepcopy_dict
2023-03-08T06:16:42.0011460Z     y[deepcopy(key, memo)] = deepcopy(value, memo)
2023-03-08T06:16:42.0011719Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 172, in deepcopy
2023-03-08T06:16:42.0011966Z     y = _reconstruct(x, memo, *rv)
2023-03-08T06:16:42.0012225Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 296, in _reconstruct
2023-03-08T06:16:42.0012464Z     value = deepcopy(value, memo)
2023-03-08T06:16:42.0012718Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 172, in deepcopy
2023-03-08T06:16:42.0012962Z     y = _reconstruct(x, memo, *rv)
2023-03-08T06:16:42.0013464Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 264, in _reconstruct
2023-03-08T06:16:42.0013687Z     y = func(*args)
2023-03-08T06:16:42.0013925Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 263, in <genexpr>
2023-03-08T06:16:42.0014189Z     args = (deepcopy(arg, memo) for arg in args)
2023-03-08T06:16:42.0014442Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 146, in deepcopy
2023-03-08T06:16:42.0014675Z     y = copier(x, memo)
2023-03-08T06:16:42.0014930Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 237, in _deepcopy_method
2023-03-08T06:16:42.0015200Z     return type(x)(x.__func__, deepcopy(x.__self__, memo))
2023-03-08T06:16:42.0015477Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 172, in deepcopy
2023-03-08T06:16:42.0015723Z     y = _reconstruct(x, memo, *rv)
2023-03-08T06:16:42.0015962Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 270, in _reconstruct
2023-03-08T06:16:42.0016213Z     state = deepcopy(state, memo)
2023-03-08T06:16:42.0016470Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 146, in deepcopy
2023-03-08T06:16:42.0016702Z     y = copier(x, memo)
2023-03-08T06:16:42.0016938Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 230, in _deepcopy_dict
2023-03-08T06:16:42.0017216Z     y[deepcopy(key, memo)] = deepcopy(value, memo)
2023-03-08T06:16:42.0017486Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 172, in deepcopy
2023-03-08T06:16:42.0017720Z     y = _reconstruct(x, memo, *rv)
2023-03-08T06:16:42.0017974Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 296, in _reconstruct
2023-03-08T06:16:42.0018224Z     value = deepcopy(value, memo)
2023-03-08T06:16:42.0018463Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 172, in deepcopy
2023-03-08T06:16:42.0018763Z     y = _reconstruct(x, memo, *rv)
2023-03-08T06:16:42.0019023Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 270, in _reconstruct
2023-03-08T06:16:42.0019279Z     state = deepcopy(state, memo)
2023-03-08T06:16:42.0019516Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 146, in deepcopy
2023-03-08T06:16:42.0019746Z     y = copier(x, memo)
2023-03-08T06:16:42.0019996Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 230, in _deepcopy_dict
2023-03-08T06:16:42.0020254Z     y[deepcopy(key, memo)] = deepcopy(value, memo)
2023-03-08T06:16:42.0020526Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 172, in deepcopy
2023-03-08T06:16:42.0020771Z     y = _reconstruct(x, memo, *rv)
2023-03-08T06:16:42.0021012Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 296, in _reconstruct
2023-03-08T06:16:42.0021262Z     value = deepcopy(value, memo)
2023-03-08T06:16:42.0021515Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 172, in deepcopy
2023-03-08T06:16:42.0021744Z     y = _reconstruct(x, memo, *rv)
2023-03-08T06:16:42.0021999Z   File "/opt/conda/envs/py_3.8/lib/python3.8/copy.py", line 272, in _reconstruct
2023-03-08T06:16:42.0022290Z     y.__setstate__(state)
2023-03-08T06:16:42.0022690Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 175, in __setstate__
2023-03-08T06:16:42.0022957Z     super().__setstate__(state)
2023-03-08T06:16:42.0023352Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/mutation_guard.py", line 112, in patched_setstate
2023-03-08T06:16:42.0023636Z     setstate(self, state)
2023-03-08T06:16:42.0024001Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1634, in __setstate__
2023-03-08T06:16:42.0024276Z     self._update_has_hooks()
2023-03-08T06:16:42.0024659Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 499, in _update_has_hooks
2023-03-08T06:16:42.0024935Z     self._backward_hooks
2023-03-08T06:16:42.0025300Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1649, in __getattr__
2023-03-08T06:16:42.0026028Z     raise AttributeError("'{}' object has no attribute '{}'".format(
2023-03-08T06:16:42.0026586Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2023-03-08T06:16:42.0027068Z AttributeError: 'Conv2d' object has no attribute '_backward_hooks'
2023-03-08T06:16:42.0027295Z 
2023-03-08T06:16:42.0027301Z 
2023-03-08T06:16:42.0027480Z You can suppress this exception and fall back to eager by setting:
2023-03-08T06:16:42.0027844Z     torch._dynamo.config.suppress_errors = True
2023-03-08T06:16:42.0028040Z 
2023-03-08T06:16:42.0028229Z TorchDynamo optimized model failed to run because of following error
2023-03-08T06:16:42.0028526Z FAIL

@ezyang
Copy link
Copy Markdown
Contributor

ezyang commented Mar 8, 2023

@pytorchbot revert -c nosignal "inductor regression"

@pytorch-bot
Copy link
Copy Markdown

pytorch-bot bot commented Mar 8, 2023

❌ 🤖 pytorchbot command failed:

@pytorchbot revert: error: the following arguments are required: -m/--message

usage: @pytorchbot revert -m MESSAGE -c
                          {nosignal,ignoredsignal,landrace,weird,ghfirst}

Try @pytorchbot --help for more info.

@ezyang
Copy link
Copy Markdown
Contributor

ezyang commented Mar 8, 2023

@pytorchbot revert -m "inductor regression" -c nosignal

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

@wconstab your PR has been successfully reverted.

pytorchmergebot added a commit that referenced this pull request Mar 8, 2023
This reverts commit 8ca264e.

Reverted #96224 on behalf of https://github.com/ezyang due to inductor regression
wconstab added a commit that referenced this pull request Mar 8, 2023
…(try 2)"


Ensure _update_has_hooks() will work regardless of which
hook dicts exist on the module.

Second try after revert of #96224 where unpickled module didn't have _backward_hooks field.

[ghstack-poisoned]
wconstab added a commit that referenced this pull request Mar 8, 2023
Ensure _update_has_hooks() will work regardless of which
hook dicts exist on the module.

Second try after revert of #96224 where unpickled module didn't have _backward_hooks field.

[ghstack-poisoned]
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Mar 12, 2023
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Mar 12, 2023
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Mar 12, 2023
cyyever pushed a commit to cyyever/pytorch_private that referenced this pull request Mar 12, 2023
ydwu4 added a commit to ydwu4/pytorch that referenced this pull request Mar 13, 2023
ydwu4 added a commit to ydwu4/pytorch that referenced this pull request Mar 13, 2023
@facebook-github-bot facebook-github-bot deleted the gh/wconstab/125/head branch June 8, 2023 19:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/trunk Trigger trunk jobs on your pull request Merged Reverted topic: not user facing topic category

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants