[jit] clean up exported source format#26788
Merged
zdevito merged 19 commits intogh/zdevito/118/basefrom Oct 15, 2019
Merged
Conversation
The previous PR in the stack removed the need to order classes/functions or have correct import statements. This resolved circular depedency issues that can arise when class constructors like ModuleList put new instances of themselves in a common namespace. This PR changes our export format to no longer produce this information. By doing so we can make the logic signficantly simpler, since we just keep track of an individual PythonPrint object per file. Notes: * PythonPrint was changed to manage its own stream/list of ranges. It was doing this anyway internally, this just makes the API more clear. * Since we are changing the serialization format, I also removed op_version_set. It is now replaced with the VERSION number that written in the zip archive. This further simplifies the code emission process. * A test of op_version_set was removed since there is no longer any behavior to test.
This was referenced Sep 25, 2019
[jit] clean up exported source format The previous PR in the stack removed the need to order classes/functions or have correct import statements. This resolved circular depedency issues that can arise when class constructors like ModuleList put new instances of themselves in a common namespace. This PR changes our export format to no longer produce this information. By doing so we can make the logic signficantly simpler, since we just keep track of an individual PythonPrint object per file. Notes: * PythonPrint was changed to manage its own stream/list of ranges. It was doing this anyway internally, this just makes the API more clear. * Since we are changing the serialization format, I also removed op_version_set. It is now replaced with the VERSION number that written in the zip archive. This further simplifies the code emission process. * A test of op_version_set was removed since there is no longer any behavior to test. gh-metadata: pytorch pytorch 26788 gh/zdevito/118/head
[jit] clean up exported source format The previous PR in the stack removed the need to order classes/functions or have correct import statements. This resolved circular depedency issues that can arise when class constructors like ModuleList put new instances of themselves in a common namespace. This PR changes our export format to no longer produce this information. By doing so we can make the logic signficantly simpler, since we just keep track of an individual PythonPrint object per file. Notes: * PythonPrint was changed to manage its own stream/list of ranges. It was doing this anyway internally, this just makes the API more clear. * Since we are changing the serialization format, I also removed op_version_set. It is now replaced with the VERSION number that written in the zip archive. This further simplifies the code emission process. * A test of op_version_set was removed since there is no longer any behavior to test. gh-metadata: pytorch pytorch 26788 gh/zdevito/118/head
[jit] clean up exported source format The previous PR in the stack removed the need to order classes/functions or have correct import statements. This resolved circular depedency issues that can arise when class constructors like ModuleList put new instances of themselves in a common namespace. This PR changes our export format to no longer produce this information. By doing so we can make the logic signficantly simpler, since we just keep track of an individual PythonPrint object per file. Notes: * PythonPrint was changed to manage its own stream/list of ranges. It was doing this anyway internally, this just makes the API more clear. * Since we are changing the serialization format, I also removed op_version_set. It is now replaced with the VERSION number that written in the zip archive. This further simplifies the code emission process. * A test of op_version_set was removed since there is no longer any behavior to test. gh-metadata: pytorch pytorch 26788 gh/zdevito/118/head
[jit] clean up exported source format The previous PR in the stack removed the need to order classes/functions or have correct import statements. This resolved circular depedency issues that can arise when class constructors like ModuleList put new instances of themselves in a common namespace. This PR changes our export format to no longer produce this information. By doing so we can make the logic signficantly simpler, since we just keep track of an individual PythonPrint object per file. Notes: * PythonPrint was changed to manage its own stream/list of ranges. It was doing this anyway internally, this just makes the API more clear. * Since we are changing the serialization format, I also removed op_version_set. It is now replaced with the VERSION number that written in the zip archive. This further simplifies the code emission process. * A test of op_version_set was removed since there is no longer any behavior to test. gh-metadata: pytorch pytorch 26788 gh/zdevito/118/head
[jit] clean up exported source format The previous PR in the stack removed the need to order classes/functions or have correct import statements. This resolved circular depedency issues that can arise when class constructors like ModuleList put new instances of themselves in a common namespace. This PR changes our export format to no longer produce this information. By doing so we can make the logic signficantly simpler, since we just keep track of an individual PythonPrint object per file. Notes: * PythonPrint was changed to manage its own stream/list of ranges. It was doing this anyway internally, this just makes the API more clear. * Since we are changing the serialization format, I also removed op_version_set. It is now replaced with the VERSION number that written in the zip archive. This further simplifies the code emission process. * A test of op_version_set was removed since there is no longer any behavior to test. gh-metadata: pytorch pytorch 26788 gh/zdevito/118/head
[jit] clean up exported source format The previous PR in the stack removed the need to order classes/functions or have correct import statements. This resolved circular depedency issues that can arise when class constructors like ModuleList put new instances of themselves in a common namespace. This PR changes our export format to no longer produce this information. By doing so we can make the logic signficantly simpler, since we just keep track of an individual PythonPrint object per file. Notes: * PythonPrint was changed to manage its own stream/list of ranges. It was doing this anyway internally, this just makes the API more clear. * Since we are changing the serialization format, I also removed op_version_set. It is now replaced with the VERSION number that written in the zip archive. This further simplifies the code emission process. * A test of op_version_set was removed since there is no longer any behavior to test. gh-metadata: pytorch pytorch 26788 gh/zdevito/118/head
[jit] clean up exported source format The previous PR in the stack removed the need to order classes/functions or have correct import statements. This resolved circular depedency issues that can arise when class constructors like ModuleList put new instances of themselves in a common namespace. This PR changes our export format to no longer produce this information. By doing so we can make the logic signficantly simpler, since we just keep track of an individual PythonPrint object per file. Notes: * PythonPrint was changed to manage its own stream/list of ranges. It was doing this anyway internally, this just makes the API more clear. * Since we are changing the serialization format, I also removed op_version_set. It is now replaced with the VERSION number that written in the zip archive. This further simplifies the code emission process. * A test of op_version_set was removed since there is no longer any behavior to test. gh-metadata: pytorch pytorch 26788 gh/zdevito/118/head
[jit] clean up exported source format The previous PR in the stack removed the need to order classes/functions or have correct import statements. This resolved circular depedency issues that can arise when class constructors like ModuleList put new instances of themselves in a common namespace. This PR changes our export format to no longer produce this information. By doing so we can make the logic signficantly simpler, since we just keep track of an individual PythonPrint object per file. Notes: * PythonPrint was changed to manage its own stream/list of ranges. It was doing this anyway internally, this just makes the API more clear. * Since we are changing the serialization format, I also removed op_version_set. It is now replaced with the VERSION number that written in the zip archive. This further simplifies the code emission process. * A test of op_version_set was removed since there is no longer any behavior to test. gh-metadata: pytorch pytorch 26788 gh/zdevito/118/head
[jit] clean up exported source format The previous PR in the stack removed the need to order classes/functions or have correct import statements. This resolved circular depedency issues that can arise when class constructors like ModuleList put new instances of themselves in a common namespace. This PR changes our export format to no longer produce this information. By doing so we can make the logic signficantly simpler, since we just keep track of an individual PythonPrint object per file. Notes: * PythonPrint was changed to manage its own stream/list of ranges. It was doing this anyway internally, this just makes the API more clear. * Since we are changing the serialization format, I also removed op_version_set. It is now replaced with the VERSION number that written in the zip archive. This further simplifies the code emission process. * A test of op_version_set was removed since there is no longer any behavior to test. gh-metadata: pytorch pytorch 26788 gh/zdevito/118/head
[jit] clean up exported source format The previous PR in the stack removed the need to order classes/functions or have correct import statements. This resolved circular depedency issues that can arise when class constructors like ModuleList put new instances of themselves in a common namespace. This PR changes our export format to no longer produce this information. By doing so we can make the logic signficantly simpler, since we just keep track of an individual PythonPrint object per file. Notes: * PythonPrint was changed to manage its own stream/list of ranges. It was doing this anyway internally, this just makes the API more clear. * Since we are changing the serialization format, I also removed op_version_set. It is now replaced with the VERSION number that written in the zip archive. This further simplifies the code emission process. * A test of op_version_set was removed since there is no longer any behavior to test. gh-metadata: pytorch pytorch 26788 gh/zdevito/118/head
suo
approved these changes
Oct 9, 2019
Member
suo
left a comment
There was a problem hiding this comment.
Looks good to me generally and cleans things up nicely. I think we need to split up the land of this for forward compat reasons though; please do that before landing.
| if (env_.count("torch") == 0) { | ||
| env_["torch"] = std::make_shared<BuiltinModule>("aten", version); | ||
| env_["ops"] = std::make_shared<OpsValue>(version); | ||
| void parsePossibleVersionNumber(Lexer& L) { |
Member
There was a problem hiding this comment.
Trying to figure out what exactly the forward compat implications are here. We already removed the necessity for import statements so that doesn't matter I guess.
So it seems the only thing left is the version number. If a newer version exports a model without the version number and a newer version tries to read it and fails, we will leave users in a bad spot.
We could avoid this by landing this change making the version number optional, waiting a bit, then landing the part of the code that removes that. Since this breakage happens on every model (since version numbers are put in every file today), I think it's worth doing the extra work to avoid the forward compat breakage.
|
|
||
| ~PythonPrintImpl() {} | ||
|
|
||
| private: |
Contributor
Author
There was a problem hiding this comment.
yes, the class is private to the file now, so there is no need to further protect its members.
[jit] clean up exported source format The previous PR in the stack removed the need to order classes/functions or have correct import statements. This resolved circular depedency issues that can arise when class constructors like ModuleList put new instances of themselves in a common namespace. This PR changes our export format to no longer produce this information. By doing so we can make the logic signficantly simpler, since we just keep track of an individual PythonPrint object per file. Notes: * PythonPrint was changed to manage its own stream/list of ranges. It was doing this anyway internally, this just makes the API more clear. * Since we are changing the serialization format, I also removed op_version_set. It is now replaced with the VERSION number that written in the zip archive. This further simplifies the code emission process. * A test of op_version_set was removed since there is no longer any behavior to test. gh-metadata: pytorch pytorch 26788 gh/zdevito/118/head
[jit] clean up exported source format The previous PR in the stack removed the need to order classes/functions or have correct import statements. This resolved circular depedency issues that can arise when class constructors like ModuleList put new instances of themselves in a common namespace. This PR changes our export format to no longer produce this information. By doing so we can make the logic signficantly simpler, since we just keep track of an individual PythonPrint object per file. Notes: * PythonPrint was changed to manage its own stream/list of ranges. It was doing this anyway internally, this just makes the API more clear. * Since we are changing the serialization format, I also removed op_version_set. It is now replaced with the VERSION number that written in the zip archive. This further simplifies the code emission process. * A test of op_version_set was removed since there is no longer any behavior to test. gh-metadata: pytorch pytorch 26788 gh/zdevito/118/head
[jit] clean up exported source format The previous PR in the stack removed the need to order classes/functions or have correct import statements. This resolved circular depedency issues that can arise when class constructors like ModuleList put new instances of themselves in a common namespace. This PR changes our export format to no longer produce this information. By doing so we can make the logic signficantly simpler, since we just keep track of an individual PythonPrint object per file. Notes: * PythonPrint was changed to manage its own stream/list of ranges. It was doing this anyway internally, this just makes the API more clear. * Since we are changing the serialization format, I also removed op_version_set. It is now replaced with the VERSION number that written in the zip archive. This further simplifies the code emission process. * A test of op_version_set was removed since there is no longer any behavior to test. gh-metadata: pytorch pytorch 26788 gh/zdevito/118/head
[jit] clean up exported source format The previous PR in the stack removed the need to order classes/functions or have correct import statements. This resolved circular depedency issues that can arise when class constructors like ModuleList put new instances of themselves in a common namespace. This PR changes our export format to no longer produce this information. By doing so we can make the logic signficantly simpler, since we just keep track of an individual PythonPrint object per file. Notes: * PythonPrint was changed to manage its own stream/list of ranges. It was doing this anyway internally, this just makes the API more clear. * Since we are changing the serialization format, I also removed op_version_set. It is now replaced with the VERSION number that written in the zip archive. This further simplifies the code emission process. * A test of op_version_set was removed since there is no longer any behavior to test. gh-metadata: pytorch pytorch 26788 gh/zdevito/118/head
[jit] clean up exported source format The previous PR in the stack removed the need to order classes/functions or have correct import statements. This resolved circular depedency issues that can arise when class constructors like ModuleList put new instances of themselves in a common namespace. This PR changes our export format to no longer produce this information. By doing so we can make the logic signficantly simpler, since we just keep track of an individual PythonPrint object per file. Notes: * PythonPrint was changed to manage its own stream/list of ranges. It was doing this anyway internally, this just makes the API more clear. * Since we are changing the serialization format, I also removed op_version_set. It is now replaced with the VERSION number that written in the zip archive. This further simplifies the code emission process. * A test of op_version_set was removed since there is no longer any behavior to test. gh-metadata: pytorch pytorch 26788 gh/zdevito/118/head
[jit] clean up exported source format The previous PR in the stack removed the need to order classes/functions or have correct import statements. This resolved circular depedency issues that can arise when class constructors like ModuleList put new instances of themselves in a common namespace. This PR changes our export format to no longer produce this information. By doing so we can make the logic signficantly simpler, since we just keep track of an individual PythonPrint object per file. Notes: * PythonPrint was changed to manage its own stream/list of ranges. It was doing this anyway internally, this just makes the API more clear. * Since we are changing the serialization format, I also removed op_version_set. It is now replaced with the VERSION number that written in the zip archive. This further simplifies the code emission process. * A test of op_version_set was removed since there is no longer any behavior to test. gh-metadata: pytorch pytorch 26788 gh/zdevito/118/head
This was referenced Oct 11, 2019
[jit] clean up exported source format The previous PR in the stack removed the need to order classes/functions or have correct import statements. This resolved circular depedency issues that can arise when class constructors like ModuleList put new instances of themselves in a common namespace. This PR changes our export format to no longer produce this information. By doing so we can make the logic signficantly simpler, since we just keep track of an individual PythonPrint object per file. Notes: * PythonPrint was changed to manage its own stream/list of ranges. It was doing this anyway internally, this just makes the API more clear. * Since we are changing the serialization format, I also removed op_version_set. It is now replaced with the VERSION number that written in the zip archive. This further simplifies the code emission process. * A test of op_version_set was removed since there is no longer any behavior to test. gh-metadata: pytorch pytorch 26788 gh/zdevito/118/head
[jit] clean up exported source format The previous PR in the stack removed the need to order classes/functions or have correct import statements. This resolved circular depedency issues that can arise when class constructors like ModuleList put new instances of themselves in a common namespace. This PR changes our export format to no longer produce this information. By doing so we can make the logic signficantly simpler, since we just keep track of an individual PythonPrint object per file. Notes: * PythonPrint was changed to manage its own stream/list of ranges. It was doing this anyway internally, this just makes the API more clear. * Since we are changing the serialization format, I also removed op_version_set. It is now replaced with the VERSION number that written in the zip archive. This further simplifies the code emission process. * A test of op_version_set was removed since there is no longer any behavior to test. gh-metadata: pytorch pytorch 26788 gh/zdevito/118/head
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
The previous PR in the stack removed the need to order classes/functions
or have correct import statements. This resolved circular depedency issues
that can arise when class constructors like ModuleList put new instances
of themselves in a common namespace.
This PR changes our export format to no longer produce this information.
By doing so we can make the logic signficantly simpler, since we just
keep track of an individual PythonPrint object per file.
Notes:
was doing this anyway internally, this just makes the API more clear.
It is now replaced with the VERSION number that written in the zip archive.
This further simplifies the code emission process.
to test.
Differential Revision: D17566438