Skip to content

Improve compatiblity with proto3 and enable reading attributes#2288

Merged
gramalingam merged 2 commits intoonnx:masterfrom
yuslepukhin:make_attr_proto3_friendly
Sep 6, 2019
Merged

Improve compatiblity with proto3 and enable reading attributes#2288
gramalingam merged 2 commits intoonnx:masterfrom
yuslepukhin:make_attr_proto3_friendly

Conversation

@yuslepukhin
Copy link
Copy Markdown
Contributor

Proto3 does not output primitive types into a stream if they match type default.
This presents a problem when C# protobuf generated code creates a model where type default are omitted. Our proto files are compatible with proto3 and in some cases special provisions exist such as AttributeType enum which actually indicate whether the Attribute was really set by the user.
Thus we remove has_i(), has_f() and has_s() checks if the corresponding enum is set.
This allows proto2 code to read models serialized by proto3 code.

  with type default values even though they are not in the stream.
@yuslepukhin yuslepukhin requested a review from a team as a code owner September 5, 2019 21:03
@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Sep 5, 2019

CLA assistant check
All committers have signed the CLA.

Copy link
Copy Markdown
Member

@houseroad houseroad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am wondering whether we can add a new CI or change an existing CI to proto3

@yuslepukhin yuslepukhin force-pushed the make_attr_proto3_friendly branch from 5dc806f to 4682fea Compare September 5, 2019 21:40
@gramalingam
Copy link
Copy Markdown
Contributor

It would help other users if we could document our guidelines for ensuring proto2 and proto3 compatibility (as well as compatibility across different languages in case it makes a difference). My understanding (for this PR) is the following, is this correct? DO NOT use the has_foo() method for any field "foo" of primitive or enum types; it is treated as being present with a default value (if absent in the serialized representation). However, for all the non-primitive types (which are always declared optional in ONNX) as well as oneof fields, it is safe to use the "has_foo" method. Is this correct?

@yuslepukhin
Copy link
Copy Markdown
Contributor Author

It would help other users if we could document our guidelines for ensuring proto2 and proto3 compatibility (as well as compatibility across different languages in case it makes a difference). My understanding (for this PR) is the following, is this correct? DO NOT use the has_foo() method for any field "foo" of primitive or enum types; it is treated as being present with a default value (if absent in the serialized representation). However, for all the non-primitive types (which are always declared optional in ONNX) as well as oneof fields, it is safe to use the "has_foo" method. Is this correct?

To be a little more specific. The reason we are able not to use has_*() on the primitives is because when we assign a value to them we also set the enum. So if enum is not set then it is UNDEFINED (0) by default, otherwise simply read the primitive type because we know the value is correct. This works when primitives are intentionally paired with enums and 0 means UNDEFINED. There are comments in the proto files to that effect.

We also do not want to use has_*() on oneof fields because in that case proto compiler automatically generates a corresponding enum. That enum value can tell us which field is set if any.

Otherwise, we can use has_*() providing we are built with proto2 library. proto3 compiler does not generate has_*() methods, which would become an issue if we migrated to proto3.

Copy link
Copy Markdown
Collaborator

@wschin wschin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! This change helps frameworks relying on proto3.

@prasanthpul prasanthpul added this to the 1.6 milestone Sep 6, 2019
@yuslepukhin
Copy link
Copy Markdown
Contributor Author

@houseroad Can you, please, be a bit more specific as to what you mean to change existing CI to proto3? Current code will not compile with proto3 interfaces, mostly because of the extensive use of has_*() methods. So that would call for refactoring. However, changing to proto3 would require even more discussions. I am trying alleviate a specific pain point where compatibility is required.

@gramalingam gramalingam merged commit 568b65a into onnx:master Sep 6, 2019
jcwchen pushed a commit to jcwchen/onnx that referenced this pull request Sep 23, 2020
…2288)

with type default values even though they are not in the stream.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants