Skip to content

extend model proto to include model local functions#3532

Merged
askhade merged 14 commits intoonnx:masterfrom
askhade:extend_modelproto
Jun 28, 2021
Merged

extend model proto to include model local functions#3532
askhade merged 14 commits intoonnx:masterfrom
askhade:extend_modelproto

Conversation

@askhade
Copy link
Copy Markdown
Contributor

@askhade askhade commented Jun 16, 2021

Signed-off-by: Ashwini Khade askhade@microsoft.com

Description
Extending model proto to include a list of model local function protos.
FYI - all .proto and .proto3 files except onnx.in.proto and onnx-operators.in.proto are auto generated

Signed-off-by: Ashwini Khade <askhade@microsoft.com>
@askhade askhade requested review from a team as code owners June 16, 2021 20:32
@askhade askhade requested a review from gramalingam June 16, 2021 20:33
@askhade askhade added this to the 1.10 milestone Jun 16, 2021
Comment thread docs/IR.md Outdated
Comment thread onnx/onnx.in.proto Outdated
Comment on lines +396 to +397
// The opset imports in FunctionProto should not conflict with the ones in
// model proto and should be mergeable. Example, if the function proto imports domain
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Can two different FunctionProtos in this list have conflicting imports? (I'm guessing they can't, but the wording here is ambiguous.)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, is it useful to be able to control the opset imports on a per-function basis, given this restriction? Would it be more ergonomic for the functions to inherit the imports from the containing ModelProto?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this originates from the use of FunctionProto defined outside of models (in defining standard ops as functions). We could potentially distinguish between the two usages and restrict some fields to be used only in FunctionProtos outside of models.

Copy link
Copy Markdown
Contributor Author

@askhade askhade Jun 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ONNX does not allow importing different versions of the same domain so there cannot be conflicting imports...

opset imports were added to FunctionProto to allow functions to easily import opsets other than what the model imports... the restriction is only on the version of the domain being unique within the model...

Comment thread onnx/onnx.in.proto Outdated
Comment thread onnx/onnx.in.proto Outdated
Comment on lines +807 to +808
// A FunctionProto body (graph) may implicitly rely on the OperatorSet that
// this function belongs to. It can also explicitly rely on more OperatorSets
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the set of functions in a ModelProto count as an "OperatorSet" for the purposes of this comment? (i.e. Can a model-local function refer to other model-local functions?)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should allow a model-local function to refer to other model-local functions.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes one model local function can refer to another model local function...

more and more I think about this I am leaning towards adding domain field to FunctionProto...

Comment thread onnx/onnx.in.proto
// model proto and should be mergeable. Example, if the function proto imports domain
// 'A' opset 10 whereas model proto also imports domain 'A' but with a different opset
// then this is a merge conflict.
repeated FunctionProto functions = 25;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does the model refer to these functions? If they were in an operator set, presumably there would be a domain, but I don't see a domain here.

Should this field be an OperatorSetProto (or some subset of that proto)?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FunctionProto has a "name" field. The model compares this with the op_type field from NodeProto.

Rama and I were discussing whether we want domain qualified names, meaning should we add domain field to the functionproto as well... we can do that if we think just the name field wont be enough...

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two possible options:
(a) We add a domain field to FunctionProto. (We then need to decide whether we want to allow these in-model functions to override out-of-model functions defined in opschema registries.)
(b) Reserve a domain for in-model functions (like 'onnx.ai.local') if we want to avoid the above.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added domain field for FunctionProto

Comment thread onnx/onnx.in.proto Outdated
askhade added 2 commits June 18, 2021 11:57
Signed-off-by: Ashwini Khade <askhade@microsoft.com>
Signed-off-by: Ashwini Khade <askhade@microsoft.com>
askhade added 3 commits June 18, 2021 16:49
Signed-off-by: Ashwini Khade <askhade@microsoft.com>
Signed-off-by: Ashwini Khade <askhade@microsoft.com>
Signed-off-by: Ashwini Khade <askhade@microsoft.com>
Comment thread onnx/onnx-ml.proto Outdated
Comment thread onnx/onnx-ml.proto Outdated
Comment thread onnx/onnx-ml.proto
Comment thread onnx/checker.cc Outdated
Comment thread onnx/checker.cc Outdated
Comment thread onnx/checker.cc Outdated
Comment thread onnx/defs/function.cc Outdated
Comment thread onnx/defs/function.cc Outdated
Comment thread onnx/defs/function.cc
}
}
if(domain_version == -1) {
ONNX_THROW("No opset import registered for domain '" + node.domain() + "' in function proto");
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, the check here means that every function in a domain "foo" must have an entry for "foo" in its opset_import with a particular version, which becomes domain_version here. Is that right?

This logic seems a little tortuous... would it make more sense to have a domain_version attribute on the FunctionProto? Or - perhaps better - the version could be inferred from the opset_import on the containing model (or inferred from the op schema, if the function is not a model-local function)?

At the very least, we need to document this requirement, because it was not obvious to me from reading the protobuf spec.

[EDIT: I see code in schema.cc that appears to add this information if it's missing. Is it still required for model-local functions?]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right every function in a domain "Foo" must have an entry for "foo" in its opset_import with a particular version.

For function ops - If this info is missing then it is inferred ( from op schema ) and implicitly added during schema finalization.

For model local functions - this info is required to be present. We can infer the version for the function from model opset imports but I wanted to reduce the differences in construction or interpretation of FunctionProto for FunctionOps vs model local functions so I decided not to make an exception for model local functions in this case.

Let me know your thoughts...

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There seem to be some corner cases where it is worth documenting what the standard requires. (a) If we have a model-local function "bar" in a domain "foo", is the model/graph required to have an import for "foo" (even though the model does not contain any calls to anything other than "foo")? (b) If so, if the FunctionProto for "bar" also includes an import for "foo", is it allowed to be different from what the graph specifies? The opset-compatibility discussion does not clarify this special-case. It doesn't seem important to allow them to be different in this case. (c) If we don't allow to be different, forcing the model to include the same information in two places seems less desirable. So, inferring it from the model's imports may be reasonable.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, it seems that model-local functions should not need to use opset-imports and should be able to get away with the model's own opset-imports (for the ops called from the function body). So, doing the same for the function's own domain (look it up from the model's imports) seems reasonable. It is also important to clarify which of the three options we go with (i) required, (ii) allowed but not required, or (iii) not allowed. In functions outside models, we do need to allow opset imports.

Comment thread onnx/defs/schema.cc Outdated
Signed-off-by: Ashwini Khade <askhade@microsoft.com>
Comment thread onnx/onnx-ml.proto Outdated
askhade added 5 commits June 22, 2021 10:30
Signed-off-by: Ashwini Khade <askhade@microsoft.com>
Signed-off-by: Ashwini Khade <askhade@microsoft.com>
Signed-off-by: Ashwini Khade <askhade@microsoft.com>
Signed-off-by: Ashwini Khade <askhade@microsoft.com>
Signed-off-by: Ashwini Khade <askhade@microsoft.com>
Comment thread onnx/checker.cc Outdated
Comment thread onnx/onnx-ml.proto Outdated
askhade added 2 commits June 25, 2021 09:32
Signed-off-by: Ashwini Khade <askhade@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants