Skip to content

Add 'ignore_index' input in the spec for SoftmaxCrossEntropyLoss and NLLLoss.#2680

Merged
chinhuang007 merged 19 commits intoonnx:masterfrom
codemzs:softmaxcrossentropyloss
Mar 30, 2020
Merged

Add 'ignore_index' input in the spec for SoftmaxCrossEntropyLoss and NLLLoss.#2680
chinhuang007 merged 19 commits intoonnx:masterfrom
codemzs:softmaxcrossentropyloss

Conversation

@codemzs
Copy link
Copy Markdown
Contributor

@codemzs codemzs commented Mar 25, 2020

This change fixes the spec for SoftmaxCrossEntropyLoss and NLLLoss to make it compatible with PyTorch.

@codemzs codemzs requested a review from a team as a code owner March 25, 2020 08:02
wschin
wschin reviewed Mar 25, 2020
View reviewed changes
Comment thread docs/Changelog.md Outdated
@lara-hdr
Copy link
Copy Markdown
Contributor

lara-hdr commented Mar 25, 2020 •
edited
Loading

@codemzs, thanks for the PR :)
I have some comments but I cannot add them inline since the lines are not modified;

NllLoss and SoftmaxCrossEntropy are functions not operators; so they have a function body in ONNX that we will need to update, even if there is a kernel.

The functions are defined in onnx/defs/math/defs.cc, line 1831 and 2320;
We will need to update the 'if' clauses 'if (!ctx.hasInput(2))' to deal with cases with 2, 3, 4 inputs.In case ignore_iindex is specified (4 inputs), we can use ScatterElements to put a 0 in the weight on the ignored index in the weights.

The other option would be to switch to an operator.

lara-hdr
lara-hdr reviewed Mar 25, 2020
View reviewed changes
Comment thread onnx/defs/math/defs.cc
lara-hdr
lara-hdr reviewed Mar 25, 2020
View reviewed changes
Comment thread onnx/backend/test/case/node/softmaxcrossentropy.py Outdated
lara-hdr
lara-hdr reviewed Mar 25, 2020
View reviewed changes
Comment thread onnx/backend/test/case/node/softmaxcrossentropy.py
lara-hdr
lara-hdr reviewed Mar 25, 2020
View reviewed changes
Comment thread onnx/backend/test/case/node/softmaxcrossentropy.py
@snnn
Copy link
Copy Markdown
Member

snnn commented Mar 25, 2020

Is there any tests to verify if the function body works as expected?

lara-hdr
lara-hdr reviewed Mar 25, 2020
View reviewed changes
Comment thread onnx/defs/math/defs.cc Outdated
@gramalingam
Copy link
Copy Markdown
Contributor

gramalingam commented Mar 25, 2020

Is there any tests to verify if the function body works as expected?

Every "node" test-case is also expanded into a test-case for the corresponding function-body: for example, like here: https://github.com/onnx/onnx/tree/master/onnx/backend/test/data/node/test_softmax_cross_entropy_mean_3d_expanded

@chinhuang007 chinhuang007 mentioned this pull request Mar 27, 2020
Closed
wschin
wschin reviewed Mar 27, 2020
View reviewed changes
Comment thread docs/Changelog.md Outdated
wschin
wschin reviewed Mar 27, 2020
View reviewed changes
Comment thread docs/Operators.md
wschin
wschin reviewed Mar 27, 2020
View reviewed changes
Comment thread onnx/backend/test/case/node/softmaxcrossentropy.py
wschin
wschin approved these changes Mar 27, 2020
View reviewed changes
Copy link
Copy Markdown
Collaborator

@wschin wschin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM. I only have one major comment about the default value of the ignored_index. Does it really need a default value if we don't have a good default?

@codemzs
Copy link
Copy Markdown
Contributor Author

codemzs commented Mar 30, 2020

@wschin I have removed the default value for ignore_index but still kept it as optional because it is easier to implement the function body. In the converter we will pass an ignore_index only if its value provided is within the range [0, C). I did this at the suggestion of @lara-hdr.

@codemzs codemzs force-pushed the softmaxcrossentropyloss branch from e548117 to 76a33a6 Compare March 30, 2020 01:04
lara-hdr
lara-hdr approved these changes Mar 30, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@lara-hdr lara-hdr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

snnn
snnn reviewed Mar 30, 2020
View reviewed changes
Comment thread onnx/defs/math/defs.cc
snnn
snnn reviewed Mar 30, 2020
View reviewed changes
Comment thread onnx/defs/math/defs.cc Outdated
@codemzs codemzs force-pushed the softmaxcrossentropyloss branch from a96aba3 to 2e82204 Compare March 30, 2020 06:56
@codemzs
Copy link
Copy Markdown
Contributor Author

codemzs commented Mar 30, 2020

@wschin / @ebarsoum / @gramalingam Please merge this change. All of the SoftmaxCrossEntropyLoss & NegativeLogLikelihoodLoss ONNX tests on ONNX Runtime are now passing with the updated spec. @lara-hdr has also verified the outputs match with that of PyTorch.

image

Thank you, @chinhuang007 for giving me time till end of weekend to wrap this change up, much appreciated! This change is ready to be shipped with ONNX 1.7 release.

CC: @SherlockNoMad

@chinhuang007 chinhuang007 merged commit 20b3e10 into onnx:master Mar 30, 2020
@chinhuang007
Copy link
Copy Markdown
Contributor

chinhuang007 commented Mar 30, 2020

@codemzs Thank you for the contribution!

@chinhuang007 chinhuang007 added this to the 1.7 milestone Mar 30, 2020
linkerzhang added a commit that referenced this pull request Mar 31, 2020
@chinhuang007 @take-cheeze
@linkerzhang @xuhdev @wschin @postrational @codemzs @lara-hdr @vinitra @gramalingam @snnn @skottmckay @sl1pkn07
Rel 1.7.103 verify (#2687)
3193b02 
* Fix Greater/LessOrEqual function definition (#2645)

* Fix Greater/LessOrEqual function definition

* Update test data

Co-authored-by: Ke Zhang <kezhan@microsoft.com>

* Suppress a warning in unsqueeze (#2637)

I keep getting this warning when building PyTorch:

```
In file included from
/home/hong/wsrc/pytorch/third_party/onnx/onnx/defs/tensor/utils.h:6,
                 from
/home/hong/wsrc/pytorch/third_party/onnx/onnx/defs/tensor/defs.cc:4:
/home/hong/wsrc/pytorch/third_party/onnx/onnx/defs/tensor/defs.cc: In
lambda function:
/home/hong/wsrc/pytorch/third_party/onnx/onnx/defs/tensor/defs.cc:1414:22:
warning: unnecessary parentheses in declaration of �i�
[-Wparentheses]
           for (size_t(i) = 0; i < axes.size(); ++i) {
                      ^
/home/hong/wsrc/pytorch/third_party/onnx/onnx/defs/schema.h:959:12:
note: in definition of macro �ONNX_OPERATOR_SET_SCHEMA_EX�
     return impl.SetName(#name)
\
            ^~~~
/home/hong/wsrc/pytorch/third_party/onnx/onnx/defs/tensor/defs.cc:1369:1:
note: in expansion of macro �ONNX_OPERATOR_SET_SCHEMA�
 ONNX_OPERATOR_SET_SCHEMA(
```

This commit should fix it and modernize the code a bit.

Co-authored-by: Ke Zhang <kezhan@microsoft.com>

* [Training] Add Adagrad optimizer operator (#1955)

* Adagrad draft

* MIMO

* Support multiple tensors to be optimized

* Address comments

* Move optimizers to a new place

Remove copied

Add momentum

Save

Remove momentum

Fix

Move constants to attributes

* Fix build

* Add shape test

Add two node tests

Update test coverage

* Fix shape inf

* Fix shape inf

* fix shape inf

* Format

* Add function type

* Merge lines

* Format

* Fix version number

* Update op version in model files

* Fix a test function and update related test files

* Update onnx/backend/test/case/node/adagrad.py

* Remove unused file

* sync docs

* Fix shape test

* sync doc

* sync with master

* Update onnx/defs/training/defs.cc

Co-Authored-By: Michał Karzyński <postrational@users.noreply.github.com>

* sync doc

* address comments

* address a minor comment

* Polish one line

Co-authored-by: Michał Karzyński <postrational@users.noreply.github.com>

* [Training] SG with Momentum Optimizer (#1959)

* SG with Momentum

* Registrate Op

Fix

Update other docs

* Add shape inference code and polish definition

* Update docs

* Add test cases and fix several bugs

* Remove accidently added copy

* Alpha -> alpha & Beta -> beta

* Clarify an attribute

* Fix an attribute

* Fix bug

* Fix missing attributes

* sync doc

* Remove unused domain

* sync with master

Co-authored-by: Chin Huang <chhuang@us.ibm.com>

* Change type of label tensor to int32/int64 in SoftmaxCrossEntropyLoss spec. (#2667)

* Update Pow input types in Opset 12 (#2666)

* Update Pow input types in Opset 12

* gen doc and tests

* remove uints and 8 bit ints

* add tests

* remove uint int x tets

* Adding CI for ONNX Debug mode (Linux, OSX) (#2651)

* adding an osx build, linux build, with and without onnx_ml for debug mode

* test debug mode with ONNX_ML=1

* Rename OPTIONAL to OPTIONAL_VALUE (#2682)

Co-authored-by: G. Ramalingam <grama@microsoft.com>

* Update Batchnorm test (#2674)

* Update Batchnorm test

* relax shape inference on scalar

* Remove unnecessary copies and std::move (#2684)

* Update sequence test case so input is not scalar and splits are specified (#2675)

* Update sequence test case to input is not scalar and splits are specified

* Add spaces to make the checker happy

* Use cmake GNUInstallDirs (#2661)

https://cmake.org/cmake/help/latest/module/GNUInstallDirs.html
this make allow install the libraries (and headers) in different location than `lib` (Gentoo uses lib64 for 64-bits libs)
also change the .cmake files for avoid conclicts if build both 32-bis and 64-bits (avoids conflict/overwrite files)

Co-authored-by: Ke Zhang <kezhan@microsoft.com>

* Add 'ignore_index' input in the spec for SoftmaxCrossEntropyLoss and NLLLoss. (#2680)

* Add 'ignore_index' input in the spec for SoftmaxCrossEntropyLoss and NLLLoss.

* Add tests.

* build break.

* build break.

* clean up.

* build break.

* Change ignore_index to attribute.

* Change ignore_index to attribute.

* PR feedback.

* PR feedback.

* Make ignore_index optional in NLLLoss.

* Build break.

* remove trailing spaces to fix build break.

* Build break.

* Update spec doc.

* Fix NLLLoss function definition to fix test: test_negative_log_likelihood_loss_input_shape_is_NCd1d2_with_weight_reduction_sum_ignore_index_expanded

* PR feedback.

* Fix test for softmax cross entropy loss to exclude ignored_index'ed weights from the sum of weights.

* Build break.

* Reduce binary size of libraries consuming ONNX (part 1/2) (#2643)

* Change the return type for the zipmap operator to match the description in the spec.

* Reduce binary size of libraries consuming ONNX (part 1/2)

* Fix build error

* Replace separate Get*Doc() functions with easy macro for greater convenience

* Add one more macro for complicated operator doc documentation.

Co-authored-by: Ke Zhang <kezhan@microsoft.com>

* Update pybind (#2340) (#2688)

* Change version number for release verification

Change version number for release verification

Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com>
Co-authored-by: Ke Zhang <kezhan@microsoft.com>
Co-authored-by: Hong Xu <hong@topbug.net>
Co-authored-by: Wei-Sheng Chin <wschin@outlook.com>
Co-authored-by: Michał Karzyński <postrational@users.noreply.github.com>
Co-authored-by: M. Zeeshan Siddiqui <mzs@microsoft.com>
Co-authored-by: Lara Haidar <haidar.lara@gmail.com>
Co-authored-by: Vinitra Swamy <vinitras@gmail.com>
Co-authored-by: Changming Sun <chasun@microsoft.com>
Co-authored-by: G. Ramalingam <grama@microsoft.com>
Co-authored-by: Changming Sun <me@sunchangming.com>
Co-authored-by: Scott McKay <skottmckay@gmail.com>
Co-authored-by: Gustavo Alvarez <462213+sl1pkn07@users.noreply.github.com>
Co-authored-by: Pranav Sharma <prs@microsoft.com>
@codemzs codemzs mentioned this pull request Apr 2, 2020
Merged
@codemzs codemzs mentioned this pull request Apr 26, 2020
Closed
jcwchen pushed a commit to jcwchen/onnx that referenced this pull request Sep 23, 2020
@codemzs
Add 'ignore_index' input in the spec for SoftmaxCrossEntropyLoss and …
cf7d5b1 
…NLLLoss. (onnx#2680)

* Add 'ignore_index' input in the spec for SoftmaxCrossEntropyLoss and NLLLoss.

* Add tests.

* build break.

* build break.

* clean up.

* build break.

* Change ignore_index to attribute.

* Change ignore_index to attribute.

* PR feedback.

* PR feedback.

* Make ignore_index optional in NLLLoss.

* Build break.

* remove trailing spaces to fix build break.

* Build break.

* Update spec doc.

* Fix NLLLoss function definition to fix test: test_negative_log_likelihood_loss_input_shape_is_NCd1d2_with_weight_reduction_sum_ignore_index_expanded

* PR feedback.

* Fix test for softmax cross entropy loss to exclude ignored_index'ed weights from the sum of weights.

* Build break.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gramalingam gramalingam gramalingam left review comments

@wschin wschin wschin approved these changes

+3 more reviewers

@snnn snnn snnn left review comments

@KsenijaS KsenijaS KsenijaS left review comments

@lara-hdr lara-hdr lara-hdr approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

1.7

Development

Successfully merging this pull request may close these issues.

7 participants

@codemzs @lara-hdr @snnn @gramalingam @chinhuang007 @wschin @KsenijaS