Skip to content

Define differentiability of Pool/Conv Ops#2893

Merged
wschin merged 2 commits intoonnx:masterfrom
ma-hei:master
Jul 21, 2020
Merged

Define differentiability of Pool/Conv Ops#2893
wschin merged 2 commits intoonnx:masterfrom
ma-hei:master

Conversation

@ma-hei
Copy link
Copy Markdown
Contributor

@ma-hei ma-hei commented Jul 13, 2020

This commit adds the differentiability tags to the
MaxPool, AveragePool, RoiPool and Conv operators.

@ma-hei ma-hei requested a review from a team as a code owner July 13, 2020 02:20
@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Jul 13, 2020

CLA assistant check
All committers have signed the CLA.

@ma-hei
Copy link
Copy Markdown
Contributor Author

ma-hei commented Jul 13, 2020

Differentiability of the Pooling operator is shown by Method 1, "Reuse Existing Deep Learning Frameworks", described here #2794.

Showing differentiability of MaxPool input/output and export of model from MXnet to ONNX:

from mxnet import gluon, nd, autograd, context
from mxnet.contrib import onnx as onnx_mxnet
import numpy as np

net = gluon.nn.HybridSequential()
net.add(gluon.nn.MaxPool1D(pool_size=2))
net.hybridize()

x = nd.random.uniform(-1,1,(1,3,4))
labels = nd.random.uniform(-1,1,(1,3,2))
L2loss = gluon.loss.L2Loss()

# showing differentiability of input
x.attach_grad()
with autograd.record():
    output = net(x)
    loss = L2loss(output, labels)

loss.backward()
print(x.grad)

# showing differentiability of output
with autograd.record():
    output = net(x)
    output.attach_grad()
    loss = L2loss(output, labels)

loss.backward()
print(output.grad)

net.export("maxpool1d")

sym = './maxpool1d-symbol.json'
params = './maxpool1d-0000.params'
onnx_file = './maxpool1d.onnx'

converted_model_path = onnx_mxnet.export_model(sym, params, [x.shape], np.float32, onnx_file)

We can see the derivatives of the input and output from the print statements:

[[[ 0.          0.05175247  0.          0.1822437 ]
  [ 0.         -0.01127579  0.          0.2635129 ]
  [ 0.         -0.11336635  0.12441261  0.        ]]]
<NDArray 1x3x4 @cpu(0)>

[[[ 0.05175247  0.1822437 ]
  [-0.01127579  0.2635129 ]
  [-0.11336635  0.12441261]]]
<NDArray 1x3x2 @cpu(0)>

@ma-hei
Copy link
Copy Markdown
Contributor Author

ma-hei commented Jul 13, 2020

Differentiability of the Conv Operator is shown similarly:

net = gluon.nn.HybridSequential()
net.add(gluon.nn.Conv1D(1, 2, weight_initializer = 'ones'))
net.hybridize()
net.initialize(ctx=context.cpu(0))

net_params = net.collect_params('0.weight|0.bias')

x = nd.random.uniform(-1,1,(1, 1, 3))
x.attach_grad()
label = nd.random.uniform(-1,1,(1,1,2))

with autograd.record():
    output = net(x)
    loss = L2loss(output, label)

loss.backward()
print(x.grad)
print(net_params['0.weight'].grad())
print(net_params['0.bias'].grad())

with autograd.record():
    output = net(x)
    output.attach_grad()
    loss = L2loss(output, label)

loss.backward()
print(output.grad)

#export of model to onnx
net.export("conv1d")
sym = './conv1d-symbol.json'
params = './conv1d-0000.params'
onnx_file = './conv1d.onnx'

converted_model_path = onnx_mxnet.export_model(sym, params, [x.shape], np.float32, onnx_file)

We see the derivatives of the input and kernel/bias parameters as well as the output derivative from the print statements:

[[[-0.45106208 -0.21056682  0.24049526]]]
<NDArray 1x1x3 @cpu(0)>

[[[0.09440736 0.1604658 ]]]
<NDArray 1x1x2 @cpu(0)>

[-0.21056682]
<NDArray 1 @cpu(0)>

[[[-0.45106208  0.24049526]]]
<NDArray 1x1x2 @cpu(0)>

@ma-hei ma-hei force-pushed the master branch 2 times, most recently from 5b7bda8 to 37650ba Compare July 13, 2020 05:32
@ma-hei
Copy link
Copy Markdown
Contributor Author

ma-hei commented Jul 13, 2020

@wschin could you give some advice on how to reproduce those build failures locally? I see that when you added the differentiability tag to operators in onnx/defs/tensor/defs.cc (https://github.com/onnx/onnx/pull/2723/files#), you added the parameter "OpSchema::Single" to operators such as Transpose, Scatter, etc. Why was that necessary?

Note: I believe whats happening is that the Operator description files need to be auto generated locally and then be added to the commit!? I ran

python3 onnx/defs/gen_doc.py

after making the changes to the defs file but docs/Operators.md doesn't get updated.

Update: I think I figured it out. I needed to run ./tools/update_doc.sh to update the docs instead of running gen_doc.py directly.

@wschin
Copy link
Copy Markdown
Collaborator

wschin commented Jul 14, 2020

        OpSchema::Optional)

NonDifferentiable. Because PoolOpSchemaGenerator is changed, we need to tag all places where PoolOpSchemaGenerator is used.


Refers to: onnx/defs/nn/defs.cc:368 in c0e1989. [](commit_id = c0e1989, deletion_comment = False)

Comment thread onnx/defs/nn/defs.cc Outdated
This commit adds the differentiability tags to the
MaxPool, AveragePool and Conv operators.
@ma-hei
Copy link
Copy Markdown
Contributor Author

ma-hei commented Jul 14, 2020

@wschin I revised the commit according to your comments. Thanks for catching those two issues.

Copy link
Copy Markdown
Collaborator

@wschin wschin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@wschin wschin added the topic: training Issues related to ONNX training label Jul 21, 2020
@wschin wschin merged commit 91bdfe4 into onnx:master Jul 21, 2020
sveta-levitan pushed a commit to sveta-levitan/onnx that referenced this pull request Aug 25, 2020
This commit adds the differentiability tags to the
MaxPool, AveragePool and Conv operators.

Co-authored-by: Wei-Sheng Chin <wschin@outlook.com>
daquexian pushed a commit to daquexian/onnx that referenced this pull request Sep 19, 2020
This commit adds the differentiability tags to the
MaxPool, AveragePool and Conv operators.

Co-authored-by: Wei-Sheng Chin <wschin@outlook.com>
Signed-off-by: daquexian <daquexian566@gmail.com>
jcwchen pushed a commit to jcwchen/onnx that referenced this pull request Sep 23, 2020
This commit adds the differentiability tags to the
MaxPool, AveragePool and Conv operators.

Co-authored-by: Wei-Sheng Chin <wschin@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

topic: training Issues related to ONNX training

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants