Define differentiability of Pool/Conv Ops#2893
Conversation
|
Differentiability of the Pooling operator is shown by Method 1, "Reuse Existing Deep Learning Frameworks", described here #2794. Showing differentiability of MaxPool input/output and export of model from MXnet to ONNX: from mxnet import gluon, nd, autograd, context
from mxnet.contrib import onnx as onnx_mxnet
import numpy as np
net = gluon.nn.HybridSequential()
net.add(gluon.nn.MaxPool1D(pool_size=2))
net.hybridize()
x = nd.random.uniform(-1,1,(1,3,4))
labels = nd.random.uniform(-1,1,(1,3,2))
L2loss = gluon.loss.L2Loss()
# showing differentiability of input
x.attach_grad()
with autograd.record():
output = net(x)
loss = L2loss(output, labels)
loss.backward()
print(x.grad)
# showing differentiability of output
with autograd.record():
output = net(x)
output.attach_grad()
loss = L2loss(output, labels)
loss.backward()
print(output.grad)
net.export("maxpool1d")
sym = './maxpool1d-symbol.json'
params = './maxpool1d-0000.params'
onnx_file = './maxpool1d.onnx'
converted_model_path = onnx_mxnet.export_model(sym, params, [x.shape], np.float32, onnx_file)We can see the derivatives of the input and output from the print statements: |
|
Differentiability of the Conv Operator is shown similarly: net = gluon.nn.HybridSequential()
net.add(gluon.nn.Conv1D(1, 2, weight_initializer = 'ones'))
net.hybridize()
net.initialize(ctx=context.cpu(0))
net_params = net.collect_params('0.weight|0.bias')
x = nd.random.uniform(-1,1,(1, 1, 3))
x.attach_grad()
label = nd.random.uniform(-1,1,(1,1,2))
with autograd.record():
output = net(x)
loss = L2loss(output, label)
loss.backward()
print(x.grad)
print(net_params['0.weight'].grad())
print(net_params['0.bias'].grad())
with autograd.record():
output = net(x)
output.attach_grad()
loss = L2loss(output, label)
loss.backward()
print(output.grad)
#export of model to onnx
net.export("conv1d")
sym = './conv1d-symbol.json'
params = './conv1d-0000.params'
onnx_file = './conv1d.onnx'
converted_model_path = onnx_mxnet.export_model(sym, params, [x.shape], np.float32, onnx_file)We see the derivatives of the input and kernel/bias parameters as well as the output derivative from the print statements: |
5b7bda8 to
37650ba
Compare
|
@wschin could you give some advice on how to reproduce those build failures locally? I see that when you added the differentiability tag to operators in onnx/defs/tensor/defs.cc (https://github.com/onnx/onnx/pull/2723/files#), you added the parameter "OpSchema::Single" to operators such as Transpose, Scatter, etc. Why was that necessary? Note: I believe whats happening is that the Operator description files need to be auto generated locally and then be added to the commit!? I ran after making the changes to the defs file but docs/Operators.md doesn't get updated. Update: I think I figured it out. I needed to run |
This commit adds the differentiability tags to the MaxPool, AveragePool and Conv operators.
|
@wschin I revised the commit according to your comments. Thanks for catching those two issues. |
This commit adds the differentiability tags to the MaxPool, AveragePool and Conv operators. Co-authored-by: Wei-Sheng Chin <wschin@outlook.com>
This commit adds the differentiability tags to the MaxPool, AveragePool and Conv operators. Co-authored-by: Wei-Sheng Chin <wschin@outlook.com> Signed-off-by: daquexian <daquexian566@gmail.com>
This commit adds the differentiability tags to the MaxPool, AveragePool and Conv operators. Co-authored-by: Wei-Sheng Chin <wschin@outlook.com>
This commit adds the differentiability tags to the
MaxPool, AveragePool, RoiPool and Conv operators.