@pytorchbot retest this please

@ezyang no problem, I understand. Is there anyway you can give us some hints on where this problem maybe in the source. We had a similar problem before reported in issue #5055. And it was fixed in #5124. That was all in c++ code. This one is in Python code.

So, the first thing I would check is that the canonicalizer is working on subgraphs. There don't appear to be structural differences; and if that's the case, the graphs SHOULD number identically. It really looks like we're not canonicalizing subgraphs.

@ezyang are you referring to the Canonicalize function here

It looks to me they are fine. If they are not would I not see the problem also happen in the forward graph?

Just to add to the problem description: The problem only happens for the backward graph . The forward graph is fine.

So for example part of the output of the backward graph of the test_milstm_fusion_cuda is below:

Output of ppc64le
%22 : Float(*, ) = aten::mul(%8, %3)
%23 : Float(, ) = aten::neg(%3)
%24 : int = prim::Constantvalue=1
%25 : Float(, ) = aten::add(%23, %24, %24)
%26 : Float(, *) = aten::mul(%22, %25)

Output of x86
%22 : Float(*, ) = aten::neg(%3)
%23 : int = prim::Constantvalue=1
%24 : Float(, ) = aten::add(%22, %23, %23)
%25 : Float(, ) = aten::mul(%8, %3)
%26 : Float(, *) = aten::mul(%25, %24)

So it seems the instructions are kind of re-arranged. Do you think this is due to canonicalization ?
I'm thinking something to do with the backward functionality. But I may be totally wrong. Need your expert opinion and help as to which source file I need to be looking at.

@avmgithub from not knowing very much about what is wrong, I would start from

@zou3519 , I've attached 2 files , one for x86 and one ppc64le . From a test run like : python -m unittest -q test_jit.TestScript.test_milstm_fusion_cuda . There's differences in the sigmoid function. I'm assuming the graph is for the backward operation. Here is an example:

x86 :
%56 : Dynamic = aten::neg(%outgate)
%57 : Dynamic = aten::add(%56, %28, %28)
%58 : Dynamic = aten::mul(%34, %outgate)
%59 : Dynamic = aten::mul(%58, %57)

ppc :
%56 : Dynamic = aten::mul(%34, %outgate)
%57 : Dynamic = aten::neg(%outgate)
%58 : Dynamic = aten::add(%57, %28, %28)
%59 : Dynamic = aten::mul(%56, %58)

There are 3 such instances since there are three sigmoid functions in the MiLSTMCell function in test_jit.py

x86.txt
ppc.txt

Let me know which source I can start looking for differences. I appreciate the help

@zou3519 When you get the chance, do know where (which source file) the sigmoid backward graph is formed ? This seems to be where the discrepancy is coming from.

example:

%70 : Dynamic = prim::GradOfname="aten::sigmoid"
block0() {
%71 : Dynamic = aten::mul(%49, %ingate)
%72 : Dynamic = aten::neg(%ingate)
%73 : Dynamic = aten::add(%72, %28, %28)
%74 : Dynamic = aten::mul(%71, %73)
-> (%74)
}

it depends, but it is most likely coming from autodiff here:

@zou3519 Thanks for the tip. It looks like if I re-arrange the line :
return {grads.at(0) * outputs.at(0) * (1 - outputs.at(0))};
to
return {((1 - outputs.at(0)) * outputs.at(0) * grads.at(0))};

And re-run the : python test/test_jit.py TestScript.test_milstm_fusion_cuda --accept
both expect files (forward and backward / ppc64 and x86_64) are now identical.

Can you please let me know if this is OK to do.

I have no idea why I have to re-arrange it to give me the same backward expect output.

@ezyang @zou3519 When you get the chance, please review suggested changes.

I tested the proposed solution on a ppc64le and it works for me.

@zou3519 Thanks for approving, was there something else that needs to be done before this can be merged. It still says merging is blocked