@ducha-aiki , as per #3229 (comment)
Attached

1. Sample output when this layer is used in CPU and GPU
2. Prototxt file used (caffenet +bn +ea)

eltwiseaffine_gpu_bug.txt
train_val_prototxt.txt

@siddharthm83 Thanks, I will take a look on it.

@siddharthm83, I have experienced it too in GPU mode, but after restarting training, it has gone. Don`t know reason for it yet...

Logs are here:
https://github.com/ducha-aiki/batchnorm-benchmark/blob/master/logs/caffenet128_lsuv_no_lrn_BatchNormAfterReLU_EA_start.log
https://github.com/ducha-aiki/batchnorm-benchmark/blob/master/logs/caffenet128_lsuv_no_lrn_BatchNormBeforeReLU_EltwiseAffine_start.log

Prototxt are here:
https://github.com/ducha-aiki/batchnorm-benchmark/blob/master/prototxt/caffenet128_lsuv_no_lrn_BatchNormAfterReLU_EA.prototxt
https://github.com/ducha-aiki/batchnorm-benchmark/blob/master/prototxt/caffenet128_lsuv_no_lrn_BatchNormBeforeReLU_EltwiseAffine.prototxt

@ducha-aiki , the log link seems broken. When you say restarting training, it is gone, what do you exactly mean (are you restarting with a saved solverstate?)

@siddharthm83 sorry, filenames changed after repo update.

But it does`t matter - I have done 100 restarts in script and it seems code is broken - it recovers only 2 times :( So time for debugging for me
By restart I mean start from same initialization - I initialize net in python and save weights.

@ducha-aiki , any luck in identifying the issue? No hurry, just checking.

@siddharthm83 sorry, ICLR rebuttal deadline. Going to fix next week.

@ducha-aiki , I think here might be the problem, not CblasTrans but CblasNoTrans.

// Gradient with respect to bias
for (int n = 0; n < num; ++n) {
  caffe_gpu_gemv<Dtype>(CblasNoTrans, channels, height * width, (Dtype)1.,
      top_diff + top[0]->offset(n), bias_multiplier_.gpu_data(), (Dtype)1.,
      bias_diff);
}

@wuzifeng thanks! Looks like it was the cause. I will finish training caffenet with it and update PR.

awesome, thanks @ducha-aiki and @wuzifeng

Travis crashes before it tries to compile anything, on stage of downloading packages.

Replaced and merged in #3591. Thanks again for this work and your benchmarks @ducha-aiki!