@apaszke, this is causing a ~200ms latency regression for NMT models. Here are the top lines from perf before/after this change:

Before:

• 43.31% NmtBenchmark NmtBenchmark [.] mkl_blas_avx2_sgemm_kernel_nocopy_NN_b0
• 8.67% NmtBenchmark NmtBenchmark [.] loop_inner866
• 4.42% NmtBenchmark libc-2.23.so [.] __memcpy_avx_unaligned
• 3.20% NmtBenchmark NmtBenchmark [.] caffe2::math::(anonymous namespace)::ReduceTensor<float, std::plus >
• 3.04% NmtBenchmark NmtBenchmark [.] caffe2::math::utils::GetIndexFromDims
• 2.84% NmtBenchmark NmtBenchmark [.] mkl_blas_avx2_xsgemv_t
• 2.84% NmtBenchmark NmtBenchmark [.] fbgemm::PackedGemmMatrix<signed char, int, true>::addr_
• 2.76% CaffeTaskThread NmtBenchmark [.] mkl_blas_avx2_xsgemv_t

After:

• 29.04% NmtBenchmark NmtBenchmark [.] mkl_blas_avx2_sgemm_kernel_nocopy_NN_b0
• 17.97% NmtBenchmark NmtBenchmark [.] mkl_blas_avx2_sgemm_kernel_nocopy_TN_b1
• 8.02% NmtBenchmark NmtBenchmark [.] mkl_blas_avx2_sgemm_kernel_nocopy_TN_b0
• 5.95% CaffeTaskThread NmtBenchmark [.] mkl_blas_avx2_xsgemv_t
• 4.28% NmtBenchmark libc-2.23.so [.] __memcpy_avx_unaligned

Is there more info I can provide to help debug this? Also cc @jamesr66a who is familiar with these models and how to optimize them.

@jmp84 can you please provide me with some example tensor sizes that appear as inputs to your GEMMs? It looks like we started triggering the transposed kernels in some cases which was not the case previously (TN vs NN).

Finally, how do you run those models? Is that via ONNX export or what?

@apaszke I think what's going on here is that we are not hitting the special quantized implementation of FC in the caffe2 backend due to the ONNX-caffe2 backend emitting "MatMul" + "Add" instead of "FC". This can be seen from loop_inner866 being present before but not after this patch. (That label is from an autogenerated GEMM kernel). I'm going to send in a patch that adds addmm fusion on the ONNX export path, but not the regular path

So for the addmm optimization - “not helpful at all” means neutral or negative? Of neutral - why not kee it if it matters for ONNX export?

It means negative in some RNN use cases I saw