Hi Yangqing,

We met problem when running Caffe. We run Caffe following the step on http://caffe.berkeleyvision.org/imagenet.html for ImageNet training. Our training set is 1000 images selected from the whole imagenet dataset and the testing set is the same.
It crashes after 600 iterations, the error message is:
“F0118 21:19:41.088841 1589 padding_layer.cu:131] Cuda kernel failed. Error: unspecified launch failure”
We run it again and it still crashes after 740 iterators, with error message:
“F0118 20:57:41.093628 27945 math_functions.cpp:45] Check failed: (cublasSgemm_v2(Caffe::cublas_handle(), cuTransB, cuTransA, N, M, K, &alpha, B, ldb, A, lda, &beta, C, N)) == CUBLAS_STATUS_SUCCESS (14 vs. 0)”

Our testbed is Ubuntu 12.04 LTS with GTX Titan, CUDA 5.5

Do you have any idea what causes this crash ? Thank you very much.

Best regards,
Minjie