pfnCI, test this please.

Jenkins CI test (for commit 055ceb9, target branch master) failed with status FAILURE.

Some numpy-scipy version issues

pfnCI, test this please.

Jenkins CI test (for commit 6e60f6c, target branch master) failed with status FAILURE.

pfnCI, test this please.

pfnCI, test this please.

Jenkins CI test (for commit 2f4970c, target branch master) failed with status FAILURE.

NCCL_ERROR 🤔

Jenkins, test this please

Jenkins CI test (for commit 2f4970c, target branch master) failed with status FAILURE.

pfnCI, test this please.

Jenkins CI test (for commit d712d44, target branch master) failed with status FAILURE.

Please Fix CIs

Sorry for taking time, I'm still unable to reproduce NCCL_ERROR_UNHANDLED_CUDA_ERROR: unhandled cuda error locally. Will try reproducing it on Jenkins interactively.
Jenkins test this please.

Jenkins CI test (for commit 92cfd88, target branch master) failed with status FAILURE.

Hi, was directed to here from cupy/cupy#1941 to see if a pair of fresh eyes help. Just throwing some random thoughts (no experience with this error):

1. Any chance Chainer is doing multi-threading on top of multiprocessing? According to NCCL release note (see here):

Using multiple processes in conjunction with multiple threads to manage the different GPUs may in some cases cause ncclCommInitRank to fail while establishing IPCs (cudaIpcOpenMemHandle). This problem does not appear when using only processes or only threads. This issue is fixed in recent driver versions, therefore, consider updating to the latest drive

2. Based on 1, it could also be that the driver/NCCL version in the Jenkins test is too old?
3. How about adding –shm-size=1g –ulimit memlock=-1 to nvidia-docker run? It's recommended in NCCL's doc.
4. A broken GPU could cause ncclCommInitRank to raise ncclUnhandledCudaError: RuntimeError: NCCL Error 1: unhandled cuda error pytorch/pytorch#11756.

can this be related to chainer/chainer#7511?

I don't think it is, @kmaehashi was able to reproduce the bug locally so we will know soon :).
It sounds mostly like some kind of environment misconfiguration.

Cool!

pfnCI, test this please.

Jenkins CI test (for commit 92cfd88, target branch master) failed with status FAILURE.

If fixing the failure is difficult for now, how about skipping the test case for the newly added configuration as a tentative workaround? This change affects a wide range of development and, IMHO, making them proceed quickly should have higher priority than fixing the failure cleanly.

How about giving a lucky id to Jenkins?

./run_combination_test.py --id 1

The current id (1) is lucky but becomes unlucky after the PR. FYI, the last daily_master_chainer-shuffled passes with 10 combinations out of 21.

What's the status?
It should not be difficult to skip affected tests as suggested by @beam2d.

Changed the test variation as suggested by @toslunar to avoid this situation (until we migrate Jenkins to new test infrastructure).
pfnCI, test this please.

Jenkins CI test (for commit 92cfd88, target branch master) failed with status FAILURE.

I'll test separately to find the proper combination that can pass the test.

pfnCI, test this please.
(--id 4 passed the test)

Jenkins CI test (for commit 92cfd88, target branch master) failed with status FAILURE.

Unlucky id 😂

Ah, sorry configuration was not updated correctly...
pfnCI, test this please.

Jenkins CI test (for commit 92cfd88, target branch master) failed with status FAILURE.

The failure is not related to this PR (will be fixed in #532).

Jenkins, test this please.

Jenkins CI test (for commit 92cfd88, target branch master) succeeded!