We've received several reports that jobs running in GitLab CI and using the Docker-in-Docker approach to build images started to hang during execution. Some examples can be found at:

• https://gitlab.com/gitlab-org/gitlab-runner/issues/6697
• https://gitlab.com/gitlab-com/support-forum/issues/5194

Please notice, that the problem is not on pulling the gitlab/gitlab-runner-helper image, which is done properly and is part of a post build step after the hanging job was terminated. The failure happens when a Docker image is being build, using the Docker-in-Docker service from docker:dind image.

Reviewing all of the cases we've found that they all are caused by using docker:dind or doker:19.03-dind, so the floating versions pointing to docker:19.03.6-dind which was released several hours ago. The time of the image release aligns with all of the reports that we've received.

The only change in the image, since the last docker:19.03.5-dind push (which works!) was done, is the automatic update from Docker 19.03.5 to Docker 19.03.6 (3dc42d4). Whatever is causing the problems, it's in Docker itself.

Leaving this as a note and a workaround description for all of you who may look here to find out problems with GitLab CI and Docker-in-Docker, which was happening in the past :)

So, if you think you're affected by this issue, set the used image explicitly to docker:19.03.5-dind.

Also... maybe it's worth to revert the release of docker:19.03.6-dind until it's fixed?