This repository was archived by the owner on May 12, 2021. It is now read-only.
This repository was archived by the owner on May 12, 2021. It is now read-only.
vsock - context deadline exceed on create command. #1319
Description
Description of problem
The vsocks job has started to fail randomly when running the docker-stability tests with the next error:
11:10:00 docker: Error response from daemon: OCI runtime create failed: Failed to check if grpc server is working: context deadline exceeded: unknown.
http://jenkins.katacontainers.io/job/kata-containers-tests-fedora-vsocks-PR/1004/console
http://jenkins.katacontainers.io/job/kata-containers-runtime-fedora-vsocks-PR/323/
From the kata-runtime log:
Mar 04 12:23:12 fedora-28-51.vexxhost.local kata-runtime[32372]: time="2019-03-04T12:23:12.295360953Z" level=info msg="VM started" arch=amd64 command=create container=4132a20fe71671403cc16d8de9745d16695d77b209f8
fcddd17e97438b3d1a6a name=kata-runtime pid=32372 sandbox=4132a20fe71671403cc16d8de9745d16695d77b209f8fcddd17e97438b3d1a6a source=virtcontainers subsystem=sandbox
Mar 04 12:23:12 fedora-28-51.vexxhost.local kata-runtime[32372]: time="2019-03-04T12:23:12.295732044Z" level=debug msg="No proxy started because of no-proxy implementation" arch=amd64 command=create container=41
32a20fe71671403cc16d8de9745d16695d77b209f8fcddd17e97438b3d1a6a name=kata-runtime pid=32372 sandbox=4132a20fe71671403cc16d8de9745d16695d77b209f8fcddd17e97438b3d1a6a source=virtcontainers subsystem=kata_agent
Mar 04 12:23:12 fedora-28-51.vexxhost.local kata-runtime[32372]: time="2019-03-04T12:23:12.296456793Z" level=info msg="proxy started" arch=amd64 command=create container=4132a20fe71671403cc16d8de9745d16695d77b20
9f8fcddd17e97438b3d1a6a name=kata-runtime pid=32372 proxy-pid=0 proxy-url="vsock://2202419104:1024" sandbox=4132a20fe71671403cc16d8de9745d16695d77b209f8fcddd17e97438b3d1a6a source=virtcontainers subsystem=kata_a
gent
Mar 04 12:23:12 fedora-28-51.vexxhost.local kata-runtime[32372]: time="2019-03-04T12:23:12.296808274Z" level=info msg="New client" arch=amd64 command=create container=4132a20fe71671403cc16d8de9745d16695d77b209f8
fcddd17e97438b3d1a6a name=kata-runtime pid=32372 source=virtcontainers subsystem=kata_agent url="vsock://2202419104:1024"
Mar 04 12:23:27 fedora-28-51.vexxhost.local kata-runtime[32372]: time="2019-03-04T12:23:27.2979244Z" level=info msg="Stopping netmon" arch=amd64 command=create container=4132a20fe71671403cc16d8de9745d16695d77b20
9f8fcddd17e97438b3d1a6a name=kata-runtime netmon-pid=32399 netmon-signal=killed pid=32372 source=virtcontainers subsystem=netmon
Mar 04 12:23:27 fedora-28-51.vexxhost.local kata-runtime[32372]: time="2019-03-04T12:23:27.301006349Z" level=info msg="Detaching endpoint" arch=amd64 command=create container=4132a20fe71671403cc16d8de9745d16695d
77b209f8fcddd17e97438b3d1a6a endpoint-type=virtual hotunplug=false name=kata-runtime pid=32372 source=virtcontainers subsystem=network
Mar 04 12:23:27 fedora-28-51.vexxhost.local kata-runtime[32372]: time="2019-03-04T12:23:27.324078004Z" level=debug msg="Network removed" arch=amd64 command=create container=4132a20fe71671403cc16d8de9745d16695d77
b209f8fcddd17e97438b3d1a6a name=kata-runtime pid=32372 source=virtcontainers subsystem=network
Mar 04 12:23:27 fedora-28-51.vexxhost.local kata-runtime[32372]: time="2019-03-04T12:23:27.324286965Z" level=info msg="Network namespace \"/var/run/netns/cni-b0b89686-ad80-d13a-63a4-043a889b757c\" deleted" arch=
amd64 command=create container=4132a20fe71671403cc16d8de9745d16695d77b209f8fcddd17e97438b3d1a6a name=kata-runtime pid=32372 source=virtcontainers subsystem=network
Mar 04 12:23:27 fedora-28-51.vexxhost.local kata-runtime[32372]: time="2019-03-04T12:23:27.333972528Z" level=debug msg="Deleting sandbox cgroup" arch=amd64 command=create container=4132a20fe71671403cc16d8de9745d
16695d77b209f8fcddd17e97438b3d1a6a name=kata-runtime pid=32372 sandbox=4132a20fe71671403cc16d8de9745d16695d77b209f8fcddd17e97438b3d1a6a source=virtcontainers subsystem=sandbox
Mar 04 12:23:27 fedora-28-51.vexxhost.local kata-runtime[32372]: time="2019-03-04T12:23:27.334232954Z" level=debug msg="Deleting no constraints cgroup" arch=amd64 command=create container=4132a20fe71671403cc16d8
de9745d16695d77b209f8fcddd17e97438b3d1a6a name=kata-runtime path=/kata pid=32372 sandbox=4132a20fe71671403cc16d8de9745d16695d77b209f8fcddd17e97438b3d1a6a source=virtcontainers subsystem=sandbox
Mar 04 12:23:27 fedora-28-51.vexxhost.local kata-runtime[32372]: time="2019-03-04T12:23:27.363404966Z" level=info msg="cleanup agent" arch=amd64 command=create container=4132a20fe71671403cc16d8de9745d16695d77b20
9f8fcddd17e97438b3d1a6a name=kata-runtime path=/run/kata-containers/shared/sandboxes/4132a20fe71671403cc16d8de9745d16695d77b209f8fcddd17e97438b3d1a6a pid=32372 source=virtcontainers subsystem=kata_agent
Mar 04 12:23:27 fedora-28-51.vexxhost.local kata-runtime[32372]: time="2019-03-04T12:23:27.36400354Z" level=debug msg="Deleting files" arch=amd64 backend=filesystem command=create container=4132a20fe71671403cc16
d8de9745d16695d77b209f8fcddd17e97438b3d1a6a name=kata-runtime path=/var/lib/vc/sbs/4132a20fe71671403cc16d8de9745d16695d77b209f8fcddd17e97438b3d1a6a pid=32372 source=virtcontainers/store subsystem=store
Mar 04 12:23:27 fedora-28-51.vexxhost.local kata-runtime[32372]: time="2019-03-04T12:23:27.364616227Z" level=debug msg="Deleting files" arch=amd64 backend=filesystem command=create container=4132a20fe71671403cc1
6d8de9745d16695d77b209f8fcddd17e97438b3d1a6a name=kata-runtime path=/run/vc/sbs/4132a20fe71671403cc16d8de9745d16695d77b209f8fcddd17e97438b3d1a6a pid=32372 source=virtcontainers/store subsystem=store
Mar 04 12:23:27 fedora-28-51.vexxhost.local kata-runtime[32372]: time="2019-03-04T12:23:27.365064884Z" level=error msg="Failed to check if grpc server is working: context deadline exceeded" arch=amd64 command=cr
eate container=4132a20fe71671403cc16d8de9745d16695d77b209f8fcddd17e97438b3d1a6a name=kata-runtime pid=32372 source=runtime