This repository was archived by the owner on May 6, 2020. It is now read-only.
This repository was archived by the owner on May 6, 2020. It is now read-only.
unable to setup network inside a pod when launching them with k8s #182
Description
Cannot launch pods using k8s and a lot of qemu and cc-proxies are left behind, since k8s retries to launch the pods.
I see this error in the cc-proxy log:
Could not setup the network: Could not setup network routes: Could not add/replace route dest(10.244.0.0/16)/src()/gw(10.244.0.1)/dev(eth0): network is unreachable
This started to happen after updating clear-containers image from 19530 to 19490. The only change between those images is: e30849c
Logs:
Dec 13 22:40:45 k8s-ok cc-proxy[80534]: time="2017-12-13T22:40:45.183542349Z" level=debug msg="Started Clear Linux Project for Intel Architecture power tweaks." name=cc-proxy pid=80534 source=qemu vm=ae9a235c61c
77e4d1533cb3bf5866794e8c9b0473260cd9afb1af0cc5cd2d63b
Dec 13 22:40:45 k8s-ok cc-proxy[80534]: time="2017-12-13T22:40:45.185319997Z" level=debug msg="{\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"startpod_start\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":129,\\\"tim
e\\\":\\\"2017-12-13T22:40:44.447623435Z\\\"}" name=cc-proxy pid=80534 source=qemu vm=ae9a235c61c77e4d1533cb3bf5866794e8c9b0473260cd9afb1af0cc5cd2d63b
Dec 13 22:40:45 k8s-ok cc-proxy[80534]: time="2017-12-13T22:40:45.207363988Z" level=debug msg="[\\x1b[0;32m OK \\x1b[0m] Started Update UTMP about System Boot/Shutdown." name=cc-proxy pid=80534 source=qemu vm=
ae9a235c61c77e4d1533cb3bf5866794e8c9b0473260cd9afb1af0cc5cd2d63b
Dec 13 22:40:45 k8s-ok cc-proxy[80534]: time="2017-12-13T22:40:45.225917685Z" level=debug msg="{\\\"interface-name\\\":\\\"eth0\\\",\\\"level\\\":\\\"info\\\",\\\"mac-address\\\":\\\"3e:fe:e7:c2:45:0d\\\",\\\"ms
g\\\":\\\"Getting interface from MAC address\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":129,\\\"time\\\":\\\"2017-12-13T22:40:44.487935468Z\\\"}" name=cc-proxy pid=80534 source=qemu vm=ae9a235c61c77e4d1533cb3
bf5866794e8c9b0473260cd9afb1af0cc5cd2d63b
Dec 13 22:40:45 k8s-ok cc-proxy[80534]: time="2017-12-13T22:40:45.228145945Z" level=debug msg="{\\\"interface-name\\\":\\\"eth0\\\",\\\"level\\\":\\\"info\\\",\\\"link\\\":\\\"\\\\u0026{LinkAttrs:{Index:2 MTU:15
00 TxQLen:1000 Name:eth0 HardwareAddr:3e:fe:e7:c2:45:0d Flags:broadcast|multicast RawFlags:4098 ParentIndex:0 MasterIndex:0 Namespace:\\\\u003cnil\\\\u003e Alias: Statistics:0xc4200f65b0 Promisc:0 Xdp:\\\\u003cn
il\\\\u003e EncapType:ether Protinfo:\\\\u003cnil\\\\u003e OperState:down}}\\\",\\\"mac-address\\\":\\\"3e:fe:e7:c2:45:0d\\\",\\\"msg\\\":\\\"Link found\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":129,\\\"time
\\\":\\\"2017-12-13T22:40:44.490209379Z\\\"}" name=cc-proxy pid=80534 source=qemu vm=ae9a235c61c77e4d1533cb3bf5866794e8c9b0473260cd9afb1af0cc5cd2d63b
Dec 13 22:40:45 k8s-ok cc-proxy[80534]: time="2017-12-13T22:40:45.245570312Z" level=debug msg="{\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"Route destination already exists, deleting\\\",\\\"name\\\":\\\"cc-agent
\\\",\\\"pid\\\":129,\\\"route-destination\\\":\\\"10.244.0.0/24\\\",\\\"time\\\":\\\"2017-12-13T22:40:44.507565961Z\\\"}" name=cc-proxy pid=80534 source=qemu vm=ae9a235c61c77e4d1533cb3bf5866794e8c9b0473260cd9af
b1af0cc5cd2d63b
Dec 13 22:40:45 k8s-ok cc-proxy[80534]: time="2017-12-13T22:40:45.247042352Z" level=debug msg="{\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"startpod_end\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":129,\\\"time\
\\":\\\"2017-12-13T22:40:44.508987937Z\\\"}" name=cc-proxy pid=80534 source=qemu vm=ae9a235c61c77e4d1533cb3bf5866794e8c9b0473260cd9afb1af0cc5cd2d63b
Dec 13 22:40:45 k8s-ok cc-proxy[80534]: time="2017-12-13T22:40:45.248181982Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"startpod\\\",\\\"error\\\":\\\"Could not setup the network: Could n
ot setup network routes: Could not add/replace route dest(10.244.0.0/16)/src()/gw(10.244.0.1)/dev(eth0): network is unreachable\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"
cc-agent\\\",\\\"pid\\\":129,\\\"time\\\":\\\"2017-12-13T22:40:44.510000189Z\\\"}" name=cc-proxy pid=80534 source=qemu vm=ae9a235c61c77e4d1533cb3bf5866794e8c9b0473260cd9afb1af0cc5cd2d63b
Dec 13 22:40:45 k8s-ok cc-proxy[80534]: time="2017-12-13T22:40:45.249930129Z" level=info msg="connection closed" client=3 name=cc-proxy pid=80534 source=proxy
Dec 13 22:40:45 k8s-ok cc-proxy[80534]: time="2017-12-13T22:40:45.271293902Z" level=debug msg="[\\x1b[0;32m OK \\x1b[0m] Reached target Clear Containers Agent Target." name=cc-proxy pid=80534 source=qemu vm=ae
9a235c61c77e4d1533cb3bf5866794e8c9b0473260cd9afb1af0cc5cd2d63b
Dec 13 22:40:46 k8s-ok cc-proxy[74758]: time="2017-12-13T22:40:46.691093754Z" level=debug msg="[ 281.393913] systemd-journald[103]: Sent WATCHDOG=1 notification." name=cc-proxy pid=74758 source=qemu vm=e3e411a4
24b54d1a522e8d25ccd5dce2922aa25f870e0a676ecbc554f855da8d
Dec 13 22:40:47 k8s-ok cc-proxy[78979]: time="2017-12-13T22:40:47.691458557Z" level=debug msg="[ 114.607556] systemd-journald[100]: Sent WATCHDOG=1 notification." name=cc-proxy pid=78979 source=qemu vm=46778b1a
4f5c56c2c618d939d45d43683d9c7024e2f5465f15d527350d838ce2
Dec 13 22:40:48 k8s-ok cc-proxy[80385]: time="2017-12-13T22:40:48.678638401Z" level=error msg="timeout waiting for process with token 4wZCKKnaiyUnWM8Jz1LeF9mx6HtCnEwPp-s4nDjaWo0=" name=cc-proxy pid=80385 section
=io source=proxy vm=09493b609d513670780b8b08fba863451698b041fc206a0ca755d17b0b031ede
Dec 13 22:40:48 k8s-ok cc-proxy[80385]: time="2017-12-13T22:40:48.678762004Z" level=debug msg="lost the shim for %s4wZCKKnaiyUnWM8Jz1LeF9mx6HtCnEwPp-s4nDjaWo0=" name=cc-proxy pid=80385 section=io source=proxy vm
=09493b609d513670780b8b08fba863451698b041fc206a0ca755d17b0b031ede
Dec 13 22:40:48 k8s-ok cc-proxy[80385]: time="2017-12-13T22:40:48.67896831Z" level=error msg="error serving client: timeout waiting for process with token 4wZCKKnaiyUnWM8Jz1LeF9mx6HtCnEwPp-s4nDjaWo0=" client=2 n
ame=cc-proxy pid=80385 source=proxy