Error response from daemon: configuration network 8-2 is in use #35101
Description
Description
It seems when an IPAM driver fails and the container is removed the network that used the IPAM driver becomes stuck and can no longer be removed. rm -rf /var/lib/docker is necessary (unless some way to edit the .db file?).
Steps to reproduce the issue:
- docker network create -d macvlan --subnet 8.2.250.0/24 --gateway 8.2.250.3 -o parent=public --ipam-driver arp-ipam3 --config-only 8-2
- docker network create -d macvlan --scope swarm --config-from 8-2 --attachable public-8-2
- docker service create ... --network public-8-2 ... my-service
- ipam driver crashes during RequestAddress.
- service create is in pending state for a long time.
- CTRL-C
- docker service rm my-service. Succeeds.
- docker network rm public-8-2. Succeeds but errors in logs that network doesn't exist?
- docker network rm 8-2. This produces the titled error. Only way to remove network now is to rm -rf /var/lib/docker/
Describe the results you received:
Cannot remove network.
Describe the results you expected:
Network should be removed.
Additional information you deem important (e.g. issue happens only occasionally):
Output of docker version:
Client:
Version: 17.09.0-ce
API version: 1.32
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:42:18 2017
OS/Arch: linux/amd64
Server:
Version: 17.09.0-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:40:56 2017
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 2
Server Version: 17.09.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: vz74i89jega7k0ccyekpplgwq
Is Manager: true
ClusterID: qrpx9z4664ts79a6eamvrrxnw
Managers: 1
Nodes: 2
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.0.21.39
Manager Addresses:
10.0.21.39:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.11.0-14-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 40
Total Memory: 141.6GiB
Name: container1
ID: IMGD:TF6D:E4FH:AT2H:SZ5C:YNIU:IB57:ATV7:Y6KU:JBIG:Z5PF:ZBMF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Additional environment details (AWS, VirtualBox, physical, etc.):
Logs:
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.RequestAddress: Post http://8.2.250.40:8181/IpamDriver.RequestAddress: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 2s"
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.RequestAddress: Post http://8.2.250.40:8181/IpamDriver.RequestAddress: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 4s"
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.RequestAddress: Post http://8.2.250.40:8181/IpamDriver.RequestAddress: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 8s"
level=info msg="Node join event for container2-ba0d08b08afe/10.0.21.40"
level=error msg="fatal task error" error="starting container failed: Post http://8.2.250.40:8181/IpamDriver.RequestAddress: dial tcp 8.2.250.40:8181: getsockopt: connection refused" module="node/agent/taskmanager" node.id=vz74i89jega7k0ccyekpplgwq service.id=scogq490jtph6f5ahigrmvxl7 task.id=f97hf1oyokft44dbusddw3os1
level=warning msg="underweighting node vz74i89jega7k0ccyekpplgwq for service scogq490jtph6f5ahigrmvxl7 because it experienced 5 failures or rejections within 5m0s" module=node node.id=vz74i89jega7k0ccyekpplgwq
level=warning msg="underweighting node awvo1u8zp17i30jb0tze4cebj for service scogq490jtph6f5ahigrmvxl7 because it experienced 5 failures or rejections within 5m0s" module=node node.id=vz74i89jega7k0ccyekpplgwq
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.ReleaseAddress: Post http://8.2.250.40:8181/IpamDriver.ReleaseAddress: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 1s"
level=info msg="Node join event for container2-ba0d08b08afe/10.0.21.40"
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.ReleaseAddress: Post http://8.2.250.40:8181/IpamDriver.ReleaseAddress: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 2s"
level=error msg="fatal task error" error="starting container failed: No such network: public-8-2" module="node/agent/taskmanager" node.id=vz74i89jega7k0ccyekpplgwq service.id=scogq490jtph6f5ahigrmvxl7 task.id=y3w4ai1tyirxtqumrp98bzadv
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.ReleaseAddress: Post http://8.2.250.40:8181/IpamDriver.ReleaseAddress: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 4s"
level=error msg="network public-8-2 remove failed: No such network: public-8-2" module="node/agent" node.id=vz74i89jega7k0ccyekpplgwq
level=error msg="remove task failed" error="No such network: public-8-2" module="node/agent" node.id=vz74i89jega7k0ccyekpplgwq task.id=njtqop3cb81v080k7f4lg5x88
level=error msg="network public-8-2 remove failed: No such network: public-8-2" module="node/agent" node.id=vz74i89jega7k0ccyekpplgwq
level=error msg="remove task failed" error="No such network: public-8-2" module="node/agent" node.id=vz74i89jega7k0ccyekpplgwq task.id=jizmlx0lgbmdo5bjg5ae2dkfs
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.ReleaseAddress: Post http://8.2.250.40:8181/IpamDriver.ReleaseAddress: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 8s"
level=warning msg="Failed to release gateway ip address 8.2.250.3 on delete of network public-8-2 (8jcmhqcq34jaeoa8qy2iguiw2): Post http://8.2.250.40:8181/IpamDriver.ReleaseAddress: dial tcp 8.2.250.40:8181: getsockopt: connection refused"
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.ReleasePool: Post http://8.2.250.40:8181/IpamDriver.ReleasePool: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 1s"
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.ReleasePool: Post http://8.2.250.40:8181/IpamDriver.ReleasePool: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 2s"
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.ReleasePool: Post http://8.2.250.40:8181/IpamDriver.ReleasePool: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 4s"
level=info msg="Node join event for container2-ba0d08b08afe/10.0.21.40"
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.ReleasePool: Post http://8.2.250.40:8181/IpamDriver.ReleasePool: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 8s"
level=warning msg="Failed to release address pool 8.2.250.0/24 on delete of network public-8-2 (8jcmhqcq34jaeoa8qy2iguiw2): Post http://8.2.250.40:8181/IpamDriver.ReleasePool: dial tcp 8.2.250.40:8181: getsockopt: connection refused"
level=warning msg="Unable to connect to plugin: 8.2.250.40:8181/IpamDriver.RequestPool: Post http://8.2.250.40:8181/IpamDriver.RequestPool: dial tcp 8.2.250.40:8181: getsockopt: connection refused, retrying in 1s"