Rebooted, dangling file in /var/lib/cni/networks/podman prevents container starting #3759
Description
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
I did a systemctl reboot; encountered a orphan /var/lib/cni/networks/podman/10.88.0.20 file
which prevented the container from starting.
ERRO[0000] Error adding network: failed to allocate for range 0: requested IP address 10.88.0.20 is not available in range set 10.88.0.1-10.88.255.254
ERRO[0000] Error while adding pod to CNI network "podman": failed to allocate for range 0: requested IP address 10.88.0.20 is not available in range set 10.88.0.1-10.88.255.254
Error: unable to start container "freeswitch-init_1": error configuring network namespace for container 7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7: failed to allocate for range 0: requested IP address 10.88.0.20 is not available in range set 10.88.0.1-10.88.255.254
Steps to reproduce the issue:
-
Upgraded Fedora 30 host, rebooted, tried to start a container
Describe the results you received:
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist
DEBU[0000] Made network namespace at /var/run/netns/cni-76dce0ef-599f-b625-7710-a7eeef4159e5 for container 7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7
INFO[0000] Got pod network &{Name:freeswitch-init_1 Namespace:freeswitch-init_1 ID:7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7 NetNS:/var/run/netns/cni-76dce0ef-599f-b625-7710-a7eeef4159e5 PortMappings:[] Networks:[podman] NetworkConfig:map[podman:{IP:10.88.0.20}]}
INFO[0000] About to add CNI network cni-loopback (type=loopback)
DEBU[0000] overlay: mount_data=nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/Z6S3A3PN6PO5DCD4E3FZZOFLO6:/var/lib/containers/storage/overlay/l/II26Q5SUSAWGOVHLKY3MKC7GNU,upperdir=/var/lib/containers/storage/overlay/8cc72886f3925230caa200d3bce8b24599897b0c64ca5a6a4d6d56871f118d4d/diff,workdir=/var/lib/containers/storage/overlay/8cc72886f3925230caa200d3bce8b24599897b0c64ca5a6a4d6d56871f118d4d/work,context="system_u:object_r:container_file_t:s0:c362,c945"
DEBU[0000] mounted container "7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7" at "/var/lib/containers/storage/overlay/8cc72886f3925230caa200d3bce8b24599897b0c64ca5a6a4d6d56871f118d4d/merged"
DEBU[0000] Created root filesystem for container 7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7 at /var/lib/containers/storage/overlay/8cc72886f3925230caa200d3bce8b24599897b0c64ca5a6a4d6d56871f118d4d/merged
INFO[0000] Got pod network &{Name:freeswitch-init_1 Namespace:freeswitch-init_1 ID:7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7 NetNS:/var/run/netns/cni-76dce0ef-599f-b625-7710-a7eeef4159e5 PortMappings:[] Networks:[podman] NetworkConfig:map[podman:{IP:10.88.0.20}]}
INFO[0000] About to add CNI network podman (type=bridge)
ERRO[0000] Error adding network: failed to allocate for range 0: requested IP address 10.88.0.20 is not available in range set 10.88.0.1-10.88.255.254
ERRO[0000] Error while adding pod to CNI network "podman": failed to allocate for range 0: requested IP address 10.88.0.20 is not available in range set 10.88.0.1-10.88.255.254
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] unmounted container "7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7"
DEBU[0000] Cleaning up container 7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Container 7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7 storage is already unmounted, skipping...
ERRO[0000] unable to start container "freeswitch-init_1": error configuring network namespace for container 7690a7bc4960b76799d906dbc07a32d174a5e75581886349d3604d455e093cf7: failed to allocate for range 0: requested IP address 10.88.0.20 is not available in range set 10.88.0.1-10.88.255.254
Describe the results you expected:
Container starts
Additional information you deem important (e.g. issue happens only occasionally):
Output of podman version:
Version: 1.4.4
RemoteAPI Version: 4
Go Version: go1.12.7
OS/Arch: linux/amd64
Output of podman info --debug:
debug:
compiler: gc
git commit: ""
go version: go1.12.7
podman version: 1.4.4
host:
BuildahVersion: 1.9.0
Conmon:
package: podman-1.4.4-4.fc30.x86_64
path: /usr/libexec/podman/conmon
version: 'conmon version 1.0.0-dev, commit: 164df8af4e62dc759c312eab4b97ea9fb6b5f1fc'
Distribution:
distribution: fedora
version: "30"
MemFree: 7664599040
MemTotal: 8340746240
OCIRuntime:
package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
path: /usr/bin/runc
version: |-
runc version 1.0.0-rc8+dev
commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
spec: 1.0.1-dev
SwapFree: 4294963200
SwapTotal: 4294963200
arch: amd64
cpus: 4
hostname: containers.localdomain
kernel: 5.2.5-200.fc30.x86_64
os: linux
rootless: false
uptime: 7m 27.03s
registries:
blocked: null
insecure: null
search:
- docker.io
- registry.fedoraproject.org
- quay.io
- registry.access.redhat.com
- registry.centos.org
store:
ConfigFile: /etc/containers/storage.conf
ContainerStore:
number: 3
GraphDriverName: overlay
GraphOptions:
- overlay.mountopt=nodev,metacopy=on
GraphRoot: /var/lib/containers/storage
GraphStatus:
Backing Filesystem: xfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "true"
ImageStore:
number: 6
RunRoot: /var/run/containers/storage
VolumePath: /var/lib/containers/storage/volumes
Additional environment details (AWS, VirtualBox, physical, etc.):
In /var/lib/cni/networks/podman I have:
10.88.0.20: contains the ID of the containerlast_reserved_ip.0: contains10.88.0.20lock
The guard file 10.88.0.20 must have been left behind from the reboot. After deleting the file, the container could start.