Rootless containers reset connection #7016
Description
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
Rootless containers default network mode is unstable. When I run a webserver inside a rootless container and try to perform multiple HTTP requests by curl I often get curl: (56) Recv failure: Connection reset by peer. After around 2 minutes any attempts to establish a connection to the port are hung. On the other hand, rootful containers or host network mode don't cause the issue. This suggests that the issue is related to slirp4netns which is a default network mode for rootless containers.
Steps to reproduce the issue:
-
podman run -d -p 8080:80 nginx:alpine -
while true; do curl -L http://127.0.0.1:8080; done
Describe the results you received:
Periodically in the output, I can see:
curl: (56) Recv failure: Connection reset by peer
Or in TCP:
46172 88.092580136 127.0.0.1 127.0.0.1 TCP 66 8080 → 39888 [RST, ACK] Seq=1 Ack=79 Win=65536 Len=0 TSval=2905973286 TSecr=2905973097
In the end, the connection cannot be finished:
Describe the results you expected:
The connection shouldn't be reset.
Additional information you deem important (e.g. issue happens only occasionally):
I attached a Wireshark capture log of the loopback interface.
podman_network_bug.pcapng.gz
The issue is not seen if the network mode is host or podman runs under root. I tried to test the issue on podman 1.6.2 and it's not reproducible there. It seems it's a regression.
Output of podman version:
Version: 2.0.2
API Version: 1
Go Version: go1.14.4
Built: Thu Jan 1 01:00:00 1970
OS/Arch: linux/amd64
Output of podman info --debug:
host:
arch: amd64
buildahVersion: 1.15.0
cgroupVersion: v1
conmon:
package: 'conmon: /usr/libexec/podman/conmon'
path: /usr/libexec/podman/conmon
version: 'conmon version 2.0.18, commit: '
cpus: 8
distribution:
distribution: neon
version: "18.04"
eventLogger: file
hostname: thinkpad-t480s
idMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
kernel: 5.3.0-62-generic
linkmode: dynamic
memFree: 1611853824
memTotal: 16664281088
ociRuntime:
name: runc
package: Unknown
path: /usr/bin/runc
version: 'runc version spec: 1.0.1-dev'
os: linux
remoteSocket:
path: /run/user/1000/podman/podman.sock
rootless: true
slirp4netns:
executable: /usr/bin/slirp4netns
package: 'slirp4netns: /usr/bin/slirp4netns'
version: |-
slirp4netns version 0.4.3
commit: unknown
swapFree: 897052672
swapTotal: 1023406080
uptime: 2h 52m 16.21s (Approximately 0.08 days)
registries:
localhost:5000:
Blocked: false
Insecure: true
Location: localhost:5000
MirrorByDigestOnly: false
Mirrors: []
Prefix: localhost:5000
search:
- docker.io
- registry.access.redhat.com
store:
configFile: /home/dmisharo/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions:
overlay.mount_program:
Executable: /usr/bin/fuse-overlayfs
Package: Unknown
Version: |-
fusermount3 version: 3.6.2
fuse-overlayfs: version 0.7.8
FUSE library version 3.4.1
using FUSE kernel interface version 7.27
graphRoot: /home/dmisharo/.local/share/containers/storage
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "false"
imageStore:
number: 20
runRoot: /run/user/1000/containers
volumePath: /home/dmisharo/.local/share/containers/storage/volumes
version:
APIVersion: 1
Built: 0
BuiltTime: Thu Jan 1 01:00:00 1970
GitCommit: ""
GoVersion: go1.14.4
OsArch: linux/amd64
Version: 2.0.2
Package info (e.g. output of rpm -q podman or apt list podman):
Listing... Done
podman/unknown,now 2.0.2~2 amd64 [installed]
Additional environment details (AWS, VirtualBox, physical, etc.):
Local physical machine