-
Notifications
You must be signed in to change notification settings - Fork 3k
Description
Issue Description
When using pasta Networking (at least using IPv6 Address Only) the following can be observed:
- OUTBOUND Traffic does NOT work
- INBOUND Traffic works (using Port Publishing / Mapping)
Steps to reproduce the issue
Steps to reproduce the issue
- Patch Pasta According to my Comment in podman 5.0.3 fails now with message pasta failed with exit code 1: couldn't set IPv6 route(s) in guest: no route to host #22824 (otherwise it ABSOLUTELY will NOT work)
- Start Podman Container (the minimal example Provided at https://passt.top/passt/about/#pasta_2 results in a timeout as well:
podman run --net=pasta --rm -ti alpine sh)
Otherwise for my Debugging Container: https://github.com/luckylinux/docker-network-troubleshooting/
Which is first build then run using (tried several options, OUTBOUND traffic fails with each and every single one of them except using slirp4netns instead of pasta):
#!/bin/bash
# Determine toolpath if not set already
relativepath="./" # Define relative path to go from this script to the root level of the tool
if [[ ! -v toolpath ]]; then scriptpath=$(cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd ); toolpath=$(realpath --canonicalize-missing ${scriptpath}/${relativepath}); fi
# Load Configuration
libpath=$(readlink --canonicalize-missing "${toolpath}/includes")
source ${libpath}/functions.sh
# Optional argument
engine=${1-"podman"}
# Container Name
containername="docker-network-troubleshooting"
# Container Image
containerimage="docker-network-troubleshooting:debian-latest"
# Containers Configuration Folder to Map
containersconfigfolder="./containers"
# Options
opts=()
# Run in the Background
opts+=("-d")
# Podman 5.x with Pasta doesn't handle Networking Correctly
# Force to use slirp4netns
#opts+=("--network=slirp4netns")
# Debug Pasta
timestamp=$(date +"%Y%m%d_%Hh%M%S")
#opts+=("--network=pasta:--pcap,./tmp/${timestamp}.pcap")
#opts+=("--network=pasta:--ipv6-only,-t,2XXX:XXXX:XXXX:ff15::2/80,-t,2XXX:XXXX:XXXX:ff15::2/443,-t,2XXX:XXXX:XXXX:ff15::2/5001,-u,2XXX:XXXX:XXXX:ff15::2/5001")
#opts+=("--network=pasta:--ipv6-only,--outbound,2XXX:XXXX:XXXX:ff15::2")
#opts+=("--network=pasta:--ipv6-only,-a,2XXX:XXXX:XXXX:ff15::2")
opts+=("--network=pasta:--ipv6-only,-a,2XXX:XXXX:XXXX:ff15::2,--pcap,./tmp/${timestamp}.pcap")
# Publish ports
opts+=("-p")
opts+=("[2XXX:XXXX:XXXX:ff15::2]:80:80/tcp")
opts+=("-p")
opts+=("[2XXX:XXXX:XXXX:ff15::2]:443:443/tcp")
opts+=("-p")
opts+=("[2XXX:XXXX:XXXX::2]:5201:5201/tcp")
opts+=("-p")
opts+=("[2XXX:XXXX:XXXX:XXXX::2]:5201:5201/udp")
# Add Capacilities
opts+=("--cap-add")
opts+=("CAP_NET_RAW")
# Enable Infinite Loop
opts+=("-e")
opts+=("ENABLE_INFINITE_LOOP=true")
# Disable Automatic Test
opts+=("-e")
opts+=("ENABLE_AUTOMATIC_TEST=false")
# Load the Environment Variables into THIS Script
if [[ -f "./.env" ]]
then
eval "$(shdotenv --env .env || echo \"exit $?\")"
fi
# Terminate and Remove Existing Containers if Any
${engine} stop --ignore ${containername}
${engine} rm --ignore ${containername}
# Run Image with Infinite Loop to prevent it from automatically terminating
#${engine} run --name=${containername} --env-file "./.env" localhost:5000/local/"${containerimage}"
echo "Executing: ${engine} run --name=${containername} ${opts[*]} localhost:5000/local/${containerimage}"
${engine} run --replace --name=${containername} ${opts[*]} localhost:5000/local/"${containerimage}"
# Open Interactive Shell with Container
#${engine} exec -it ${containername} /bin/bash
- Try to perform a TCP Connection:
curl -6 ifconfig.mefor a Remote Server orcurl -6 https://[2XXX:XXXX:XXXX::0:1]for my Local OPNSense Router Web Interface
Timeout will be reached.
Describe the results you received
Timeout reached when running the above mentioned curl Commands.
Furthermore:
ping -6(ICMPv6) of LAN Hosts from WITHIN the Container (even on a different Subnet) work correctlyping -6(ICMPv6) of WAN Hosts from WITHIN the Container Failstraceroute6work for LAN Hosts (TCP) BUT it takes around 30 seconds, while the reported TTL is shown as ~ 1.7 mstraceroute6fails for WAN Hosts (both ICMPv6 or TCP)- Analysis of Wireshark PCAP Shows:
- DNS Fails (fallback to .<MYDOMAIN.TLD>, where search = <MYDOMAIN.TLD>)
- ICMPv6 seems to work correctly
- TCP Flags: Lines are either RED (for RST Flag) or BLACK (for SYN Flag & TCP Port Number Reused)
- RST Flag: present
- SYN Flag: present
- TCP Flags missing: Notably absent are ACK or FIN (among others), so the Connection initialized from within the container never gets answered by the outside Services
Describe the results you expected
Able to reach outside Services normally.
curl -6 ifconfig.meshould succeed- DNS should resolve correctly
- Package Managers (
apk,apt, etc) should be able to download Lists & Updates (when building Containers)
podman info output
host:
arch: amd64
buildahVersion: 1.36.0
cgroupControllers:
- cpu
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon-2.1.10-1.fc40.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.1.10, commit: '
cpuUtilization:
idlePercent: 99.26
systemPercent: 0.41
userPercent: 0.34
cpus: 8
databaseBackend: sqlite
distribution:
distribution: fedora
variant: server
version: "40"
eventLogger: journald
freeLocks: 2025
hostname: podmanserver15.MYDOMAIN.TLD
idMappings:
gidmap:
- container_id: 0
host_id: 1002
size: 1
- container_id: 1
host_id: 655360
size: 65536
uidmap:
- container_id: 0
host_id: 1002
size: 1
- container_id: 1
host_id: 655360
size: 65536
kernel: 6.8.11-300.fc40.x86_64
linkmode: dynamic
logDriver: journald
memFree: 2777296896
memTotal: 8171499520
networkBackend: netavark
networkBackendInfo:
backend: netavark
dns:
package: aardvark-dns-1.10.0-1.fc40.x86_64
path: /usr/libexec/podman/aardvark-dns
version: aardvark-dns 1.10.0
package: netavark-1.10.3-3.fc40.x86_64
path: /usr/libexec/podman/netavark
version: netavark 1.10.3
ociRuntime:
name: crun
package: crun-1.15-1.fc40.x86_64
path: /usr/bin/crun
version: |-
crun version 1.15
commit: e6eacaf4034e84185fd8780ac9262bbf57082278
rundir: /run/user/1002/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
os: linux
pasta:
executable: /usr/local/bin/pasta
package: Unknown
version: |
pasta 2024_06_07.8a83b53-9-g4070bac
Copyright Red Hat
GNU General Public License, version 2 or later
<https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
remoteSocket:
exists: true
path: /run/user/1002/podman/podman.sock
rootlessNetworkCmd: pasta
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: true
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.2.2-2.fc40.x86_64
version: |-
slirp4netns version 1.2.2
commit: 0ee2d87523e906518d34a6b423271e4826f71faf
libslirp: 4.7.0
SLIRP_CONFIG_VERSION_MAX: 4
libseccomp: 2.5.5
swapFree: 10656407552
swapTotal: 10733215744
uptime: 30h 35m 48.00s (Approximately 1.25 days)
variant: ""
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
localhost:5000:
Blocked: false
Insecure: true
Location: localhost:5000
MirrorByDigestOnly: false
Mirrors: null
Prefix: localhost:5000
PullFromMirror: ""
search:
- docker.MYDOMAIN.TLD
- docker.io
- ghcr.io
store:
configFile: /home/podman/.config/containers/storage.conf
containerStore:
number: 8
paused: 0
running: 8
stopped: 0
graphDriverName: overlay
graphOptions:
overlay.mount_program:
Executable: /usr/bin/fuse-overlayfs
Package: fuse-overlayfs-1.13-1.fc40.x86_64
Version: |-
fusermount3 version: 3.16.2
fuse-overlayfs: version 1.13-dev
FUSE library version 3.16.2
using FUSE kernel interface version 7.38
overlay.mountopt: nodev
graphRoot: /zdata/PODMAN/STORAGE
graphRootAllocated: 298237820928
graphRootUsed: 3645636608
graphStatus:
Backing Filesystem: zfs
Native Overlay Diff: "false"
Supports d_type: "true"
Supports shifting: "true"
Supports volatile: "true"
Using metacopy: "false"
imageCopyTmpDir: /home/podman/containers/tmp
imageStore:
number: 222
runRoot: /run/user/1002
transientStore: false
volumePath: /zdata/PODMAN/VOLUMES
version:
APIVersion: 5.1.0
Built: 1716940800
BuiltTime: Wed May 29 02:00:00 2024
GitCommit: ""
GoVersion: go1.22.3
Os: linux
OsArch: linux/amd64
Version: 5.1.0Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
Yes
Additional environment details
Running on Fedora 40 in a KVM AMD64 Virtual Machine (Proxmox VE).
Additional information
Routes from within the Container:
root@CONTAINER:/opt/app# ip -6 route
64:ff9b:1::1:0:0/96 via 2XXX:XXXX:XXXX:1::1:66 dev ens18 metric 1024 pref medium
64:ff9b:1::2:0:0/96 via 2XXX:XXXX:XXXX:1::1:67 dev ens18 metric 1024 pref medium
2XXX:XXXX:XXXX:1::/64 dev ens18 proto ra metric 1002 mtu 1500 pref medium
2XXX:XXXX:XXXX:1::/64 via 2XXX:XXXX:XXXX:1::1 dev ens18 metric 1024 pref medium
2XXX:XXXX:XXXX:ff15::/64 dev ens18 proto kernel metric 256 pref medium
fe80::/64 dev ens18 proto kernel metric 256 pref medium
default via fe80::XXXX:XXXX:XXXX:XXXX dev ens18 proto ra metric 1002 mtu 1500 pref medium
root@CONTAINER:/opt/app# ip -6 route show table local
local ::1 dev lo proto kernel metric 0 pref medium
local 2XXX:XXXX:XXXX:ff15::2 dev ens18 proto kernel metric 0 pref medium
local fe80::XXXX:XXXX:XXXX:XXXX dev ens18 proto kernel metric 0 pref medium
multicast ff00::/8 dev ens18 proto kernel metric 256 pref medium
Addresses from within the Container:
root@CONTAINER:/opt/app# ip -6 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
inet6 ::1/128 scope host proto kernel_lo
valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 state UNKNOWN qlen 1000
inet6 2XXX:XXXX:XXXX:ff15::2/64 scope global nodad
valid_lft forever preferred_lft forever
inet6 fe80::XXXX:XXXX:XXXX:XXXX/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
Basically, the only way to get OUTBOUND connectivity is to switch to slirp4netns.
This affects both podman build and podman run (probably also buildah).
Further Information available in: #22824