Skip to content

Pasta Networking - OUTBOUND Traffic does NOT work, INBOUND Traffic works #23003

@luckylinux

Description

@luckylinux

Issue Description

When using pasta Networking (at least using IPv6 Address Only) the following can be observed:

  • OUTBOUND Traffic does NOT work
  • INBOUND Traffic works (using Port Publishing / Mapping)

Steps to reproduce the issue

Steps to reproduce the issue

  1. Patch Pasta According to my Comment in podman 5.0.3 fails now with message pasta failed with exit code 1: couldn't set IPv6 route(s) in guest: no route to host #22824 (otherwise it ABSOLUTELY will NOT work)
  2. Start Podman Container (the minimal example Provided at https://passt.top/passt/about/#pasta_2 results in a timeout as well: podman run --net=pasta --rm -ti alpine sh)

Otherwise for my Debugging Container: https://github.com/luckylinux/docker-network-troubleshooting/

Which is first build then run using (tried several options, OUTBOUND traffic fails with each and every single one of them except using slirp4netns instead of pasta):

#!/bin/bash

# Determine toolpath if not set already
relativepath="./" # Define relative path to go from this script to the root level of the tool
if [[ ! -v toolpath ]]; then scriptpath=$(cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd ); toolpath=$(realpath --canonicalize-missing ${scriptpath}/${relativepath}); fi

# Load Configuration
libpath=$(readlink --canonicalize-missing "${toolpath}/includes")
source ${libpath}/functions.sh


# Optional argument
engine=${1-"podman"}

# Container Name
containername="docker-network-troubleshooting"

# Container Image
containerimage="docker-network-troubleshooting:debian-latest"

# Containers Configuration Folder to Map
containersconfigfolder="./containers"

# Options
opts=()

# Run in the Background
opts+=("-d")

# Podman 5.x with Pasta doesn't handle Networking Correctly
# Force to use slirp4netns
#opts+=("--network=slirp4netns")

# Debug Pasta
timestamp=$(date +"%Y%m%d_%Hh%M%S")
#opts+=("--network=pasta:--pcap,./tmp/${timestamp}.pcap")
#opts+=("--network=pasta:--ipv6-only,-t,2XXX:XXXX:XXXX:ff15::2/80,-t,2XXX:XXXX:XXXX:ff15::2/443,-t,2XXX:XXXX:XXXX:ff15::2/5001,-u,2XXX:XXXX:XXXX:ff15::2/5001")
#opts+=("--network=pasta:--ipv6-only,--outbound,2XXX:XXXX:XXXX:ff15::2")
#opts+=("--network=pasta:--ipv6-only,-a,2XXX:XXXX:XXXX:ff15::2")
opts+=("--network=pasta:--ipv6-only,-a,2XXX:XXXX:XXXX:ff15::2,--pcap,./tmp/${timestamp}.pcap")

# Publish ports
opts+=("-p")
opts+=("[2XXX:XXXX:XXXX:ff15::2]:80:80/tcp")
opts+=("-p")
opts+=("[2XXX:XXXX:XXXX:ff15::2]:443:443/tcp")
opts+=("-p")
opts+=("[2XXX:XXXX:XXXX::2]:5201:5201/tcp")
opts+=("-p")
opts+=("[2XXX:XXXX:XXXX:XXXX::2]:5201:5201/udp")

# Add Capacilities
opts+=("--cap-add")
opts+=("CAP_NET_RAW")

# Enable Infinite Loop
opts+=("-e")
opts+=("ENABLE_INFINITE_LOOP=true")

# Disable Automatic Test
opts+=("-e")
opts+=("ENABLE_AUTOMATIC_TEST=false")

# Load the Environment Variables into THIS Script
if [[ -f "./.env" ]]
then
   eval "$(shdotenv --env .env || echo \"exit $?\")"
fi

# Terminate and Remove Existing Containers if Any
${engine} stop --ignore ${containername}
${engine} rm --ignore ${containername}

# Run Image with Infinite Loop to prevent it from automatically terminating
#${engine} run --name=${containername} --env-file "./.env" localhost:5000/local/"${containerimage}"
echo "Executing: ${engine} run --name=${containername} ${opts[*]} localhost:5000/local/${containerimage}"
${engine} run --replace --name=${containername} ${opts[*]} localhost:5000/local/"${containerimage}"

# Open Interactive Shell with Container
#${engine} exec -it ${containername} /bin/bash

  1. Try to perform a TCP Connection: curl -6 ifconfig.me for a Remote Server or curl -6 https://[2XXX:XXXX:XXXX::0:1] for my Local OPNSense Router Web Interface

Timeout will be reached.

Describe the results you received

Timeout reached when running the above mentioned curl Commands.

Furthermore:

  • ping -6 (ICMPv6) of LAN Hosts from WITHIN the Container (even on a different Subnet) work correctly
  • ping -6 (ICMPv6) of WAN Hosts from WITHIN the Container Fails
  • traceroute6 work for LAN Hosts (TCP) BUT it takes around 30 seconds, while the reported TTL is shown as ~ 1.7 ms
  • traceroute6 fails for WAN Hosts (both ICMPv6 or TCP)
  • Analysis of Wireshark PCAP Shows:
    • DNS Fails (fallback to .<MYDOMAIN.TLD>, where search = <MYDOMAIN.TLD>)
    • ICMPv6 seems to work correctly
    • TCP Flags: Lines are either RED (for RST Flag) or BLACK (for SYN Flag & TCP Port Number Reused)
      • RST Flag: present
      • SYN Flag: present
    • TCP Flags missing: Notably absent are ACK or FIN (among others), so the Connection initialized from within the container never gets answered by the outside Services

Describe the results you expected

Able to reach outside Services normally.

  • curl -6 ifconfig.me should succeed
  • DNS should resolve correctly
  • Package Managers (apk, apt, etc) should be able to download Lists & Updates (when building Containers)

podman info output

host:
  arch: amd64
  buildahVersion: 1.36.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.26
    systemPercent: 0.41
    userPercent: 0.34
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: server
    version: "40"
  eventLogger: journald
  freeLocks: 2025
  hostname: podmanserver15.MYDOMAIN.TLD
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 655360
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 655360
      size: 65536
  kernel: 6.8.11-300.fc40.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 2777296896
  memTotal: 8171499520
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-3.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.15-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/user/1002/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/local/bin/pasta
    package: Unknown
    version: |
      pasta 2024_06_07.8a83b53-9-g4070bac
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1002/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 10656407552
  swapTotal: 10733215744
  uptime: 30h 35m 48.00s (Approximately 1.25 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  localhost:5000:
    Blocked: false
    Insecure: true
    Location: localhost:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:5000
    PullFromMirror: ""
  search:
  - docker.MYDOMAIN.TLD
  - docker.io
  - ghcr.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 8
    paused: 0
    running: 8
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.13-1.fc40.x86_64
      Version: |-
        fusermount3 version: 3.16.2
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.16.2
        using FUSE kernel interface version 7.38
    overlay.mountopt: nodev
  graphRoot: /zdata/PODMAN/STORAGE
  graphRootAllocated: 298237820928
  graphRootUsed: 3645636608
  graphStatus:
    Backing Filesystem: zfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /home/podman/containers/tmp
  imageStore:
    number: 222
  runRoot: /run/user/1002
  transientStore: false
  volumePath: /zdata/PODMAN/VOLUMES
version:
  APIVersion: 5.1.0
  Built: 1716940800
  BuiltTime: Wed May 29 02:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.3
  Os: linux
  OsArch: linux/amd64
  Version: 5.1.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Running on Fedora 40 in a KVM AMD64 Virtual Machine (Proxmox VE).

Additional information

Routes from within the Container:

root@CONTAINER:/opt/app# ip -6 route
64:ff9b:1::1:0:0/96 via 2XXX:XXXX:XXXX:1::1:66 dev ens18 metric 1024 pref medium
64:ff9b:1::2:0:0/96 via 2XXX:XXXX:XXXX:1::1:67 dev ens18 metric 1024 pref medium
2XXX:XXXX:XXXX:1::/64 dev ens18 proto ra metric 1002 mtu 1500 pref medium
2XXX:XXXX:XXXX:1::/64 via 2XXX:XXXX:XXXX:1::1 dev ens18 metric 1024 pref medium
2XXX:XXXX:XXXX:ff15::/64 dev ens18 proto kernel metric 256 pref medium
fe80::/64 dev ens18 proto kernel metric 256 pref medium
default via fe80::XXXX:XXXX:XXXX:XXXX dev ens18 proto ra metric 1002 mtu 1500 pref medium

root@CONTAINER:/opt/app# ip -6 route show table local
local ::1 dev lo proto kernel metric 0 pref medium
local 2XXX:XXXX:XXXX:ff15::2 dev ens18 proto kernel metric 0 pref medium
local fe80::XXXX:XXXX:XXXX:XXXX dev ens18 proto kernel metric 0 pref medium
multicast ff00::/8 dev ens18 proto kernel metric 256 pref medium

Addresses from within the Container:

root@CONTAINER:/opt/app# ip -6 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host proto kernel_lo 
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 state UNKNOWN qlen 1000
    inet6 2XXX:XXXX:XXXX:ff15::2/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::XXXX:XXXX:XXXX:XXXX/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

Basically, the only way to get OUTBOUND connectivity is to switch to slirp4netns.

This affects both podman build and podman run (probably also buildah).

Further Information available in: #22824

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.locked - please file new issue/PRAssist humans wanting to comment on an old issue or PR with locked comments.pastapasta(1) bugs or features

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions