Skip to content

portmap nftables backend invalid rules (dnat from input hook + invalid ipv6 rules) #1115

@champtar

Description

@champtar

Trying the new nftables backend, it fails on EL 9.4

# nft --version
nftables v1.0.9 (Old Doc Yak #3)
# uname -r
5.14.0-427.40.1.el9_4.x86_64

conf

{
  "type": "portmap",
  "capabilities": {"portMappings": true},
  "backend": "nftables",
  "conditionsV4": ["ip", "daddr", "!=", "{ 127.0.0.0/8, 198.19.254.254 }"]
},

Error

Nov 05 17:15:34 atsc2 kubelet[4258]: E1105 17:15:34.352838    4258 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err=<
Nov 05 17:15:34 atsc2 kubelet[4258]:         rpc error: code = Unknown desc = failed to setup network for sandbox "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621": plugin type="multus-cni" name="multus-cni-network" failed (add): [traefik/appliance-ingress-traefik-698cd97568-5xg88/3908060c-0419-481c-865d-3230b95c3a84:mgmt]: error adding container to network "mgmt": plugin type="portmap" failed (add): unable to set up nftables rules for port mappings: /dev/stdin:13:1-182: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 81 dnat ip addr . port to 198.18.1.194 . 10081 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:14:1-184: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 8444 dnat ip addr . port to 198.18.1.194 . 18444 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:15:1-182: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 82 dnat ip addr . port to 198.18.1.194 . 10082 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:16:1-184: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 8445 dnat ip addr . port to 198.18.1.194 . 18445 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:17:1-182: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 83 dnat ip addr . port to 198.18.1.194 . 10083 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:18:1-184: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 8446 dnat ip addr . port to 198.18.1.194 . 18446 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:19:1-168: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport masquerading ip saddr 198.18.1.194 ip daddr 198.18.1.194 masquerade comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:20:1-165: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport masquerading ip saddr 127.0.0.1 ip daddr 198.18.1.194 masquerade comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:  >

Using strace, the rules are:

add table ip cni_hostport { comment "CNI portmap plugin" ; }
add chain ip cni_hostport hostports
add chain ip cni_hostport hostip_hostports
add chain ip cni_hostport input { type nat hook input priority -100 ; }
flush chain ip cni_hostport input
add rule ip cni_hostport input ip daddr != { 127.0.0.0/8, 198.19.254.254 } jump hostip_hostports
add rule ip cni_hostport input ip daddr != { 127.0.0.0/8, 198.19.254.254 } jump hostports
add chain ip cni_hostport output { type nat hook output priority -100 ; }
flush chain ip cni_hostport output
add rule ip cni_hostport output ip daddr != { 127.0.0.0/8, 198.19.254.254 } jump hostip_hostports
add rule ip cni_hostport output ip daddr != { 127.0.0.0/8, 198.19.254.254 } fib daddr type local jump hostports
add chain ip cni_hostport masquerading { type nat hook postrouting priority 100 ; }
add rule ip cni_hostport hostports ip protocol tcp th dport 81 dnat ip addr . port to 198.18.0.56 . 10081 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 8444 dnat ip addr . port to 198.18.0.56 . 18444 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 82 dnat ip addr . port to 198.18.0.56 . 10082 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 8445 dnat ip addr . port to 198.18.0.56 . 18445 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 83 dnat ip addr . port to 198.18.0.56 . 10083 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 8446 dnat ip addr . port to 198.18.0.56 . 18446 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport masquerading ip saddr 198.18.0.56 ip daddr 198.18.0.56 masquerade comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport masquerading ip saddr 127.0.0.1 ip daddr 198.18.0.56 masquerade comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"

I think instead of

add chain ip cni_hostport input { type nat hook input priority -100 ; }

we should use

add chain ip cni_hostport input { type nat hook prerouting priority -100 ; }

(and rename input to prerouting)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions