Skip to content

No connectivity from client to server on the same host via kube-proxy ClusterIP when Cilium E/W LB is disabled #10567

@aanm

Description

@aanm

client-pod: 10.0.0.13:XXX
service-ip: 172.20.62.249:9090
backend: 10.0.0.115:9090

tcpdump -i any:

00:41:31.413413 IP 10.0.0.13.40106 > 172.20.62.249.9090: Flags [S], seq 179494632, win 64390, options [mss 1370,sackOK,TS val 2655386555 ecr 0,nop,wscale 7], length 0
00:41:31.413492 IP 10.0.0.13.40106 > 10.0.0.115.9090: Flags [S], seq 179494632, win 64390, options [mss 1370,sackOK,TS val 2655386555 ecr 0,nop,wscale 7], length 0
00:41:31.413513 IP 10.0.0.115.9090 > 10.0.0.13.40106: Flags [S.], seq 2265501166, ack 179494633, win 65184, options [mss 1370,sackOK,TS val 2584967544 ecr 2655386555,nop,wscale 7], length 0
00:41:31.413527 IP 10.0.0.115.9090 > 10.0.0.13.40106: Flags [S.], seq 2265501166, ack 179494633, win 65184, options [mss 1370,sackOK,TS val 2584967544 ecr 2655386555,nop,wscale 7], length 0
00:41:31.413537 IP 10.0.0.13.40106 > 10.0.0.115.9090: Flags [R], seq 179494633, win 0, length 0
00:41:31.413544 IP 10.0.0.13.40106 > 10.0.0.115.9090: Flags [R], seq 179494633, win 0, length 0

monitor output:

Policy verdict log: flow 0xdece3b7d local EP ID 2591, remote ID 2, dst port 9090, proto 6, ingress false, action allow, match all, 10.0.0.13:39772 -> 172.20.62.249:9090 tcp SYN
-> stack flow 0xdece3b7d identity 7644->2 state new ifindex 0 orig-ip 0.0.0.0: 10.0.0.13:39772 -> 172.20.62.249:9090 tcp SYN
Policy verdict log: flow 0x492eb6dd local EP ID 2591, remote ID 56645, dst port 39772, proto 6, ingress true, action allow, match all, 10.0.0.115:9090 -> 10.0.0.13:39772 tcp SYN, ACK
-> endpoint 2591 flow 0x492eb6dd identity 56645->7644 state new ifindex lxc10371e843f08 orig-ip 10.0.0.115: 10.0.0.115:9090 -> 10.0.0.13:39772 tcp SYN, ACK

cilium config map

  auto-direct-node-routes: "false"
  bpf-ct-global-any-max: "262144"
  bpf-ct-global-tcp-max: "524288"
  bpf-nat-global-max: "841429"
  cluster-name: default
  debug: "false"
  enable-external-ips: "false"
  enable-host-reachable-services: "false"
  enable-ipv4: "true"
  enable-ipv6: "false"
  enable-metrics: "true"
  enable-node-port: "false"
  enable-remote-node-identity: "true"
  enable-well-known-identities: "false"
  enable-xt-socket-fallback: "true"
  identity-allocation-mode: crd
  install-iptables-rules: "true"
  k8s-require-ipv4-pod-cidr: "true"
  kube-proxy-replacement: disabled
  masquerade: "true"
  monitor-aggregation: medium
  monitor-aggregation-flags: all
  monitor-aggregation-interval: 5s
  node-port-mode: hybrid
  operator-api-serve-addr: 127.0.0.1:9234
  operator-prometheus-serve-addr: :6942
  policy-audit-mode: "false"
  preallocate-bpf-maps: "false"
  prometheus-serve-addr: :9090
  sidecar-istio-proxy-image: cilium/istio_proxy
  synchronize-k8s-nodes: "true"
  tofqdns-enable-poller: "false"
  tunnel: vxlan
  wait-bpf-mount: "false"

iptables rules

$ sudo iptables-save | grep 9090
-A KUBE-SEP-VTRHTEKTWBKBIVAE -p tcp -m tcp -j DNAT --to-destination 10.0.0.115:9090
-A KUBE-SERVICES -d 172.20.62.249/32 -p tcp -m comment --comment "kube-system/prometheus:webui cluster IP" -m tcp --dport 9090 -j KUBE-SVC-SMBNPD2J27EUPM6V
$ sudo iptables-save | grep KUBE-SVC-SMBNPD2J27EUPM6V
:KUBE-SVC-SMBNPD2J27EUPM6V - [0:0]
-A KUBE-SERVICES -d 172.20.62.249/32 -p tcp -m comment --comment "kube-system/prometheus:webui cluster IP" -m tcp --dport 9090 -j KUBE-SVC-SMBNPD2J27EUPM6V
-A KUBE-SVC-SMBNPD2J27EUPM6V -j KUBE-SEP-VTRHTEKTWBKBIVAE
$ sudo iptables-save | grep KUBE-SEP-VTRHTEKTWBKBIVAE
:KUBE-SEP-VTRHTEKTWBKBIVAE - [0:0]
-A KUBE-SEP-VTRHTEKTWBKBIVAE -s 10.0.0.115/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-VTRHTEKTWBKBIVAE -p tcp -m tcp -j DNAT --to-destination 10.0.0.115:9090
-A KUBE-SVC-SMBNPD2J27EUPM6V -j KUBE-SEP-VTRHTEKTWBKBIVAE

commit a734d81

Metadata

Metadata

Assignees

Labels

area/datapathImpacts bpf/ or low-level forwarding details, including map management and monitor messages.kind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions