Skip to content

Node-Local dns with LocalRedirectPolicy not working on pods listening on port 53 #20164

Closed
#26144
Closed
Node-Local dns with LocalRedirectPolicy not working on pods listening on port 53#20164
#26144
Labels
area/datapathImpacts bpf/ or low-level forwarding details, including map management and monitor messages.area/lrpImpacts Local Redirect Policy.kind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.kind/duplicateThere is another issue which contains additional details.
@weikinhuang

Description

@weikinhuang
weikinhuang
opened on Jun 10, 2022

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

Hi, I'm just started running into an issue with node-local-dns and cilium local redirect policy. Everything was working fine previously on 20.04.

My previous nodes running

Linux n101 5.4.0-107-generic #121-Ubuntu SMP Thu Mar 24 16:04:27 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

worked just fine, however my new nodes are running ubuntu 22.04 with kernel

Linux n001 5.15.0-35-generic #36-Ubuntu SMP Sat May 21 02:24:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

does not seem to work properly for some pods, and does for others

I'm thinking it might be a kernel issue, because previously I was also using the same cilium version (1.11.4, then I tried again with cilium 1.11.5)
Currently i'm on:
kubernetes 1.24.1
cilium 1.11.5
ubuntu 22.04 up to date

Cilium Version

Client: 1.11.5 b0d3140 2022-05-10T02:47:42+02:00 go version go1.17.9 linux/amd64
Daemon: 1.11.5 b0d3140 2022-05-10T02:47:42+02:00 go version go1.17.9 linux/amd64

Kernel Version

Linux n002 5.15.0-35-generic #36-Ubuntu SMP Sat May 21 02:24:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:26:19Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:18:48Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}

Sysdump

cilium-sysdump-20220616-083851.zip

Relevant log output

#### WITH POLICY DISABLED

root@dnstest:/root# dig @10.3.0.10 web.site.svc.cluster.local

; <<>> DiG 9.16.27-Debian <<>> @10.3.0.10 web.site.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52788
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: fe7eba1c9b7d2ba8 (echoed)
;; QUESTION SECTION:
;web.site.svc.cluster.local. IN A

;; ANSWER SECTION:
web.site.svc.cluster.local. 25 IN A 10.3.249.244

;; Query time: 0 msec
;; SERVER: 10.3.0.10#53(10.3.0.10)
;; WHEN: Tue Jun 07 02:12:13 UTC 2022
;; MSG SIZE  rcvd: 151

WITH POLICY ENABLED

root@dnstest:/root# dig @10.3.0.10 web.site.svc.cluster.local

; <<>> DiG 9.16.27-Debian <<>> @10.3.0.10 web.site.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 2353
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: f3135a06af572481 (echoed)
;; QUESTION SECTION:
;web.site.svc.cluster.local. IN A

;; Query time: 999 msec
;; SERVER: 10.3.0.10#53(10.3.0.10)
;; WHEN: Tue Jun 07 02:13:08 UTC 2022
;; MSG SIZE  rcvd: 88

to node-local-dns pod in the same host

root@dnstest:/root# dig @10.2.3.72 web.site.svc.cluster.local

; <<>> DiG 9.16.27-Debian <<>> @10.2.3.72 web.site.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64896
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: e6244d4c09352f06 (echoed)
;; QUESTION SECTION:
;web.site.svc.cluster.local. IN A

;; ANSWER SECTION:
web.site.svc.cluster.local. 6 IN A 10.3.249.244

;; Query time: 3 msec
;; SERVER: 10.2.3.72#53(10.2.3.72)
;; WHEN: Tue Jun 07 02:14:02 UTC 2022
;; MSG SIZE  rcvd: 151

tcpdump policy off, here it goes straight to the coredns pods in kube-dns correctly

02:47:44.076470 lxc700b5d19881b In  IP 10.2.3.143.53337 > 10.2.0.42.53: Flags [S], seq 2601814125, win 65535, options [mss 1460,sackOK,TS val 3298041818 ecr 0,nop,wscale 9], length 0
02:47:44.076536 eth0  Out IP 10.2.3.143.53337 > 10.2.0.42.53: Flags [S], seq 2601814125, win 65535, options [mss 1460,sackOK,TS val 3298041818 ecr 0,nop,wscale 9], length 0
02:47:44.076666 eth0  In  IP 10.2.0.42.53 > 10.2.3.143.53337: Flags [S.], seq 2145065476, ack 2601814126, win 65535, options [mss 1460,sackOK,TS val 833701055 ecr 3298041818,nop,wscale 9], length 0
02:47:44.076703 lxc700b5d19881b Out IP 10.2.0.42.53 > 10.2.3.143.53337: Flags [S.], seq 2145065476, ack 2601814126, win 65535, options [mss 1460,sackOK,TS val 833701055 ecr 3298041818,nop,wscale 9], length 0
02:47:44.076728 lxc700b5d19881b In  IP 10.2.3.143.53337 > 10.2.0.42.53: Flags [.], ack 1, win 128, options [nop,nop,TS val 3298041818 ecr 833701055], length 0
02:47:44.076747 eth0  Out IP 10.2.3.143.53337 > 10.2.0.42.53: Flags [.], ack 1, win 128, options [nop,nop,TS val 3298041818 ecr 833701055], length 0
02:47:44.076825 lxc700b5d19881b In  IP 10.2.3.143.53337 > 10.2.0.42.53: Flags [P.], seq 1:91, ack 1, win 128, options [nop,nop,TS val 3298041819 ecr 833701055], length 90 64571+ [1au] A? web.site.svc.cluster.local. (88)
02:47:44.076847 eth0  Out IP 10.2.3.143.53337 > 10.2.0.42.53: Flags [P.], seq 1:91, ack 1, win 128, options [nop,nop,TS val 3298041819 ecr 833701055], length 90 64571+ [1au] A? web.site.svc.cluster.local. (88)
02:47:44.076914 eth0  In  IP 10.2.0.42.53 > 10.2.3.143.53337: Flags [.], ack 91, win 128, options [nop,nop,TS val 833701055 ecr 3298041819], length 0
02:47:44.076928 lxc700b5d19881b Out IP 10.2.0.42.53 > 10.2.3.143.53337: Flags [.], ack 91, win 128, options [nop,nop,TS val 833701055 ecr 3298041819], length 0
02:47:44.077554 eth0  In  IP 10.2.0.42.53 > 10.2.3.143.53337: Flags [P.], seq 1:154, ack 91, win 128, options [nop,nop,TS val 833701056 ecr 3298041819], length 153 64571*- 1/0/1 A 10.3.249.244 (151)
02:47:44.077590 lxc700b5d19881b Out IP 10.2.0.42.53 > 10.2.3.143.53337: Flags [P.], seq 1:154, ack 91, win 128, options [nop,nop,TS val 833701056 ecr 3298041819], length 153 64571*- 1/0/1 A 10.3.249.244 (151)
02:47:44.077598 lxc700b5d19881b In  IP 10.2.3.143.53337 > 10.2.0.42.53: Flags [.], ack 154, win 131, options [nop,nop,TS val 3298041819 ecr 833701056], length 0
02:47:44.077617 eth0  Out IP 10.2.3.143.53337 > 10.2.0.42.53: Flags [.], ack 154, win 131, options [nop,nop,TS val 3298041819 ecr 833701056], length 0
02:47:44.078027 lxc700b5d19881b In  IP 10.2.3.143.53337 > 10.2.0.42.53: Flags [F.], seq 91, ack 154, win 131, options [nop,nop,TS val 3298041820 ecr 833701056], length 0
02:47:44.078061 eth0  Out IP 10.2.3.143.53337 > 10.2.0.42.53: Flags [F.], seq 91, ack 154, win 131, options [nop,nop,TS val 3298041820 ecr 833701056], length 0
02:47:44.078632 eth0  In  IP 10.2.0.42.53 > 10.2.3.143.53337: Flags [F.], seq 154, ack 92, win 128, options [nop,nop,TS val 833701057 ecr 3298041820], length 0
02:47:44.078713 lxc700b5d19881b Out IP 10.2.0.42.53 > 10.2.3.143.53337: Flags [F.], seq 154, ack 92, win 128, options [nop,nop,TS val 833701057 ecr 3298041820], length 0
02:47:44.078728 lxc700b5d19881b In  IP 10.2.3.143.53337 > 10.2.0.42.53: Flags [.], ack 155, win 131, options [nop,nop,TS val 3298041820 ecr 833701057], length 0
02:47:44.078758 eth0  Out IP 10.2.3.143.53337 > 10.2.0.42.53: Flags [.], ack 155, win 131, options [nop,nop,TS val 3298041820 ecr 833701057], length 0

tcpdump policy on, as noted it still tries to go to 10.3.0.10 instead of 10.2.3.72

02:45:11.932993 lxc700b5d19881b In  IP 10.2.3.143.44363 > 10.3.0.10.53: Flags [S], seq 1070528412, win 65535, options [mss 1460,sackOK,TS val 2627356549 ecr 0,nop,wscale 9], length 0
02:45:11.935114 lxc700b5d19881b In  IP 10.2.3.143.42160 > 10.3.0.10.53: 2512+ NS? . (17)
02:45:12.949409 lxc700b5d19881b In  IP 10.2.3.143.44363 > 10.3.0.10.53: Flags [S], seq 1070528412, win 65535, options [mss 1460,sackOK,TS val 2627357565 ecr 0,nop,wscale 9], length 0
02:45:13.437332 lxc700b5d19881b In  IP 10.2.3.143.44013 > 10.3.0.10.53: 48493+ NS? . (17)
02:45:14.938081 lxc700b5d19881b In  IP 10.2.3.143.40206 > 10.3.0.10.53: 30742+ NS? . (17)
02:45:14.965460 lxc700b5d19881b In  IP 10.2.3.143.44363 > 10.3.0.10.53: Flags [S], seq 1070528412, win 65535, options [mss 1460,sackOK,TS val 2627359581 ecr 0,nop,wscale 9], length 0
02:45:16.438784 lxc700b5d19881b In  IP 10.2.3.143.45143 > 10.3.0.10.53: 31560+ NS? . (17)
02:45:17.939217 lxc700b5d19881b In  IP 10.2.3.143.55114 > 10.3.0.10.53: 53360+ NS? . (17)
02:45:19.125544 lxc700b5d19881b In  IP 10.2.3.143.44363 > 10.3.0.10.53: Flags [S], seq 1070528412, win 65535, options [mss 1460,sackOK,TS val 2627363741 ecr 0,nop,wscale 9], length 0
02:45:19.440629 lxc700b5d19881b In  IP 10.2.3.143.40681 > 10.3.0.10.53: 13073+ NS? . (17)
02:45:20.941308 lxc700b5d19881b In  IP 10.2.3.143.57037 > 10.3.0.10.53: 13942+ NS? . (17)
02:45:21.926173 lxc700b5d19881b In  IP 10.2.3.143.57207 > 10.3.0.10.53: Flags [S], seq 1423905828, win 65535, options [mss 1460,sackOK,TS val 2627366542 ecr 0,nop,wscale 9], length 0
02:45:22.443111 lxc700b5d19881b In  IP 10.2.3.143.40074 > 10.3.0.10.53: 29603+ NS? . (17)
02:45:22.933385 lxc700b5d19881b In  IP 10.2.3.143.57207 > 10.3.0.10.53: Flags [S], seq 1423905828, win 65535, options [mss 1460,sackOK,TS val 2627367549 ecr 0,nop,wscale 9], length 0
02:45:23.945404 lxc700b5d19881b In  IP 10.2.3.143.42131 > 10.3.0.10.53: 2748+ NS? . (17)

Anything else?

I have already tried the sysctl fix + reboot mentioned in #19909.

net.ipv4.conf.cilium_host.rp_filter=0
net.ipv4.conf.cilium_net.rp_filter=0
net.ipv4.conf.lo.rp_filter=0
net.ipv4.conf.lxc*.rp_filter=0

Slack thread https://cilium.slack.com/archives/C53TG4J4R/p1654571691219659

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/datapathImpacts bpf/ or low-level forwarding details, including map management and monitor messages.area/lrpImpacts Local Redirect Policy.kind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.kind/duplicateThere is another issue which contains additional details.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions