-
Notifications
You must be signed in to change notification settings - Fork 3.7k
CI: ci-l7: Timeout waiting for response to forwarded proxied DNS lookup #43930
Copy link
Copy link
Open
Labels
area/CIContinuous Integration testing issue or flakeContinuous Integration testing issue or flakearea/fqdnAffects the FQDN policies featureAffects the FQDN policies featurearea/proxyImpacts proxy components, including DNS, Kafka, Envoy and/or XDS servers.Impacts proxy components, including DNS, Kafka, Envoy and/or XDS servers.ci/flakeThis is a known failure that occurs in the tree. Please investigate me!This is a known failure that occurs in the tree. Please investigate me!
Description
CI failure
seen at https://github.com/cilium/cilium/actions/runs/21025016183/job/60447404626
✅ The sysdump has been saved to cilium-sysdump-conn-disrupt-test-cilium-upgrade-ipsec-10-concurrent-20260115-085711.zip
[.] Action [check-log-errors/no-errors-in-logs:pkg/fqdn/dnsproxy:kind-kind/kube-system/cilium-lfmzj (config)]
. [.] Action [check-log-errors/no-errors-in-logs:pkg/fqdn/dnsproxy:kind-kind/kube-system/cilium-lfmzj (mount-cgroup)]
. [.] Action [check-log-errors/no-errors-in-logs:pkg/fqdn/dnsproxy:kind-kind/kube-system/cilium-lfmzj (apply-sysctl-overwrites)]
. [.] Action [check-log-errors/no-errors-in-logs:pkg/fqdn/dnsproxy:kind-kind/kube-system/cilium-lfmzj (mount-bpf-fs)]
. [.] Action [check-log-errors/no-errors-in-logs:pkg/fqdn/dnsproxy:kind-kind/kube-system/cilium-operator-77f45489bb-vszwx (cilium-operator)]
.
📋 Test Report [cilium-test-1]
❌ 1/2 tests failed (1/28 actions), 8 tests skipped, 0 scenarios skipped:
Test [check-log-errors]:
🟥 check-log-errors/no-errors-in-logs:pkg/fqdn/dnsproxy:kind-kind/kube-system/cilium-lfmzj (cilium-agent): Found 1 logs in kind-kind/kube-system/cilium-lfmzj (cilium-agent) matching list of errors that must be investigated:
time=2026-01-15T08:51:19.214922113Z level=warn source=/go/src/github.com/cilium/cilium/pkg/fqdn/dnsproxy/proxy.go:1112 msg="Timeout waiting for response to forwarded proxied DNS lookup" module=agent.controlplane.fqdn.dns-proxy dnsName=fake.external.service.cilium.svc.cluster.local. ipAddr=10.244.3.153:50441 DNSRequestID=37254 endpointID=104 identity=13662 error="read udp 10.244.3.153:50441->10.244.3.162:53: i/o timeout" (1 occurrences)
⛑️ The following owners are responsible for reliability of the testsuite:
- @cilium/fqdn (no-errors-in-logs:pkg/fqdn/dnsproxy)
- @cilium/ci-structure (.github/workflows/conformance-l7.yaml)
[cilium-test-1] 1 tests failed
cilium-sysdump-conn-disrupt-test-cilium-upgrade-ipsec-10-concurrent-20260115-085711.zip
pod: cilium-lfmzj
looked at the sysdump, we're getting an NXDOMAIN from the upstream DNS, and simply forwarding it to the client .
2026-01-15T08:51:09.205526884Z time=2026-01-15T08:51:09.205456297Z level=debug source=/go/src/github.com/cilium/cilium/pkg/fqdn/dnsproxy/proxy.go:960 msg="Handling DNS query from endpoint" module=agent.controlplane.fqdn.dns-proxy dnsName=fake.external.service.cilium.cilium-test-4.svc.cluster.local. ipAddr=10.244.3.153:50441 DNSRequestID=36472
2026-01-15T08:51:09.205932350Z time=2026-01-15T08:51:09.205730611Z level=debug source=/go/src/github.com/cilium/cilium/pkg/fqdn/dnsproxy/proxy.go:1007 msg="Found target server to of DNS request secID" module=agent.controlplane.fqdn.dns-proxy dnsName=fake.external.service.cilium.cilium-test-4.svc.cluster.local. ipAddr=10.244.3.153:50441 DNSRequestID=36472 endpointID=104 identity=13662 secID="{Source:custom-resource overwrittenLegacySource: ID:63376 _:[] modifiedByLegacyAPI:true shadowed:false}" server=10.244.3.162
2026-01-15T08:51:09.205941248Z time=2026-01-15T08:51:09.205834671Z level=debug source=/go/src/github.com/cilium/cilium/pkg/fqdn/dnsproxy/proxy.go:1043 msg="Forwarding DNS request for a name that is allowed" module=agent.controlplane.fqdn.dns-proxy dnsName=fake.external.service.cilium.cilium-test-4.svc.cluster.local. ipAddr=10.244.3.153:50441 DNSRequestID=36472 endpointID=104 identity=13662
2026-01-15T08:51:09.210227115Z time=2026-01-15T08:51:09.209963815Z level=debug source=/go/src/github.com/cilium/cilium/pkg/fqdn/dnsproxy/proxy.go:1123 msg="Received DNS response to proxied lookup" module=agent.controlplane.fqdn.dns-proxy dnsName=fake.external.service.cilium.cilium-test-4.svc.cluster.local. ipAddr=10.244.3.153:50441 DNSRequestID=36472 endpointID=104 identity=13662 resp=";; opcode: QUERY, status: NXDOMAIN, id: 36472\n;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 1232\n\n;; QUESTION SECTION:\n;fake.external.service.cilium.cilium-test-4.svc.cluster.local.\tIN\t A\n\n;; AUTHORITY SECTION:\ncluster.local.\t30\tIN\tSOA\tns.dns.cluster.local. hostmaster.cluster.local. 1768466985 7200 1800 86400 30\n"
2026-01-15T08:51:09.210388472Z time=2026-01-15T08:51:09.210254498Z level=debug source=/go/src/github.com/cilium/cilium/pkg/fqdn/dnsproxy/proxy.go:1126 msg="Notifying with DNS response to original DNS query" module=agent.controlplane.fqdn.dns-proxy dnsName=fake.external.service.cilium.cilium-test-4.svc.cluster.local. ipAddr=10.244.3.153:50441 DNSRequestID=36472 endpointID=104 identity=13662
2026-01-15T08:51:09.211065352Z time=2026-01-15T08:51:09.210937413Z level=debug source=/go/src/github.com/cilium/cilium/pkg/fqdn/dnsproxy/proxy.go:1137 msg="Responding to original DNS query" module=agent.controlplane.fqdn.dns-proxy dnsName=fake.external.service.cilium.cilium-test-4.svc.cluster.local. ipAddr=10.244.3.153:50441 DNSRequestID=36472 endpointID=104 identity=13662
2026-01-15T08:51:09.213020036Z time=2026-01-15T08:51:09.211859631Z level=debug source=/go/src/github.com/cilium/cilium/pkg/fqdn/dnsproxy/udp.go:232 msg="dnsproxy: Wrote DNS response" module=agent.controlplane.fqdn.dns-proxy writtenBytes=182 totalBytes=182 source=10.244.3.162:53 destination=10.244.3.153:50441
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
area/CIContinuous Integration testing issue or flakeContinuous Integration testing issue or flakearea/fqdnAffects the FQDN policies featureAffects the FQDN policies featurearea/proxyImpacts proxy components, including DNS, Kafka, Envoy and/or XDS servers.Impacts proxy components, including DNS, Kafka, Envoy and/or XDS servers.ci/flakeThis is a known failure that occurs in the tree. Please investigate me!This is a known failure that occurs in the tree. Please investigate me!