There is something wrong with DNS in Azure, not very clear what it is yet - more details to follow.
One way it manifest itself is that pods deployed in kube-system, such as Hubble UI, fail to resolve $KUBERNETES_SERVICE_HOST. It turns out that in AKS the value of KUBERNETES_SERVICE_HOST gets set to something like ilya-test--ilya-test-1-da2a1f-9923c925.hcp.westeurope.azmk8s.io for pods in kube-system, and more traditional service IP in all other namespaces.
Quite crucially, it appears that quite a few of connectivity test pods are not reaching ready state at all:
echo-a-558b9b6dc4-pmsh8 1/1 Running 0 5h42m
echo-b-59d5ff8b98-r4hx8 1/1 Running 0 5h42m
echo-b-host-f4bd98474-rbpgz 1/1 Running 0 5h42m
host-to-b-multi-node-clusterip-7bb8b4f964-qgsf6 1/1 Running 52 5h42m
host-to-b-multi-node-headless-5c5676647b-56xbt 1/1 Running 50 5h42m
pod-to-a-646cccc5df-t8blr 1/1 Running 101 5h42m
pod-to-a-allowed-cnp-56f4cfd999-fnppn 0/1 CrashLoopBackOff 99 5h42m
pod-to-a-external-1111-7c5c99c6d9-gnmfk 1/1 Running 0 5h42m
pod-to-a-l3-denied-cnp-5dc8d69b7f-q4nvb 1/1 Running 0 5h42m
pod-to-b-intra-node-b9454c7c6-sc9lq 0/1 CrashLoopBackOff 99 5h42m
pod-to-b-intra-node-nodeport-6cc56666dc-tmqt9 0/1 CrashLoopBackOff 100 5h42m
pod-to-b-multi-node-clusterip-754d5ff9d-9gzwg 0/1 CrashLoopBackOff 99 5h42m
pod-to-b-multi-node-headless-7876749b84-sz4zz 1/1 Running 46 5h42m
pod-to-b-multi-node-nodeport-6d8fc65c99-ld8hv 0/1 CrashLoopBackOff 99 5h42m
pod-to-external-fqdn-allow-google-cnp-6478db9cd9-d74xk 0/1 CrashLoopBackOff 99 5h42m
There is something wrong with DNS in Azure, not very clear what it is yet - more details to follow.
One way it manifest itself is that pods deployed in
kube-system, such as Hubble UI, fail to resolve$KUBERNETES_SERVICE_HOST. It turns out that in AKS the value ofKUBERNETES_SERVICE_HOSTgets set to something likeilya-test--ilya-test-1-da2a1f-9923c925.hcp.westeurope.azmk8s.iofor pods inkube-system, and more traditional service IP in all other namespaces.Quite crucially, it appears that quite a few of connectivity test pods are not reaching ready state at all: