cmd/{k8s-operator,k8s-nameserver},k8s-operator: update nameserver config with records for ingress/egress proxies #11019

irbekrm · 2024-02-02T19:29:00Z

This PR adds functionality to the operator to optionally populate DNS records config for the in-cluster ts.net nameserver (added in #11017) with records for cluster proxies' MagicDNS names:

For Tailscale Ingress the record is a mapping between Ingresss MagicDNS name and the proxy's Pod IP
For cluster egress proxies configured via tailscale.com/tailnet-fqdn annotation the record is a mapping between the annotation's value and the proxy's Pod IP

No records are currently created for any other proxy types.

The records are created only if there is a ready tailscale.com/v1alpha1.DNSConfig instance in the cluster (see #11017 ) i.e if users have explicitly configured the operator to deploy the in-cluster ts.net nameserver.

Below see the user flow for a setup where:

cluster A runs a service exposed over Tailscale Ingress that should be accesible by callers on tailnet as well as cluster workloads via the same DNS name (the MagicDNS of the Ingress) (see use case description).
cluster A uses kube-dns.
cluster B's workloads also need to access the service in cluster A and would like to do that using the same Ingress. They can do it by exposing the Ingress to their cluster (see cross-cluster connectivity). Because A's Service is exposed over HTTPS, they need to use the MagicDNS name of the Ingress (so it needs to resolve within the cluster). See a use case.
cluster B uses CoreDNS.

Details

In cluster A:

Deploy the operator
Create a DNSConfig to tell the operator to deploy an in-cluster ts.net nameserver, i.e


$ kubectl apply -f ./cmd/k8s-operator/examples/dnsconfig.yaml

Update kube-dns ConfigMap to tell kube-dns to use the ts.net nameserver to resolve ts.net DNS names


$ k get dnsconfig
NAME     NAMESERVERIP
ts-dns   10.27.1.90
$ kubectl get cm kube-dns -n kube-system -oyaml
apiVersion: v1
data:
  stubDomains: |
    {
      "ts.net": ["10.27.1.90"]
    }
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
...

Create a workload exposed via tailscale Ingress.
Note the use of tailscale.com/experimental-forward-cluster-traffic-via-ingress annotation- this makes the ingress proxy to listen on the Pod's IP address which it does not do by default.


apiVersion: apps/v1
kind: Deployment
metadata:
  name: kuardfoobar
spec:
  selector:
    matchLabels:
      app: kuardfoobar
  template:
    metadata:
      labels:
        app: kuardfoobar
    spec:
      containers:
      - image: gcr.io/kuar-demo/kuard-amd64:1
        name: kuardfoobar
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: kuardfoobar
spec:
  ports:
  - port: 80
    targetPort: 8080
  type: ClusterIP
  selector:
    app: kuardfoobar
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kuardfoobar 
  annotations:
    tailscale.com/experimental-forward-cluster-traffic-via-ingress: "true"
spec:
  tls:
  - hosts:
    - "dnstest"
  rules:
  - http:
      paths:
      - backend:
          service:
            name: kuardfoobar
            port:
              number: 80
        pathType: Prefix 
        path: /
  ingressClassName: tailscale

Observe the MagicDNS name of the Ingress and that the workload can be accessed over that from a non-tailscale Pod in cluster:


$ kubectl get ingress
NAME          CLASS       HOSTS   ADDRESS                      PORTS     AGE
kuardfoobar   tailscale   *       dnstest-3.tailbd97a.ts.net   80, 443   64m
$ kubectl exec -it somepod -- sh
/ # curl -vvv https://dnstest-3.tailbd97a.ts.net
* Host dnstest-3.tailbd97a.ts.net:443 was resolved.
* IPv6: (none)
* IPv4: 10.0.0.6
*   Trying 10.0.0.6:443...
* Connected to dnstest-3.tailbd97a.ts.net (10.0.0.6) port 443
....

In cluster B:

Deploy the operator
Create a DNSConfig to tell the operator to deploy an in-cluster ts.net nameserver, i.e


$ kubectl apply -f ./cmd/k8s-operator/examples/dnsconfig.yaml

Update CoreDNS config to tell it to use the ts.net nameserver to resolve ts.net DNS names:


$ kubectl get dnsconfig
NAME     NAMESERVERIP
ts-dns   10.100.198.212
$ kubectl get cm coredns -n kube-system -oyaml
apiVersion: v1
data:
  Corefile: |
    ts.net {
      cache 5
      errors
      forward . 10.100.198.212
    }
    .:53 {
       
    }
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system

Create a tailscale egress proxy with a tailscale.com/tailnet-fqdn annotation pointing at the Ingress in cluster A:


kind: Service
metadata:
  annotations:
    tailscale.com/tailnet-fqdn: dnstest-3.tailbd97a.ts.net
  name: kuard-egress
spec:
  type: ExternalName

Verify that you can access the workload in cluster A over the MagicDNS name of Ingress in cluster A from a non-tailscale Pod in cluster B:


$ kubectl exec -it sompod -- sh
 # curl -vvv https://dnstest-3.tailbd97a.ts.net
* Host dnstest-3.tailbd97a.ts.net:443 was resolved.
* IPv6: (none)
* IPv4: 172.31.24.228
*   Trying 172.31.24.228:443...
* Connected to dnstest-3.tailbd97a.ts.net (172.31.24.228) port 443
...

Notes:

users have to manually update kube-dns/CoreDNS config to add the stub nameserver's Service IP
IPv6 Pod addresses currently not supported
To access Tailscale Ingress from within a cluster via its MagicDNS name, users also have to annotate the Ingress resource with tailscale.com/experimental-forward-cluster-traffic-via-ingress annotation, see
cmd/{containerboot,k8s-operator/deploy/manifests}: optionally allow proxying cluster traffic to a cluster target via ingress proxy #11036
I have also tested that non-operator managed tailscale instances that accept Tailscale DNS, running in a Pod in a cluster configured with our nameserver, still use 100.100.100.100 to resolve tailnet DNS names. Operator configured proxies do not accept Tailscale DNS, so they would use the ts.net nameserver. I cannot think of cases where this would be an issue (because they don't need to talk to anything on tailnet via MagicDNS)

cc @jaxxstorm

Next steps:

update build infra to publish nameserver images

irbekrm · 2024-02-02T21:13:37Z

cmd/k8s-operator/svc.go

 // annotation or of the deprecated tailscale.com/ts-tailnet-target-ip
 // annotation. If neither is set, it returns an empty string. If both are set,
 // it returns the value of the new annotation.
-func (a *ServiceReconciler) tailnetTargetAnnotation(svc *corev1.Service) string {


These changes make these functions re-usable for dns-records reconciler.

irbekrm · 2024-04-04T14:12:04Z

cmd/k8s-operator/deploy/examples/dnsconfig.yaml

+  nameserver:
+    image:
+      repo: gcr.io/csi-test-290908/nameserver
+      tag: v0.0.1ns


I will add functionality to actually push the nameserver image in a follow-up PR

irbekrm · 2024-04-04T14:16:16Z

cmd/k8s-operator/dnsrecords.go

+// The records that it creates are:
+//   - For tailscale Ingress, a mapping of the Ingress's MagicDNSName to the IP address of
+//     the ingress proxy Pod.
+//   - For egress proxies configured via tailscale.com/tailnet-fqdn annotation, a
+//     mapping of the the tailnet FQDN to the IP address of the egress proxy Pod.
+//


Initially I was creating records for all proxy types. However, the L3 ingress proxies don't route cluster traffic to the backend. Also no-one has asked for this functionality for cluster ingress proxies and with regards to the egress proxies, it seems good enough to only implement it for the ones configured via tailscale.com/tailnet-fqdn annotation and not tailscale.com/tailnet-ip. We can always add more functionality later.

irbekrm · 2024-04-04T14:30:24Z

cmd/k8s-operator/dnsrecords.go

+// egress configured via tailscale.com/tailnet-fqdn annotation.
+//
+// For Ingress, the record is a mapping between the MagicDNSName of the Ingress, retrieved from
+// ingress.status.loadBalancer.ingress.hostname field and the proxy Pod IP addresses


Currently always a single IP address

irbekrm · 2024-04-04T15:01:09Z

cmd/k8s-operator/dnsrecords.go

+	ips := make([]string, 0)
+	for _, ep := range eps.Endpoints {
+		for _, ip := range ips {
+			if !net.IsIPv4String(ip) {


I'll add IPv6 support in a follow-up. At the moment the nameserver only responds to queries for A records.

Scalahansolo · 2024-04-29T16:31:17Z

What's the status of this? did this PR get abandoned in favor of a different approach?

irbekrm · 2024-04-29T17:11:15Z

Hi @Scalahansolo , it's not abandoned, just that we're a bit low bandwidth right now. The current plan is still to get it in.

…ords for proxies. This commit adds functionality to automatically populate DNS records for the in-cluster ts.net nameserver to allow cluster workloads to resolve MagicDNS names associated with operator's proxies. The records are created as follows: * For tailscale Ingress proxies there will be a record mapping the MagicDNS name of the Ingress device and each proxy Pod's IP address. * For cluster egress proxies, configured via tailscale.com/tailnet-fqdn annotation, there will be a record for each proxy Pod, mapping the MagicDNS name of the exposed tailnet workload to the proxy Pod's IP. No records will be created for any other proxy types. Records will only be created if users have configured the operator to deploy an in-cluster ts.net nameserver by applying tailscale.com/v1alpha1.DNSConfig. It is user's responsibility to add the ts.net nameserver as a stub nameserver for ts.net DNS names. https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#configuration-of-stub-domain-and-upstream-nameserver-using-coredns https://cloud.google.com/kubernetes-engine/docs/how-to/kube-dns#upstream_nameservers See also #11017 Updates #10499 Signed-off-by: Irbe Krumina <irbe@tailscale.com>

oxtoacart

Generally LGTM, though I'm not familiar with the K8s APIs being used.

I left a few comments/questions.

cmd/k8s-operator/dnsrecords.go

irbekrm · 2024-05-02T14:05:25Z

k8s-operator/apis/v1alpha1/types_tsdnsconfig.go

 	Conditions []ConnectorCondition `json:"conditions"`
 	// +optional
-	NameserverStatus *NameserverStatus `json:"nameserverStatus"`
+	Nameserver *NameserverStatus `json:"nameserver"`


Updated this to not have dnsconfig.status.nameserverStatus

Signed-off-by: Irbe Krumina <irbe@tailscale.com>

Updates #11019 Signed-off-by: Irbe Krumina <irbe@tailscale.com>

…NSConfig description Also removes hardcoded image repo/tag from example DNSConfig resource as the operator now knows how to default those. Updates #11019 Signed-off-by: Irbe Krumina <irbe@tailscale.com>

…NSConfig description (#11971) Also removes hardcoded image repo/tag from example DNSConfig resource as the operator now knows how to default those. Updates #11019 Signed-off-by: Irbe Krumina <irbe@tailscale.com>

…fig with records for ingress/egress proxies (tailscale#11019) cmd/k8s-operator: optionally update dnsrecords Configmap with DNS records for proxies. This commit adds functionality to automatically populate DNS records for the in-cluster ts.net nameserver to allow cluster workloads to resolve MagicDNS names associated with operator's proxies. The records are created as follows: * For tailscale Ingress proxies there will be a record mapping the MagicDNS name of the Ingress device and each proxy Pod's IP address. * For cluster egress proxies, configured via tailscale.com/tailnet-fqdn annotation, there will be a record for each proxy Pod, mapping the MagicDNS name of the exposed tailnet workload to the proxy Pod's IP. No records will be created for any other proxy types. Records will only be created if users have configured the operator to deploy an in-cluster ts.net nameserver by applying tailscale.com/v1alpha1.DNSConfig. It is user's responsibility to add the ts.net nameserver as a stub nameserver for ts.net DNS names. https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#configuration-of-stub-domain-and-upstream-nameserver-using-coredns https://cloud.google.com/kubernetes-engine/docs/how-to/kube-dns#upstream_nameservers See also tailscale#11017 Updates tailscale#10499 Signed-off-by: Irbe Krumina <irbe@tailscale.com>

…NSConfig description (tailscale#11971) Also removes hardcoded image repo/tag from example DNSConfig resource as the operator now knows how to default those. Updates tailscale#11019 Signed-off-by: Irbe Krumina <irbe@tailscale.com>

…fig with records for ingress/egress proxies (tailscale#11019) cmd/k8s-operator: optionally update dnsrecords Configmap with DNS records for proxies. This commit adds functionality to automatically populate DNS records for the in-cluster ts.net nameserver to allow cluster workloads to resolve MagicDNS names associated with operator's proxies. The records are created as follows: * For tailscale Ingress proxies there will be a record mapping the MagicDNS name of the Ingress device and each proxy Pod's IP address. * For cluster egress proxies, configured via tailscale.com/tailnet-fqdn annotation, there will be a record for each proxy Pod, mapping the MagicDNS name of the exposed tailnet workload to the proxy Pod's IP. No records will be created for any other proxy types. Records will only be created if users have configured the operator to deploy an in-cluster ts.net nameserver by applying tailscale.com/v1alpha1.DNSConfig. It is user's responsibility to add the ts.net nameserver as a stub nameserver for ts.net DNS names. https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#configuration-of-stub-domain-and-upstream-nameserver-using-coredns https://cloud.google.com/kubernetes-engine/docs/how-to/kube-dns#upstream_nameservers See also tailscale#11017 Updates tailscale#10499 Signed-off-by: Irbe Krumina <irbe@tailscale.com>

…NSConfig description (tailscale#11971) Also removes hardcoded image repo/tag from example DNSConfig resource as the operator now knows how to default those. Updates tailscale#11019 Signed-off-by: Irbe Krumina <irbe@tailscale.com>

irbekrm marked this pull request as draft February 2, 2024 19:29

irbekrm changed the title ~~Irbekrm/kubednsrecords~~ cmd/k8s-operator: update nameserver config with records for ingress/egress proxies Feb 2, 2024

irbekrm force-pushed the irbekrm/kubednsrecords branch 4 times, most recently from 15ec9ad to 741167e Compare February 2, 2024 20:01

irbekrm mentioned this pull request Feb 2, 2024

cmd/{k8s-nameserver,k8s-operator},k8s-operator: add a kube nameserver, make operator deploy it #11017

Merged

irbekrm commented Feb 2, 2024

View reviewed changes

irbekrm force-pushed the irbekrm/kubednsrecords branch 3 times, most recently from e2b89bb to caf7cdb Compare February 4, 2024 20:31

irbekrm mentioned this pull request Feb 5, 2024

cmd/{containerboot,k8s-operator/deploy/manifests}: optionally allow proxying cluster traffic to a cluster target via ingress proxy #11036

Merged

irbekrm force-pushed the irbekrm/kubednsrecords branch 2 times, most recently from a6ff826 to 703e7af Compare February 5, 2024 19:50

irbekrm mentioned this pull request Feb 5, 2024

FR: Kubernetes operator: make it possible to resolve Tailnet FQDNs from within the Kubernetes cluster #10499

Closed

irbekrm mentioned this pull request Mar 29, 2024

K8s Operator: Cross cluster Ingress / Egress not working #11500

Open

irbekrm force-pushed the irbekrm/kubednsrecords branch 3 times, most recently from 256cc93 to 58766a6 Compare April 4, 2024 14:09

irbekrm commented Apr 4, 2024

View reviewed changes

irbekrm force-pushed the irbekrm/kubednsrecords branch from 58766a6 to e2fd26a Compare April 4, 2024 14:18

irbekrm commented Apr 4, 2024

View reviewed changes

irbekrm force-pushed the irbekrm/kubednsrecords branch 2 times, most recently from 1567e91 to 7c266d6 Compare April 4, 2024 14:54

irbekrm commented Apr 4, 2024

View reviewed changes

irbekrm force-pushed the irbekrm/kubednsrecords branch 4 times, most recently from be183ff to dd28a4a Compare April 4, 2024 15:17

This was referenced Apr 8, 2024

Expose Kubernetes API Server MagicDNS w/ Egress + SSL #11647

Open

cmd/{k8s-nameserver,k8s-operator},k8s-operator: reverts k8s nameserver functionality #11669

Merged

irbekrm marked this pull request as draft April 9, 2024 06:28

irbekrm mentioned this pull request Apr 29, 2024

cmd/{k8s-nameserver,k8s-operator},k8s-operator: add a kube nameserver, make operator deploy it #11919

Merged

irbekrm force-pushed the irbekrm/kubednsrecords branch from dd28a4a to 615f714 Compare April 29, 2024 18:04

irbekrm force-pushed the irbekrm/kubednsrecords branch from 615f714 to 3f8e137 Compare April 30, 2024 19:21

irbekrm marked this pull request as ready for review April 30, 2024 19:22

oxtoacart approved these changes May 1, 2024

View reviewed changes

irbekrm force-pushed the irbekrm/kubednsrecords branch from 61e1240 to 4b77860 Compare May 2, 2024 14:04

irbekrm commented May 2, 2024

View reviewed changes

code review feedback

0bfaaa5

Signed-off-by: Irbe Krumina <irbe@tailscale.com>

irbekrm force-pushed the irbekrm/kubednsrecords branch from 4b77860 to 0bfaaa5 Compare May 2, 2024 14:06

irbekrm merged commit 19b31ac into main May 2, 2024

irbekrm deleted the irbekrm/kubednsrecords branch May 2, 2024 16:29

irbekrm added a commit that referenced this pull request May 2, 2024

k8s-operator/apis/v1alpha1: update DNSConfig description

d79411c

Updates #11019 Signed-off-by: Irbe Krumina <irbe@tailscale.com>

irbekrm mentioned this pull request May 2, 2024

k8s-operator/apis/v1alpha1: update DNSConfig CRD description #11971

Merged

irbekrm added a commit that referenced this pull request May 2, 2024

k8s-operator/apis/v1alpha1: update DNSConfig description

d6cc266

Updates #11019 Signed-off-by: Irbe Krumina <irbe@tailscale.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

cmd/{k8s-operator,k8s-nameserver},k8s-operator: update nameserver config with records for ingress/egress proxies #11019

cmd/{k8s-operator,k8s-nameserver},k8s-operator: update nameserver config with records for ingress/egress proxies #11019

Uh oh!

irbekrm commented Feb 2, 2024 •

edited

Loading

Uh oh!

irbekrm Feb 2, 2024

Uh oh!

irbekrm Apr 4, 2024

Uh oh!

irbekrm Apr 4, 2024

Uh oh!

irbekrm Apr 4, 2024 •

edited

Loading

Uh oh!

irbekrm Apr 4, 2024 •

edited

Loading

Uh oh!

Scalahansolo commented Apr 29, 2024

Uh oh!

irbekrm commented Apr 29, 2024

Uh oh!

oxtoacart left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

irbekrm May 2, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cmd/{k8s-operator,k8s-nameserver},k8s-operator: update nameserver config with records for ingress/egress proxies #11019

cmd/{k8s-operator,k8s-nameserver},k8s-operator: update nameserver config with records for ingress/egress proxies #11019

Uh oh!

Conversation

irbekrm commented Feb 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

irbekrm Feb 2, 2024

Choose a reason for hiding this comment

Uh oh!

irbekrm Apr 4, 2024

Choose a reason for hiding this comment

Uh oh!

irbekrm Apr 4, 2024

Choose a reason for hiding this comment

Uh oh!

irbekrm Apr 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

irbekrm Apr 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Scalahansolo commented Apr 29, 2024

Uh oh!

irbekrm commented Apr 29, 2024

Uh oh!

oxtoacart left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

irbekrm May 2, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

irbekrm commented Feb 2, 2024 •

edited

Loading

irbekrm Apr 4, 2024 •

edited

Loading

irbekrm Apr 4, 2024 •

edited

Loading