[Kubernetes] Make HA master setup compatible with SONiC buildimage join process by isabelmsft · Pull Request #2306 · sonic-net/sonic-mgmt

isabelmsft · 2020-10-04T04:29:33Z

Description of PR

Summary:
Make HA master setup compatible with automated SONiC buildimage join process

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Approach

What is the motivation for this PR?

The current SONiC buildimage assumes the Kubernetes conf file is available at HTTP server on master; this conf file is used to join the SONiC worker node to the Kubernetes master. Previously, the HA testbed master expected SONiC worker nodes to join manually via a command output during master initialization. This PR allows automated join of the SONiC DUT to the HA Kubernetes master by downloading the necessary conf file from a new HTTP server hosted at the master VIP.

How did you do it?

Host Apache server on HAProxy node, as the SONiC worker node communicates with the VIP (IP of HAProxy node)
Set up necessary firewall and SSL certs
Copy conf file from Kubernetes master to HAProxy node and enable download
Change HAProxy node to be listening on port 6443 (to be compatible with port_check in SONiC buildimage)

How did you verify/test it?

Created HA master on testbed server via testbed-cli.sh script
Joined SONiC DUT automatically by configuring master server VIP via sudo config kube server ip <HAPROXY NODE IP/VIP>
show kube server on SONiC DUT shows master is connected
kubectl get nodes on master shows all 3 master nodes and SONiC DUT

qiluo-msft · 2020-10-20T22:02:20Z

ansible/roles/k8s_master/tasks/init_master_leader.yml

+  become: yes
+  fetch:
+    src: /etc/kubernetes/admin.conf
+    dest: /tmp/helper/admin.conf


/tmp/helper/ [](start = 10, length = 12)

Why not set dest to /var/www/html directly? #Closed

This saves the file to host server so that the file can be transferred from one VM to another VM. init_master_leader.yml is running on Master1 VM, but I need the file on the HAProxy VM

Then you may remove this fetch completely, and use below copy with src as /etc/kubernetes/admin.conf ?

In reply to: 508867244 [](ancestors = 508867244)

Then I get this error:
"Could not find or access '/etc/kubernetes/admin.conf' on the Ansible Controller. When I wrote the original code I also thought there would be a more straightforward way to copy the file from one VM to another, but using the /tmp directory was the only way I tried that worked for me

I understand now. Please give correct comments, especially on the src/dest machines.

In reply to: 508884204 [](ancestors = 508884204,508867244)

To clarify, I get that error because the /etc/kubernetes/admin.conf is on the server, not on the VM.
For fetch module:

src is the master VM, on which /etc/kubernetes/admin.conf is present

dest is the server

For copy module:

src is the server

dest is the HAProxy VM

qiluo-msft · 2020-10-20T22:02:50Z

ansible/roles/k8s_master/tasks/init_master_leader.yml

+  copy:
+    src: /tmp/helper/admin.conf
+    dest: /var/www/html/admin.conf
+    mode: '777'


777 [](start = 11, length = 3)

Everyone can write or del it, is it safe? #Closed

I changed to 755 permissions. This was the most restrictive I could make it while ensuring that the SONiC DUT could still download the file properly. This is also in-line with the response here https://askubuntu.com/questions/451922/apache-access-denied-because-search-permissions-are-missing

qiluo-msft · 2020-10-20T22:05:06Z

ansible/roles/k8s_haproxy/tasks/main.yml

+  file:
+    path: "/var/www/html"
+    state: directory
+    mode: '777'


777 [](start = 11, length = 3)

Is it too broad permission? #Closed

I changed to 755 permissions. This was the most restrictive I could make it while ensuring that the SONiC DUT could still download the file properly. This is also in-line with the response here https://askubuntu.com/questions/451922/apache-access-denied-because-search-permissions-are-missing

qiluo-msft · 2020-10-20T22:06:03Z

ansible/roles/k8s_haproxy/tasks/main.yml

+    src: "default-ssl.conf.j2"
+    dest: "/etc/apache2/sites-available/default-ssl.conf"
+
+- name: Enable Apaache SSL


Apaache [](start = 15, length = 7)

typo? #Closed

Yes, fixed :)

qiluo-msft · 2020-10-20T22:06:32Z

ansible/roles/k8s_master/tasks/init_master_leader.yml


 - name: Initializing Kubernetes cluster
-  shell: kubeadm init --control-plane-endpoint {{ haproxy_ip }}:80 --upload-certs --apiserver-advertise-address {{ ansible_ens2.ipv4.address }} --skip-phases=addon/kube-proxy --skip-phases=addon/coredns --ignore-preflight-errors=NumCPU
+  shell: kubeadm init --control-plane-endpoint {{ haproxy_ip }}:{{ k8s_port }} --upload-certs --apiserver-advertise-address {{ ansible_ens2.ipv4.address }} --skip-phases=addon/kube-proxy --skip-phases=addon/coredns


beadm init --control-plane-endpoint {{ haproxy_ip }}:{{ k8 [](start = 11, length = 58)

This line is too long to read. Could you split? #Closed

Yes, I split it

qiluo-msft

As comments

isabelmsft · 2020-10-28T00:55:09Z

retest vsimage please

Including change: * 7ff8f75 2022-06-03 | Revert "[portsorch]: Prevent LAG member configuration when port has active ACL binding (sonic-net#2165)" (sonic-net#2306) (HEAD -> 202205, github/202205) [bingwang-ms] Signed-off-by: Ying Xie <ying.xie@microsoft.com>

isabelmsft and others added 3 commits October 3, 2020 05:21

Make compatible w systemd integration

2b06a63

Finish making compatible with systemd integration

99be079

Fixed permissions and config file format

9ac1fcf

isabelmsft changed the title ~~[Kubernetes] Make HA testbed K8s master compatible with kube-systemd feature buildimage changes~~ [Kubernetes] Make HA master setup compatible with SONiC buildimage join process Oct 4, 2020

lguohan requested a review from renukamanavalan October 4, 2020 19:08

Remove coredns service

f2aa290

isabelmsft requested a review from qiluo-msft October 12, 2020 20:47

qiluo-msft reviewed Oct 20, 2020

View reviewed changes

Address comments from PR

0929ba6

Clarify note for admin.conf transfer onto HAProxy node

9c2f497

qiluo-msft approved these changes Oct 28, 2020

View reviewed changes

isabelmsft merged commit 725a50c into sonic-net:master Oct 28, 2020

Conversation

isabelmsft commented Oct 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Uh oh!

qiluo-msft Oct 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qiluo-msft Oct 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qiluo-msft Oct 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qiluo-msft Oct 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qiluo-msft Oct 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qiluo-msft left a comment

Choose a reason for hiding this comment

Uh oh!

isabelmsft commented Oct 28, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

isabelmsft commented Oct 4, 2020 •

edited

Loading

qiluo-msft Oct 20, 2020 •

edited

Loading

qiluo-msft Oct 20, 2020 •

edited

Loading

qiluo-msft Oct 20, 2020 •

edited

Loading

qiluo-msft Oct 20, 2020 •

edited

Loading

qiluo-msft Oct 20, 2020 •

edited

Loading