@blueorangutan package

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Codecov Report

Attention: Patch coverage is 25.00000% with 12 lines in your changes missing coverage. Please review.

Project coverage is 16.14%. Comparing base (f2a6a2f) to head (3ac3af0).
Report is 1 commits behind head on 4.20.

@@            Coverage Diff            @@
##               4.20   #10861   +/-   ##
=========================================
  Coverage     16.13%   16.14%           
- Complexity    13239    13242    +3     
=========================================
  Files          5656     5656           
  Lines        497622   497626    +4     
  Branches      60339    60338    -1     
=========================================
+ Hits          80302    80320   +18     
+ Misses       408370   408355   -15     
- Partials       8950     8951    +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13370

@blueorangutan test

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[SF] Trillian test result (tid-13295)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 57243 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10861-t13295-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

@sjanssen15

you can build the projects in a container and copy the JAR to your management server, by following the steps below

    # 1. create docker container, and go into it

    docker run -d -it --name ubuntu22 ubuntu:22.04
    docker exec -it ubuntu22 bash

    # 2. clone repo

    apt update
    apt install -y git wget
    git clone https://github.com/apache/cloudstack.git -b 4.20.0.0

    # 3. apply patch


    wget https://github.com/apache/cloudstack/pull/10861.patch
    git config --global user.email "you@example.com"
    git config --global user.name "Your Name"
    git am 10861.patch
    git log -n2 # patch should be applied

    # 4. install build dependencies


    echo "y" | apt build-dep .

    git clone https://github.com/shapeblue/cloudstack-nonoss.git nonoss && cd nonoss && bash -x install-non-oss.sh && cd ..
    rm -fr nonoss

    # 5. build the project


    mvn -P developer,systemvm -Dnoredist clean install -DskipTests

    # 6. post-build

    # please copy client/target/cloud-client-ui-4.20.0.0.jar to  /usr/share/cloudstack-management/lib/cloudstack-4.20.0.0.jar on the management servers
    # restart cloudstack-management service

@sjanssen15

you can build the projects in a container and copy the JAR to your management server, by following the steps below

    # 1. create docker container, and go into it

    docker run -d -it --name ubuntu22 ubuntu:22.04
    docker exec -it ubuntu22 bash

    # 2. clone repo

    apt update
    apt install -y git wget
    git clone https://github.com/apache/cloudstack.git -b 4.20.0.0

    # 3. apply patch


    wget https://github.com/apache/cloudstack/pull/10861.patch
    git config --global user.email "you@example.com"
    git config --global user.name "Your Name"
    git am 10861.patch
    git log -n2 # patch should be applied

    # 4. install build dependencies


    echo "y" | apt build-dep .

    git clone https://github.com/shapeblue/cloudstack-nonoss.git nonoss && cd nonoss && bash -x install-non-oss.sh && cd ..
    rm -fr nonoss

    # 5. build the project


    mvn -P developer,systemvm -Dnoredist clean install -DskipTests

    # 6. post-build

    # please copy client/target/cloud-client-ui-4.20.0.0.jar to  /usr/share/cloudstack-management/lib/cloudstack-4.20.0.0.jar on the management servers
    # restart cloudstack-management service

Hi, thank you! I just build this version and it does seem to work now when creating a network and allocating a subnet. However I notice the CPU usages is 100% and the environment is very slow. Any idea if I should continu testing or first fix this issue?

Hi, thank you! I just build this version and it does seem to work now when creating a network and allocating a subnet. However I notice the CPU usages is 100% and the environment is very slow. Any idea if I should continu testing or first fix this issue?

@sjanssen15
it should not happen.
can you check management-server.log to see if there are some errors or exceptions ?

Hi, thank you! I just build this version and it does seem to work now when creating a network and allocating a subnet. However I notice the CPU usages is 100% and the environment is very slow. Any idea if I should continu testing or first fix this issue?

@sjanssen15 it should not happen. can you check management-server.log to see if there are some errors or exceptions ?

It seemed to be caused by 2 old KVM hosts that were trying to connect to this management server but they are from a previous installation, they were kinda DDOSing this machine. After adding them the issue disappeared. Tomorrow I will test some more regarding the router and stuff.

Hi, thank you! I just build this version and it does seem to work now when creating a network and allocating a subnet. However I notice the CPU usages is 100% and the environment is very slow. Any idea if I should continu testing or first fix this issue?

@sjanssen15 it should not happen. can you check management-server.log to see if there are some errors or exceptions ?

It seemed to be caused by 2 old KVM hosts that were trying to connect to this management server but they are from a previous installation, they were kinda DDOSing this machine. After adding them the issue disappeared. Tomorrow I will test some more regarding the router and stuff.

Good, thanks for testing

If there is no major issue, we can merge this PR into 4.20.1

Hi, thank you! I just build this version and it does seem to work now when creating a network and allocating a subnet. However I notice the CPU usages is 100% and the environment is very slow. Any idea if I should continu testing or first fix this issue?

@sjanssen15 it should not happen. can you check management-server.log to see if there are some errors or exceptions ?

It seemed to be caused by 2 old KVM hosts that were trying to connect to this management server but they are from a previous installation, they were kinda DDOSing this machine. After adding them the issue disappeared. Tomorrow I will test some more regarding the router and stuff.

Good, thanks for testing

If there is no major issue, we can merge this PR into 4.20.1

I'm really sorry to bother you, but I'm going from error to error. Can you see why I'm unable to run any instances? System VM's are running so KVM is working. I have 3 hosts added to the cluster. Could it be related to the network?
unable-to-allocate.txt

I'm really sorry to bother you, but I'm going from error to error. Can you see why I'm unable to run any instances? System VM's are running so KVM is working. I have 3 hosts added to the cluster. Could it be related to the network? unable-to-allocate.txt

it looks the vm template has a template tag as emty string ('')
please update template_tag in vm_template to NULL

I'm really sorry to bother you, but I'm going from error to error. Can you see why I'm unable to run any instances? System VM's are running so KVM is working. I have 3 hosts added to the cluster. Could it be related to the network? unable-to-allocate.txt

it looks the vm template has a template tag as emty string ('') please update template_tag in vm_template to NULL

To give you a quick update, still not sure how I fixed it but I reset some template permissions and tags and it seemed to start working. Strange since this is a clean install.

I have now a VXLAN segment with a virtual router attached to it. I'm able to ssh into the VR and ping the running instance (from a different host as well). I'm however unable to reach the public network where the VR is attached to. I do have the correct VLAN and type configured. On the KVM host itself I'm able to reach the public gateway via that cloudbridge.

I have now a VXLAN segment with a virtual router attached to it. I'm able to ssh into the VR and ping the running instance (from a different host as well). I'm however unable to reach the public network where the VR is attached to. I do have the correct VLAN and type configured. On the KVM host itself I'm able to reach the public gateway via that cloudbridge.

You cannot reach the gateway 10.4.152.1 from the VR, right ?

can you assign a temporary IP (in 10.4.152.1/24 network) to the linux bridge (brXXXX.1452) on the kvm host, and check if the .1 and .102 are reachable ?

I have now a VXLAN segment with a virtual router attached to it. I'm able to ssh into the VR and ping the running instance (from a different host as well). I'm however unable to reach the public network where the VR is attached to. I do have the correct VLAN and type configured. On the KVM host itself I'm able to reach the public gateway via that cloudbridge.

You cannot reach the gateway 10.4.152.1 from the VR, right ?

can you assign a temporary IP (in 10.4.152.1/24 network) to the linux bridge (brXXXX.1452) on the kvm host, and check if the .1 and .102 are reachable ?

I tried with ip addr add 10.4.152.14/24 dev brens1np1-1452 but still won't work. I'm able to ping the 10.4.152.1 since my cloudbr3 is configured with a working configuration. I use the following netplan config (I redacted the other bridges). One perhaps important caveat: I have to rename "ens7f1np1" since they are too long in order for Linux to create bridges for them, this is known limitation (15 characters) (brens7f1np1-1452 = 16 chars). Could this be the issue perhaps?

network:
  version: 2
  ethernets:
    eno5np0: {}
    eno6np1: {}
    ens0np0:
      match:
        name: ens7f0np0
      set-name: ens0np0
    ens1np1:
      match:
        name: ens7f1np1
      set-name: ens1np1
  vlans:
    < redacted >
    ens1np1.1452:
      id: 1452
      link: "ens1np1"
  bridges:
    < redacted >
    cloudbr3:
      addresses:
      - "10.4.152.10/24"
      routes:
      - to: "10.4.152.0/24"
        via: "10.4.152.1"
      dhcp4: false
      dhcp6: false
      interfaces: [ ens1np1.1452 ]

@sjanssen15
I think the problem is that ens1np1.1452 uses cloudbr3 as its master bridge.
You can try to stop the VRs and rename cloudbr3 to brens1np1-1452

Just to add, after troubleshooting this week we got it working in our environment 👍🏻 thanks Wei

merging based on user testing