I couldn't confirm before upgrading, but post upgrading & renewing the host certificates, vm instances wouldn't deploy on an aarch64 host due to libvirt/vnc certificate errors (I had checked the pem file existed & had a certificate content). After comment the tls/vnc config in /etc/libvirt/qemu.conf I could get unsecured console to work in the web/novnc popup. I didn't hit this kind of issue on x86 host, so likely this could be just an aarch64 env/config issue.
The following was seen in the cloudstack-agent logs on the aarch64 host:
2024-11-27 12:47:59,353 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:[]) (logid:bd7b5a1e) Creating volume 54eeeb93-bccc-47f4-bc81-e14db979c9fa from template 92105cb5-2728-40b8-ae28-64a621f493e0 in pool 63229d40-f348-4636-8707-74ab46ab28d0 (Filesystem) with size (100.00 GB) 107374182400
2024-11-27 12:47:59,354 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:[]) (logid:bd7b5a1e) Attempting to create volume 54eeeb93-bccc-47f4-bc81-e14db979c9fa (Filesystem) in pool 63229d40-f348-4636-8707-74ab46ab28d0 with size (3.50 GB) 3758096384
2024-11-27 12:47:59,597 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:[]) (logid:bd7b5a1e) Trying to fetch storage pool 63229d40-f348-4636-8707-74ab46ab28d0 from libvirt
2024-11-27 12:47:59,625 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:[]) (logid:bd7b5a1e) Trying to fetch storage pool 63229d40-f348-4636-8707-74ab46ab28d0 from libvirt
2024-11-27 12:47:59,947 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-5:[]) (logid:bd7b5a1e) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not available. Transformations will not be applied.
2024-11-27 12:47:59,947 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-5:[]) (logid:bd7b5a1e) Groovy scripting engine is not initialized. Data transformation skipped.
2024-11-27 12:48:00,892 WARN [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-5:[]) (logid:bd7b5a1e) LibvirtException org.libvirt.LibvirtException: internal error: process exited while connecting to monitor: 2024-11-27T07:18:00.671827Z qemu-system-aarch64: Cannot load CA certificate '/etc/pki/libvirt-vnc/ca-cert.pem': Error while reading file.
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.domainCreateXML(Unknown Source)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1909)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:87)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:48)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1945)
at com.cloud.agent.Agent.processRequest(Agent.java:686)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1109)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
ISSUE TYPE
COMPONENT NAME
CLOUDSTACK VERSION
CONFIGURATION
Adv zone without security groups, with two clusters - one of x86 with three x86 hosts and one aarch64 cluster with a single arm64 host.
I couldn't confirm before upgrading, but post upgrading & renewing the host certificates, vm instances wouldn't deploy on an aarch64 host due to libvirt/vnc certificate errors (I had checked the pem file existed & had a certificate content). After comment the tls/vnc config in
/etc/libvirt/qemu.confI could get unsecured console to work in the web/novnc popup. I didn't hit this kind of issue on x86 host, so likely this could be just an aarch64 env/config issue.The following was seen in the cloudstack-agent logs on the aarch64 host:
ISSUE TYPE
COMPONENT NAME
CLOUDSTACK VERSION
CONFIGURATION
Adv zone without security groups, with two clusters - one of x86 with three x86 hosts and one aarch64 cluster with a single arm64 host.